Emergency commands for encrypted Archlinux installations

ArchLinux is a fantastic rolling distribution but sometimes you will end up with a broken or semi-broken system that you need to fix. Getting back on tracks might sound even scarier when running a full disk encryption installation on SSD but LVM brings some nice goodies that will make your life easier. For information, I have run LUKS encryption on LVM for many years and I can tell you that it is rock solid and very robust. I have tortured it many many times, powered it off without any notice and haven’t (yet) lost a single bit of data, although there is always this half second pause after entering the initial passphrase that makes you think “…is it going to be able to decrypt my drive?…”.

Perform filesystem check on LVM partitions

fsck on lvm can be a bit tricky. You need to activate the lvm with some special options to be able to access it:

    # loadkeys fr
    # cryptsetup luksOpen /dev/sdaX mylvm
    # vgscan
    # vgchange -ay

if the above fails use the following command:

    # vgchange --ignorelockingfailure -ay
    # lvscan --ignorelockingfailure (this command should now work)

And then perform the filesystem check:

    # fsck /dev/volumegroup/lvname

Mounting encrypted partitions with USB Flashdrive

First of all let’s get familiar with emergency boot in the case something goes wrong with your kernel or filesystem. The only thing you need is an ArchLinux LiveCD USB drive (also tested with other Linux based Live CD). Please note that even your Android smartphone can act as a bootable USB flash drive with ArchLinux ISO stored on its SD card and using DriveDroid application.

After booting the LiveCD you might want to run fdisk/gdisk /dev/sda to find out about the lvm partition number and also make sure that the Flash drive hasn’t been mounted as /dev/sda.

# loadkeys fr
# modprobe dm-crypt (might be required on non-ArchLinux distributions)
# cryptsetup luksOpen /dev/sdaX mylvm
# vgscan
# vgchange -ay
# mount /dev/mapper/vgroup-root /mnt
# mkdir /mnt/boot
# mount /dev/sdaX /mnt/boot
# mkdir /mnt/home
# mount /dev/mapper/vgroup-home /mnt/home

From there you can see your files and also chroot into your SSD installation to perform maintenance tasks:

 # arch-chroot /mnt

Perform stress-free ArchLinux upgrades

In very rare cases you will end up with a fully broken system when using ArchLinux rolling release. Although significant updates like Gnome can have some glitches I recommend to take advantage of the LVM snapshot capabilities to get avoid any stress.

Backup kernel files

The /boot directory is not encrypted and stays outside the LVM space, and thus LVM snapshots. Therefore we need to back it up separately. Best is to create a sub-directoy in /boot so the backup kernels don’t get listed on GRUB.

# mkdir /boot/archives

Then prior to any [core] kernel update:

# sudo cp /boot/initramfs-linux.img /boot/archives/.
# sudo cp /boot/ /boot/vmlinuz-linux /boot/archives/.

and for CK kernel you need two more lines:

# sudo cp /boot/initramfs-linux-ck.img /boot/archives/.
# sudo cp /boot/ /boot/vmlinuz-linux-ck /boot/archives/.

Perform system snapshot prior to upgrade

I have reserved 5G at the end of my LVM space for snapshots (1GB for home and 4GB for root).

# sudo lvcreate -L 4g -s -n root-snapshot /dev/vgroup/root
# sudo lvcreate -L 1g -s -n home-snapshot /dev/vgroup/home

From that point you can go ahead and start breaking things. Just make sure not to overload the snapshots (like launching a VM stored in your home directory) as you will lose the capability to go back in time. I typically use the command script to record the changes to be made on the system during a pacman -Syu and also save in a text file the name of the packages to be upgraded.

Remove snapshots when upgrade went fine

Then I reboot and check if everything is working back. When you are happy with the result you can safely remove the snapshots.

# sudo lvremove /dev/vgroup/root-snapshot                                                                                                                            :(
# sudo lvremove /dev/vgroup/home-snapshot                                                                                                                            :(

Go back to snapshot state in case of problem

Boot from ArchLinux USB flash drive

# loadkeys fr
# cryptsetup luksOpen /dev/sdaX mylvm
# vgscan
# vgchange -ay
# lvconvert --merge /dev/vgroup/root-snapshot

Please note that merge can also be used to restore home-snapshot if required.

# lvconvert --merge /dev/vgroup/home-snapshot

You will also need to manually copy back any kernels that you have saved in /boot/archive directory prior to reboot. First we need to mount /boot directory (remember it is not part of LVM).

# mount /dev/sdaY /mnt
# cp /mnt/archives/initramfs-linux.img /mnt/.
# cp /mnt/archives/vmlinuz-linux /mnt/.

and for CK kernel you need two more lines:

# cp /mnt/archives/initramfs-linux-ck.img /mnt/.
# cp /mnt/archives/vmlinuz-linux-ck /mnt/.
Share Comments
comments powered by Disqus