ArchLinux is a fantastic rolling distribution but sometimes you will end up with a broken or semi-broken system that you need to fix. Getting back on tracks might sound even scarier when running a full disk encryption installation on SSD but LVM brings some nice goodies that will make your life easier. For information, I have run LUKS encryption on LVM for many years and I can tell you that it is rock solid and very robust. I have tortured it many many times, powered it off without any notice and haven’t (yet) lost a single bit of data, although there is always this half second pause after entering the initial passphrase that makes you think “…is it going to be able to decrypt my drive?…”.
Perform filesystem check on LVM partitions
fsck on lvm can be a bit tricky. You need to activate the lvm with some special options to be able to access it:
# loadkeys fr
# cryptsetup luksOpen /dev/sdaX mylvm
# vgscan
# vgchange -ay
if the above fails use the following command:
# vgchange --ignorelockingfailure -ay
# lvscan --ignorelockingfailure (this command should now work)
And then perform the filesystem check:
# fsck /dev/volumegroup/lvname
Mounting encrypted partitions with USB Flashdrive
First of all let’s get familiar with emergency boot in the case something goes wrong with your kernel or filesystem. The only thing you need is an ArchLinux LiveCD USB drive (also tested with other Linux based Live CD). Please note that even your Android smartphone can act as a bootable USB flash drive with ArchLinux ISO stored on its SD card and using DriveDroid application.
After booting the LiveCD you might want to run fdisk/gdisk /dev/sda to find out about the lvm partition number and also make sure that the Flash drive hasn’t been mounted as /dev/sda.
# loadkeys fr
# modprobe dm-crypt (might be required on non-ArchLinux distributions)
# cryptsetup luksOpen /dev/sdaX mylvm
# vgscan
# vgchange -ay
# mount /dev/mapper/vgroup-root /mnt
# mkdir /mnt/boot
# mount /dev/sdaX /mnt/boot
# mkdir /mnt/home
# mount /dev/mapper/vgroup-home /mnt/home
From there you can see your files and also chroot into your SSD installation to perform maintenance tasks:
# arch-chroot /mnt
Perform stress-free ArchLinux upgrades
In very rare cases you will end up with a fully broken system when using ArchLinux rolling release. Although significant updates like Gnome can have some glitches I recommend to take advantage of the LVM snapshot capabilities to get avoid any stress.
Backup kernel files
The /boot directory is not encrypted and stays outside the LVM space, and thus LVM snapshots. Therefore we need to back it up separately. Best is to create a sub-directoy in /boot so the backup kernels don’t get listed on GRUB.
# mkdir /boot/archives
Then prior to any [core] kernel update:
# sudo cp /boot/initramfs-linux.img /boot/archives/.
# sudo cp /boot/ /boot/vmlinuz-linux /boot/archives/.
and for CK kernel you need two more lines:
# sudo cp /boot/initramfs-linux-ck.img /boot/archives/.
# sudo cp /boot/ /boot/vmlinuz-linux-ck /boot/archives/.
Perform system snapshot prior to upgrade
I have reserved 5G at the end of my LVM space for snapshots (1GB for home and 4GB for root).
# sudo lvcreate -L 4g -s -n root-snapshot /dev/vgroup/root
# sudo lvcreate -L 1g -s -n home-snapshot /dev/vgroup/home
From that point you can go ahead and start breaking things. Just make sure not to overload the snapshots (like launching a VM stored in your home directory) as you will lose the capability to go back in time. I typically use the command script to record the changes to be made on the system during a pacman -Syu and also save in a text file the name of the packages to be upgraded.
Remove snapshots when upgrade went fine
Then I reboot and check if everything is working back. When you are happy with the result you can safely remove the snapshots.
# sudo lvremove /dev/vgroup/root-snapshot :(
# sudo lvremove /dev/vgroup/home-snapshot :(
Go back to snapshot state in case of problem
Boot from ArchLinux USB flash drive
# loadkeys fr
# cryptsetup luksOpen /dev/sdaX mylvm
# vgscan
# vgchange -ay
# lvconvert --merge /dev/vgroup/root-snapshot
Please note that merge can also be used to restore home-snapshot if required.
# lvconvert --merge /dev/vgroup/home-snapshot
You will also need to manually copy back any kernels that you have saved in /boot/archive directory prior to reboot. First we need to mount /boot directory (remember it is not part of LVM).
# mount /dev/sdaY /mnt
# cp /mnt/archives/initramfs-linux.img /mnt/.
# cp /mnt/archives/vmlinuz-linux /mnt/.
and for CK kernel you need two more lines:
# cp /mnt/archives/initramfs-linux-ck.img /mnt/.
# cp /mnt/archives/vmlinuz-linux-ck /mnt/.