How to pivot your Qubes OS system entirely to a root-on-ZFS setup

Following from the earlier guide to pivot your Qubes OS storage pool into ZFS, this time we learn how to pivot your entire system into ZFS.

We'll clone all of the root file system into ZFS, make a copy of the boot partition and ESP partition, set the system to boot from the newly-created clones, and finally reclaim the storage in the old device.

But first — why would you want to do that?

• Reap all the benefits of ZFS for all the storage in your Qubes OS system — not just the storage of your qubes.
• Reclaim the storage used by the former Qubes storage pool (whether lvm or reflink), and use it for ZFS.
• Consolidate all the disk storage management operations under ZFS.  ZFS is so much nicer.

Assumptions

• You've followed the guide How to store your Qubes OS VMs in a ZFS pool already, so your target storage is already set up as per the guide.
  • /dev/sda still contains a complete Qubes OS system you're currently booting, where
    • sda1 is the EFI system partition,
    • sda2 is your Qubes /boot partition, and
    • sda3 contains either an encrypted LVM pool or an encrypted btrfs file system (with your root file system in it, but no qube storage any more).
  • /dev/sdb1 contains a clone of /dev/sda1 (your EFI System Partition) which you made before.
    • This clone should be up to date.  If not, re-clone it using the procedure described in the prior guide (linked above).
  • /dev/sdb2 contains a clone of /dev/sda2 (your Qubes /boot partition) which you made before.
    • This clone should be up to date.  If not, re-clone it using the procedure described in the prior guide (linked above).
  • /dev/sdb3 contains an encrypted, operational ZFS pool, which already has all your qube storage within it, and is registered in /etc/crypttab.
    • Your ZFS pool is named laptop for the purposes of this exercise.
• Your system boots using EFI rather than legacy BIOS — it has program efibootmgr available and efibootmgr -v shows a boot entry.

Install required software

Two key packages will be required to make this work:

• zfs-dracut — the Dracut support for ZFS.
  • You should take advantage of the same mechanism you used to install ZFS to install this module.
  • RPMs for all the ZFS packages that will work in Qubes OS 4.1 are available here.
• grub-zfs-fixer — a module that lets GRUB understand ZFS root file systems.
  • This package is provisionally available for Qubes OS 4.1 here.

Create the root file system

zfs create -o mountpoint=/laptop -p laptop/ROOT/os

You should now have a dataset ready for the cloning, mounted under /laptop.

Create the swap partition

zfs create -o primarycache=metadata -o compression=zle \
           -o logbias=throughput -o sync=always \
           -o primarycache=metadata -o secondarycache=none \
           -o com.sun:auto-snapshot=false -V 8G laptop/swap
mkswap /dev/zvol/laptop/swap

And now you have swap.

Synchronize the data of the running root file system into ZFS

rsync -vaxHAXSP / /laptop/ ; rmdir /laptop/laptop

From this point on, we will operate exclusively on the cloned system, and no longer on the former system.  Any changes you perform to the running system will not be available in the new one.

Configure the mount table of the new system

Find out the block device UUIDs for /dev/sdb1 and /dev/sdb2 by using command blkid /dev/sdb1 /dev/sdb2

Edit /laptop/etc/fstab to change the /boot UUID to the UUID of /dev/sdb2, and the /boot/efi UUID to the UUID of /dev/sdb1.

Also change the / file system record to have device laptop/ROOT/os with type zfs.

Finally, change the entry for the swap to use device /dev/zvol/laptop/swap.

Save your edits.  You are done with this.

Edit GRUB menu information

The file /laptop/etc/default/grub contains a variable GRUB_CMDLINE_LINUX that has a number of options which are now obsolete.  Edit it to perform the following changes:

• Find out the UUID of the encrypted device /dev/sdb3 which backs your ZFS storage pool.
• Change the rd.luks.uuid option to contain the UUID of /dev/sdb3.  This tells the initial RAM disk to unlock your ZFS pool on boot.
• Remove any rd.lvm options from the variable.  These would prevent your system from booting as it attempts to start the LVM volume group which will no longer be available during boot.
• Remove the root option if it is present.  This will be automatically generated.

Save your edits.

Apply the boot configuration changes to the new system

mount --bind /dev /laptop/dev
mount --bind /proc /laptop/proc
mount --bind /sys /laptop/sys
mount --bind /sys/firmware/efi/efivars /laptop/sys/firmware/efi/efivars
mount /dev/sdb2 /laptop/boot
mount /dev/sdb1 /laptop/boot/efi
chroot /laptop grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg

Once you have done this, you must edit /laptop/boot/efi/EFI/qubes/grub.cfg as follows:

• Locate the first boot entry recorded there, and within that boot entry, find the line that starts with module2 /vmlinuz....
• Within that line you will find a root= option.  Ensure the root option says root=laptop/ROOT/os.
• In the same line, add rd.break=pre-mount.
• Then save your edits.

From /laptop/boot/crypttab, remove all devices not required to boot the system.  The only line that must remain there is the line that corresponds to the encrypted device backing your ZFS pool.  To know which one of the lines to keep, you can check the UUID of the block device of the ZFS pool (in our guide it's /dev/sdb3) with command blkid /dev/sdb3.  Now regenerate the RAM disks in your cloned system:

chroot /laptop dracut -fv --regenerate-all

Now to create the EFI boot entry for the brand new finished root-on-ZFS system:

chroot /laptop efibootmgr -v -c -u -L "Qubes ZFS" -l /EFI/qubes/grubx64.efi -d /dev/sdb -p 1

Reboot to BIOS and select the new option

Once you've gone into BIOS (hit F2 or DEL after reboot), go to your BIOS boot options, and select Qubes ZFS from the menu.

Save your BIOS options and reboot the system.

Unlock the system and set the root file system mount point

As the system reboots this time, it will ask for your encryption password normally.

Once you've entered it, it will drop you to a shell where it says Press ENTER for maintenance (Control+D to continue).

Press ENTER!

Now you're dropped into an initial RAM disk shell.

At this point, you must inform the system that it should mount the new ZFS root file system, not on /laptop, but on /.  We could not do this before because the file system cannot be double-mounted on / (it technically can be, but it screws up with your system majorly, and you will not be able to come out of that easily).

# export the pool
zpool export laptop
# import it, instructing ZFS to mount everything under /tempmount
zpool import laptop -N -R /tempmount
# set the mountpoint — because of -R above, it will not conflict
zfs set mountpoint=/ laptop/ROOT/os
# export the pool again
zpool export laptop
# reimport the pool one more time
zpool import -N laptop

Now exit that shell.

The system will boot normally, and it will be running 100% ZFS from this point on.

Write the final GRUB configuration

Now that your system is 100% on ZFS (which you can verify by looking at what is mounted on / with the mount command), you can proceed with writing the final GRUB configuration.

grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg

Test it once again by rebooting your system.  You should no longer be dropped into a rescue or initial RAM disk shell, because the rd.break option was just erased from the GRUB menu entry.

Reclaim the storage formerly occupied by the default Qubes storage configuration

At this point you have a handy choice:

1. Remove the no-longer-used storage device that carries your old, non-ZFS-root Qubes OS system.
2. Use that storage as additional ZFS storage.

We'll explore option 2 here.

Now that the former block device is free to use, you can simply add it to your ZFS pool:

# Partition the device again
fdisk /dev/sda
# ...
# ... delete all partitions ...
# ... create a single whole-disk partition ...
# ... save and quit ...

# Encrypt the device — use the same password as your boot encryption password!
cryptsetup luksFormat --type=luks2 --align-payload=4096 /dev/sda1
cryptsetup luksOpen /dev/sda1 luks-`blkid -s UUID -o value /dev/sda1`

# Add it to your pool
zpool add laptop /dev/disk/by-id/dm-uuid-*-luks-blkid -s UUID -o value /dev/sda1`

# Register the new device in crypttab so it is available during boot
dev=`blkid -s UUID -o value /dev/sda1`
echo luks-$dev UUID=$dev none discard >> /etc/crypttab
dracut -fv --regenerate-all

You must add the UUID of the encrypted device (available as $dev in the script above) to your /etc/default/grub GRUB_CMDLINE_LINUX variable, in the same format as the other rd.luks.uuid= option already there.  To make this change take effect, run:

grub2-mkconfig -o /boot/efi/EFI/qubes.grub.cfg

Presto!  Now you have extra storage, and it's all on ZFS.