How to install Fedora on top of ZFS

This time, we will help you set up a newly installed Fedora system on your computer, which will boot from a ZFS file system. In a later guide, we'll show how to migrate an existing Fedora system to ZFS. After you are done setting up your ZFS system this way, you can keep ZFS properly updated using this Ansible playbook.

If you find any errors during this process, file a bug immediately on this bug tracker. I welcome bug reports to get this thing to work for everyone who uses this page.

What's this ZFS thing? Why would I want it on my computer?

ZFS is, bar none, the best file system ever invented. It's better than ext4, better than XFS, and better than btrfs (which is, to date, the most advanced file system included in the Linux kernel).

If you want high reliability, data integrity and ease of administration for your computer, then you want to install Fedora atop ZFS.

I'll help you do exactly that.

Gather what you'll need

This guide should take about 25 to 30 minutes of your time, unless you're making a speed run. You will need:

1. A computer with a hard disk (or SSD, same thing). Conceivably, you could install on an USB drive too. We'll stick to the hard disk in this case.
2. A USB drive or CD-ROM with Fedora 16 Live booting from it. You can record a CD or use liveusb-creator to afford your USB the Live CD.

Install Fedora

We are going to install a Fedora copy on your computer.

Boot Fedora Live

Tell your computer that you want to boot from the USB drive or CD-ROM containing the Fedora Live installation you have at hand. For this, the instructions will vary from computer to computer, so we won't say much about that.

Start the installation program

Now run the Install to Hard Disk program in the System tools menu.

You can use the installation program as usual. We'll just have to make a quick stop on the partitioning part, where we'll hold your hands. Select Create custom layout from the options:

Partition your disk correctly

OK, now that you're you're here, continue and create the following partition layout:

1. A BIOS boot partition, 2 MB in size
2. A /boot partition (ext4 or ext2, whatever you prefer) around 500 MB in size
3. A swap partition, probably twice as big as the RAM memory in your system.
4. An ext4 partition, mounted nowhere. You must select Fill to maximum available size
5. Another ext4 partition, mounted nowhere. You must select the same Fill option as before.
6. Once you have done this, select the very last partition, and change it so it mounts on /.

In sum: it's very important that the last partition on disk is the / partition, and that the swap partition be at the beginning of the disk. This is how it will look like in the end:

Now hit Next.

Continue the installation after rebooting

You can continue the installation as normal. When the installation is done, eject the CD or USB drive, and reboot. After rebooting, complete the Fedora post-install wizard. Here's the product of your work:

Disable SELinux

ZFS has full SELinux support, but I have never tried it. So, I will ask you to disable it, unless you know how to get it to work. If you do, see the top of this page to file a bug with instructions on how to get it to work. I will happily update this tutorial with your instructions. For the time being, however:

As root, edit /etc/selinux/config — if the file exists, then disable SELinux by changing SELINUX to disabled. Save the file.

Then disable enforcement for the duration of your current session by running the command:

setenforce 0

Yes, SELinux is important. We know. For the time being, however, your system will require that you disable SELinux unless you can get it to work (and, if you do, please file that bug I asked about).

Install ZFS

Since the installation of ZFS is a bit involved, we'll give you a few steps to run by hand and then a script that will take care of ZFS just fine. The following steps you will need to run as root, so open a terminal window and run the following commands as root.

Install dependencies first

You'll need to a series of dependencies to compile ZFS, so install them first:

command -v dnf && cmd=dnf || cmd=yum ; $cmd install -y git python

Clone the ZFS Fedora installer

OK, after doing this, clone this ZFS Fedora installer repository

cd /usr/src
git clone https://github.com/Rudd-O/zfs-fedora-installer

This should net you one directoriy in /usr/src containing these sources.

Run deploy-zfs

At this point, you're ready to install ZFS. Run it:

/usr/src/zfs-fedora-installer/deploy-zfs

This will leave you with a number of RPM packages in /usr/src/grub-zfs-fixer/RPMS/noarch, /usr/src/spl and /usr/src/zfs. They will be already installed and ready to go.

Reboot

Since the kernel-devel package is only available for the kernel version you were running at the time of the install — confirm with uname -r in a terminal window — the ZFS modules are only going to be available on that kernel. So you are now going to reboot using the same kernel.

Reboot your machine. When it reboots, select the same kernel you were using before (for the record: normally, it's the first choice). After boot is done, confirm that you're using the latest kernel using the commands rpm -q kernel and uname -r.

Prepare the ZFS pool and file systems

OK. You have rebooted. Good.

Remember when we created a partition that was empty initially during the installation? Excellent. We're going to create a pool right there. In the following example, we assume that your root file system is mounted in /dev/sda5 and the empty space is mounted in /dev/sda4. It would be a good idea to check this fact with the mount and parted commands before you proceed.

Create the storage pool

Run the following commands:

zpool create pool -m none -o ashift=12 /dev/sda4

The -m none parameter tells the pool to not mount itself anywhere. The -o ashift=12 parameter is optional, but it will make your pool perform better on advanced-format 4K sector disks (like pretty much all newer SSDs and hard disks).

If you did it right, you should be able to do the following and see a screen similar to this:

Create the root file system, along with other file systems

Strictly speaking, this step is optional. You could use your pool exactly as it was when it was created. However, it's a good idea to follow the next step because it allows you to carry multiple operating systems and system directories in your pool more easily, so it's recommended.

zfs create pool/ROOT
zfs create pool/ROOT/fedora

You should see something like this, after listing them:

pool/ROOT/fedora will be your root file system. Feel free to create as many file systems underneath it as you would like, now. pool/ROOT/fedora/home, pool/ROOT/fedora/var, go wild like this:

zfs create pool/ROOT/fedora/home
zfs create pool/ROOT/fedora/var

Note: there is an exception to this. Remember that /usr cannot be an independent file system for now -- you can create it and set its canmount property to no, then create child file systems with canmount=yes within /usr, but you cannot directly mount any file system on /usr without your system breaking completely. This issue will be addressed in the future by the ZFS team. In the meantime. /usr must be available as part of the root file system. The same applies to /etc.

Set optional properties on the pool

This is an optional step. Now is the best time to set properties like compression or no access time updates on your pool. If you have a decent CPU, compression will give you higher performance, especially with slower disks. Disabling access time recording gives an extra boost. This is the moment to set them:

zfs set compression=on pool
zfs set atime=off pool

OK. Now every byte hitting your pool will be compressed. Nice space savings!

Export the pool and reimport it with an alternate root

Run the following commands:

zpool export pool
zpool import -o altroot=/sysroot pool

This will make the ZFS system mount any file systems within the pool in /sysroot.

Mount the file system that will be the destination for your operating system

Set the mountpoint property on the root file system for your operating system:

zfs set mountpoint=/ pool/ROOT/fedora

Your file systems will now look like this:

Feel free to set the mountpoint property for other file systems you may have created, that aren't within pool/ROOT/fedora. These will get mounted upon boot, transparently, without any intervention.

Migrate and configure the operating system

Copy the operating system to /sysroot

The next step is straightforward and it only requires waiting a bit:

rsync -axvHASX / /sysroot/

Note the slash in the end of /sysroot/!

Populate /dev

You'll need the device files too. We will use bind mounts to resolve that:

mount --bind /dev /sysroot/dev

Chroot into the /sysroot file system

That's it. Use chroot to enter that directory as if it was the root.

Mount /dev, /boot, /sys and /proc

Within the chroot:

mount -t proc proc /proc
mount -t sysfs sys /sys
mount /boot

This is an approximation of how your system should look like (within the chroot) after mounting some of these file systems.

Edit fstab to fix the root file system

Now you'll be editng the root file system entry in fstab. Edit the file /etc/fstab, and change the line that mounts / as follows:

pool/ROOT/fedora / zfs defaults 0 0

You may want to replace defaults with noatime for higher performance (the root file system gets mounted as a separate task during early boot). This is how it should look like:

Redo GRUB configuration

We will be using the special grub2-mkconfig that we patched earlier for this purpose (by way of installing grub-zfs-fixer). Within the chroot, run the following command:

/usr/src/grub2-mkconfig | tee /boot/grub2/grub.cfg

Your screen should look a bit like this. Note the root=ZFS=pool/ROOT/fedora parameter! This is what tells the initial RAM disk to boot your ZFS-backed root file system (the nitty-gritty of what happens is on this source file).

Regenerate the initial RAM disk

OK, now it's time to regenerate the initial RAM disk. The following command accomplishes that:

dracut -fv --kver `uname -r`

Ignore any alarming messages this command gives you about no such dataset. These are temporary error messages because you are in a chroot.

Unmount mounted file systems inside the chroot, and exit the chroot

Clean up your chroot, and exit it.

umount /boot
umount /sys
umount /proc
umount /dev
exit

Export the pool

To ensure a clean system, export the pool now.

zpool export pool

If this process hangs, just wait thirty seconds, and then hard power off your machine. ZFS won't lose a bit because of this.

Reboot

Reboot your operating system. The system should boot normally, this time from ZFS.

Confirm that your system is now using ZFS as the root file system

Run the mount command. It should show you the following, confirming that ZFS is now your root file system, and everything works normally:

Enlarge the pool to take the unused space up

The last and final step is short and sweet. It involves nuking the ext4 partition and enlarging the ZFS partition to take up the empty space. We'll show you how to do that right now.

Nuke the ext4 partition

Open parted on a terminal as root. Follow these steps:

• Remove partition 5: rm 4
  This is the ext4 partition we created earlier that was mounted on /.
• Resize partition 4 to take up the newly freed space:
  • Change to sectors unit: unit s
  • Show the list of partitions: p
  • Make a note of the starting size of partition 4, and the total size of the disk.
  • Remove partition 4: rm 4
    Ignore any errors.
  • Create a new partition: mkpart.
    • Give it no name.
    • Set its type to ext4 (doesn't matter, really).
    • Set its starting point to the old partition 4's starting point.
    • Set its ending point to the size of the disk in sectors minus 1.
    • Ignore all errors.

After all the changes, parted should look like this:

Reboot your machine now. As long as the starting sector of the ZFS partition remains unchanged, ZFS will be unfazed by this change in partitions.

Enlarge the pool

After the reboot, run the command:

zpool online -e pool /dev/sda4

Once you have done that, the pool will have gobbled up the new space in the bigger partition. Look:

That's it!

And that is it. That is really the end. You have a system that runs ZFS and boots from it. Great job! From now on, every time you update the kernel, the system will automatically rebuild the ZFS kernel modules and deploy them to your newer initial RAM disk (or at least that is what ought to happen).

Taking care of ZFS-specific updates and unbootable systems after updates

A very important thing to remember is that this process provides no mechanism to update ZFS automatically. This means: from time to time, you need to update ZFS in order to keep up with changes that might make your system unbootable. If, after an update, your system won't boot, then you can use the boot menu to start your system with an older working kernel, and then update ZFS on your system by running a process very similiar to the one you ran before:

cd /usr/src/zfs-fedora-installer  # this changes to the installer directory
git fetch origin                  # this fetches the latest updates
git reset --hard origin/master    # this checks out the necessary files
./deploy-zfs --upgrade            # this rebuilds and upgrades the RPMs
dracut -fv --regenerate-all       # this regenerates the initial RAM disks

In the future, the ZFS on Linux project will release more enhancements to get your system up and running with the latest version of ZFS without this process. For the moment, your system is going to do fine.

Good luck, and happy hacking!

Huge thanks to @nevermaker for helping me with test-driving the entire process and documenting each step!