Fedora with Root on ZFS - installation notes

### Fedora: Root on ZFS

- Fedora: 28
- ZFS on Linux: 0.7.9
___

#### A. Prepare source

1. [Download](https://alt.fedoraproject.org) *Fedora Workstation* network installer image.
2. Boot live media by adding `inst.gpt` at the end of the boot line and set up with:
    - a boot partition `/boot/efi` of size 512MB
    - a root partition `/` of size *rest of the disk*
    - minimal installation
3. Reboot and:
    - run `dnf -y update`
    - set hostname, networking, *idmapd*
    - output of `rpm -qa|grep "grub2-efi-x64\|shim-x64"` should contain:
    <pre><code>grub2-efi-x64
    shim-x64</code></pre>
    - install UEFI modules and some required packages:
    <pre><code>dnf -y install grub2-efi-x64-modules kernel-devel gdisk rsync bridge-utils</code></pre>
    - add system wide profile customization in `/etc/profile.d/`
    - run `init 6`
4. Create scripts:

    - to run `dracut` and make `GRUB` ZFS aware:
    <pre><code>#!/bin/bash
    kver=$1
    sh -x /usr/bin/dracut -fv --kver $kver
    mount /boot/efi
    grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
    grubby --set-default-index 0
    mkdir -p /boot/efi/EFI/fedora/x86_64-efi
    cp -a /usr/lib/grub/x86_64-efi/zfs* /boot/efi/EFI/fedora/x86_64-efi
    umount /boot/efi</code></pre>

    - to enter `chroot`:
    <pre><code>#!/bin/bash
    target=$1
    mount -t proc  proc $target/proc
    mount -t sysfs sys $target/sys
    mount -o bind /dev $target/dev
    mount -o bind /dev/pts $target/dev/pts
    chroot $target /bin/env -i HOME=/root \
        TERM="$TERM" PS1='[\u@chroot \W]\$ ' \
        PATH=/bin:/usr/bin:/sbin:/usr/sbin \
        /bin/bash --login
    echo "Exiting chroot environment..."
    umount $target/dev/pts
    umount $target/dev/
    umount $target/sys/
    umount $target/proc/</code></pre>
    - make both scripts executable

5. Add ZFS support  to `GRUB` in `/etc/default/grub`:
  <pre><code>GRUB_PRELOADED_MODULES="zfs"</code></pre>
so that it can detect the ZFS filesystem; and
<pre><code>GRUB_DISABLE_OS_PROBER=true</code></pre>
so that `grub2-mkconfig` does not throw errors.

6. Fix `GRUB`'s handling of pool device path names by creating `/etc/profile.d/grub2_zpool_fix.sh` containing
<pre><code>export ZPOOL_VDEV_NAME_PATH=YES</code></pre>
The effect is that `zpool status` will list the full path of the block device: `/dev/disk/by-id/nvme0n1-part2` instead of `nvme0n1-part2`.

7. Prepare for future kernel updates:
<pre><code>ln -s /usr/local/sbin/zmogrify /etc/kernel/postinst.d</code></pre>

8. Add [ZFS on Linux](https://github.com/zfsonlinux/zfs/wiki/Fedora) repository then install and load ZFS:
<pre><code>dnf install zfs zfs-dracut
modprobe zfs</code></pre>

9. Disable cache and enable scan:
<pre><code>systemctl disable zfs-import-cache
systemctl enable zfs-import-scan</code></pre>

10. Regenerate `initramfs`:
<pre><code>dracut -fv --kver `uname -r`</code></pre>

11. Reboot
___

#### B. Prepare target

1. Wipe disks:
<pre><code>sgdisk --zap-all /dev/disk/by-id/nvme0n1
sgdisk --zap-all /dev/disk/by-id/nvme1n1</code></pre>

2. Create boot partition (UEFI) and dedicate the rest to ZFS:
<pre><code>sgdisk -n1:1M:+512M -t1:EF00 /dev/disk/by-id/nvme0n1
sgdisk -n2:0:0 -t2:BF01 /dev/disk/by-id/nvme0n1
sgdisk -n1:1M:+512M -t1:EF00 /dev/disk/by-id/nvme1n1
sgdisk -n2:0:0 -t2:BF01 /dev/disk/by-id/nvme1n1</code></pre>

3. Format boot partition:
<pre><code>mkfs.fat -F32 /dev/disk/by-id/nvme0n1-part1
mkfs.fat -F32 /dev/disk/by-id/nvme1n1-part1</code></pre>

4. Create the pool:
<pre><code>zpool create -m none -O canmount=off -o ashift=12 -o cachefile=none rpool mirror /dev/disk/by-id/nvme0n1-part2 /dev/disk/by-id/nvme1n1-part2</code></pre>

5. Create a container dataset:
<pre><code>zfs create -o canmount=off -o mountpoint=none rpool/ROOT</code></pre>

6. Create the root dataset:
<pre><code>zfs create rpool/ROOT/default
zfs set acltype=posixacl rpool/ROOT/default # systemd-journald's /var/log/journal
zfs set xattr=sa rpool/ROOT/default
zfs set compression=on rpool/ROOT/default
zfs set atime=off rpool/ROOT/default
</code></pre>

7. Create the `home` dataset:
<pre><code>zfs create -o compression=on -o atime=off -o xattr=sa -o acltype=posixacl rpool/home</code></pre>

8. Create a dataset for storing data:
<pre><code>zfs create -o compression=on -o atime=off -o xattr=sa -o acltype=posixacl rpool/local</code></pre>

9. Create a `swap` volume:
<pre><code>zfs create -V 16G -b $(getconf PAGESIZE) -o compression=zle -o logbias=throughput -o sync=always -o primarycache=metadata -o secondarycache=none -o com.sun:auto-snapshot=false rpool/swap
mkswap /dev/zvol/rpool/swap</code></pre>

10. Export pool:
<pre><code>zpool export rpool</code></pre>

11. Import pool into an alternate mount point:
<pre><code>zpool import rpool -d /dev/disk/by-id -o altroot=/sysroot</code></pre>

12. Set the mountpoint for the root and for `home` and `local` datasets:
<pre><code>zfs set mountpoint=/ rpool/ROOT/default
zfs set mountpoint=/home rpool/home # directory is empty
zfs set mountpoint=/local rpool/local # directory is empty</code></pre>

13. Check mountpoints.

14. Transfer data:
<pre><code>rsync -WSHAXhavx / /sysroot/</code></pre>

15. Copy EFI partition from the source to the target:
<pre><code>cd
mkdir here
mount /dev/disk/by-id/nvme0n1-part1 here
cp -a /boot/efi/* here
umount here
rmdir here</code></pre>

15. Add boot and swap partitions to `fstab`:
<pre><code>blkid /dev/disk/by-id/nvme0n1-part1</code></pre>
Replace content of `/sysroot/etc/fstab` with:
<pre><code>UUID=&lt;UUID from blkid&gt;  /boot/efi  vfat  umask=0077,shortname=winnt  0  2
/dev/zvol/rpool/swap    swap       swap  defaults                    0  0</code></pre>

17. Enter `chroot`:
<pre><code>zenter /sysroot</code></pre>

18. Needed by F28's `GRUB` is a line containing:
<pre><code>GRUB_DEVICE_BOOT=/dev/disk/by-id/nvme0n1-part2</code></pre>
in `/etc/default/grub`.

19. Transform target into a ZFS aware system:
<pre><code>zmogrify \`uname  -r\`</code></pre>

20. Confirm that `GRUB` can recognize the ZFS filesystem:
<pre><code>grub2-probe /</code></pre>
should output `zfs`.

21. Exit `chroot`

22. Export pool:
<pre><code>zpool export rpool</code></pre>

23. Reboot

24. Take an initial snapshot:
<pre><code>zfs snapshot rpool/ROOT/default@initial</code></pre>

25. Create user and group

26. Complete installation:
<pre><code>dnf -y groupinstall "Fedora Workstation"
systemctl set-default graphical.target
systemctl enable gdm.service</code></pre>

27. Reboot

___

*References*:

 * [https://www.csparks.com/BootFedoraZFS/index.md](https://www.csparks.com/BootFedoraZFS/index.md)
 * [http://matthewheadlee.com/projects/zfs-as-root-filesystem-on-fedora](http://matthewheadlee.com/projects/zfs-as-root-filesystem-on-fedora)
 * [https://ramsdenj.com/2016/06/23/arch-linux-on-zfs-part-2-installation.html](https://ramsdenj.com/2016/06/23/arch-linux-on-zfs-part-2-installation.html)
 * [https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS](https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS)
 * [https://wiki.archlinux.org/index.php/ZFS](https://wiki.archlinux.org/index.php/ZFS)
 * [https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-use-a-zvol-as-a-swap-device](https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-use-a-zvol-as-a-swap-device)
 * [https://illumos.org/man/1M/zpool](https://illumos.org/man/1M/zpool)
 * [https://illumos.org/man/1M/zfs](https://illumos.org/man/1M/zfs)
 * [https://docs.fedoraproject.org/en-US/fedora/f28/install-guide/advanced/Boot_Options](https://docs.fedoraproject.org/en-US/fedora/f28/install-guide/advanced/Boot_Options)