FreeBSD 9 with ZFS bootdisk

A large part of this tutorial is based on the excellent tutorials by George Kontostanos.

Preparation
First of all, create a USB memory stick with the FreeBSD installer, as explained in the FreeBSD handbook in the section on Prepare the Installation Media.

Run the FreeBSD installer and select "install" to start and installation procedure.

After setting the keyboard settings, hostname and selecting the components to install, you will reach the partitioning stage.

Create the ZFS partition
At the partitioning stage, select "shell". (Thus not "Guided" or "Manual".)

You first get a helpful message:


 * Use this shell to set up partitions for the new system. When finished, mount the system at /mnt and place an fstab file for the new system at /tmp/bsdinstall_etc/fstab. Then type 'exit'. You can also enter the partition editor at any time by entering 'bsdinstall partedit'.

First, determine the names of your disks:         at scbus0 target 0 lun 0 (ada0,pass0)    at scbus3 target 0 lun 0 (ada1,pass1)                at scbus7 target 0 lun 0 (da0,pass2)
 * 1) camcontrol devlist

For more info, you can search dmesg:
 * 1) grep 'ada0' /var/run/dmesg.boot
 * 2) egrep '[ac]?da[0-9]' /var/run/dmesg.boot

In this case, the USB key with the installer was located at /dev/da0, and the SSD which would contain the new system at /dev/ada0. The 3 TByte HDD at /dev/ada1 is not of importance yet (it will be configured later).

Once determined, use gpart to list the existing partitions (this list was engineered to show a few examples):

=>     0  7821312  da0  BSD  (3.7G) 0 1095504    1  freebsd-ufs  (534M) 1095504 6725808       - free -  (3.2G) =>     34  30932925  ada0  GPT  (14G) 34        6        - free -  (3.0K) 40      984     1  freebsd-boot  (492k) 1024 30932797     2  freebsd-zfs  (14G) =>     34  30932925  ada1  GPT  (14G) 34      500     1  freebsd-boot  (250k) 534 10485760     2  freebsd-ufs  (5.0G) 10485922 16777216     3  freebsd-ufs  (8.0G) 27263138  3667968     4  freebsd-swap  (1.8G) 30931106     1853        - free -  (926k)
 * 1) gpart show

If a disk has no known recognised partition table, it is not listed, or gpart gives an error: gpart: No such geom: ada0
 * 1) gpart show ada0

If the disk already has partitions (and you no longer want the data on it), you delete all but the first partition. There is no need to delete the "freebsd-boot" partition, as you will need that anyway.


 * 1) gpart delete -i 4 ada0
 * 2) gpart delete -i 3 ada0
 * 3) gpart delete -i 2 ada0

If the disk does not yet have a partition table or boot partition, you should create that first:


 * 1) gpart create -s gpt ada0
 * 2) gpart add -b 40 -s 984 -t freebsd-boot ada0
 * 3) gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

Previously, a freebsd-boot partition of 94 blocks was enough; this is no longer true for FreeBSD 9.2. The listed 984 blocks should be more than enough. Unfortunately, gpart does not seem to print an error in case the bootcode is too large for the partition.

Modern disks have a block size of 4 kiByte, but the GPT block size is only 512 bytes. For performance, it's best to align partitions with 4k block sizes. You either make sure each start (-b) and size (-s) size is dividable by 8, or simply specify.

Next, create the ZFS partition and give it a label (in this case, I choose the name 'ssd0'). If you want the ZFS partition to use all disk space (which is a sane default), there is no need to specify the size parameter:


 * 1) gpart add -t freebsd-zfs -a 4k -l ssd0 ada0

However, I wanted to leave the last 4 GByte of the disk for two ZFS Intend Logs (ZIL) to speed up ZFS on other disks. In addition, I decided to use a ZFS device for swapping instead of a zvol.


 * 1) gpart add -t freebsd-swap -l swap0 -a 4k -s 4G ada0
 * 2) gpart add -t freebsd-zfs -l ssd0 -a 4k -s 100453096 ada0
 * 3) gpart add -t freebsd-zfs -l zil0 -a 4k -s 4194304 ada0
 * 4) gpart add -t freebsd-zfs -l zil1 -a 4k -s 4194304 ada0

The result is:

=>     34   117231341     ada0  GPT  (14G) 34          6           - free -  (3.0K) 40        984   ada0p1  freebsd-boot  (492k) 1024    8388608   ada0p2  freebsd-swap  (4.0G) 8389632  100453096   ada0p3  freebsd-zfs  (47.9G) 108842728    4194304   ada0p4  freebsd-zfs  (2.0G) 113037032    4194304   ada0p5  freebsd-zfs  (2.0G) 117231336          5           - free -  (2.5K)
 * 1) gpart show -p ada0

Create the ZFS file system
Now it's time to create the ZFS file system. We will call it zroot, but you can pick any name you like. The device name is /dev/gpt/ followed by the name you gave to the freebsd-zfs partition above. Mount it at /mnt as requested by the FreeBSD Installer:


 * 1) zpool create -R /mnt zroot /dev/gpt/ssd0

If you want to create a mirror of two disks, the command is:


 * 1) zpool create -R /mnt zroot mirror /dev/gpt/ssd0 /dev/gpt/ssd1

This will mount the zroot partition at /mnt, as can be confirmed with mount:

% mount zroot on /mnt (zfs, local, nfsv4acls) % zpool list NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT zroot 51.5G   104K  51.5G     0%  1.00x  ONLINE  /mnt ~% zfs list NAME           USED  AVAIL  REFER  MOUNTPOINT zroot          104K  51.3G    31K  /mnt

Note: please use the  option, not the   option. The  option temporarily sets the mount point of the zpool (the altroot) to /mnt, exactly what we want. The  option permanently sets the mount point of the root dataset to /mnt, which is not what we want. In case you accidentally used the -m option, you should correct with:


 * 1) zfs set mountpoint=/ zroot
 * 2) zfs set cachefile=none zroot

ZFS configuration
Now proceed to configure the create zroot pool:


 * 1) zpool set bootfs=zroot zroot
 * 2) zfs set checksum=fletcher4 zroot

Now create datasets. Datasets are similar to partitions in other file systems. Each dataset can have some attributes. For example, you can specify that /tmp may not exceed a certain size.


 * 1) zfs create zroot/var
 * 2) zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
 * 3) zfs create -o setuid=off zroot/tmp
 * 4) chmod 1777 /mnt/tmp

Optionally disable keeping track of the access time of file (this reduces the number of writes to a SSD):

zfs set atime=off zroot

If you have not create a swap device, you can create a swap dataset:


 * 1) zfs create -V8G zroot/swap
 * 2) zfs set org.freebsd:swap=on zroot/swap
 * 3) zfs set checksum=off zroot/swap

Otherwise, make sure to mount your swap device(s):

/dev/gpt/swap0                none                    swap    sw              0       0 /dev/gpt/swap1                none                    swap    sw              0       0 EOF
 * 1) cat << EOF > /tmp/bsdinstall_etc/fstab
 * 2) Device                       Mountpoint              FStype  Options         Dump    Pass#

You may create multiple mount points with ZFS if desired.

Installation of FreeBSD
If you're done, exit the shell to proceed with the regular installation:


 * 1) exit

Alternatively, you can do the installation manually:


 * 1) cd /usr/freebsd-dist/
 * 2) tar --unlink -xpJf base.txz -C /mnt
 * 3) tar --unlink -xpJf kernel.txz -C /mnt
 * 4) tar --unlink -xpJf ports.txz -C /mnt

and edit /mnt/etc/rc.conf to your liking.

Configuration of Boot Parameters
At the end of the installation procedure, you must first configure some boot parameters before you can reboot. Otherwise, booting will fail because the ZFS driver is not yet loaded.

You must make these changes from the command line, and have two changes of dropping in the shell after the installation process.
 * 1) You can select "Yes" when asked "The installation is now finished. Before exiting the installer, would you like to open a shell in the new system to make any final manual modifications?"
 * 2) Select "No" to the previous question, but select "Live CD" when asked "Installation of FreeBSD is complete! Would you like to reboot into the installed system now?" You can log in as root (with an empty password).

The difference between the two is that in the first case, you end up in chrooted jail, while you end up in the full system in the second case. Let's assume you choose the first option, than you need to edit the file  and add these two lines (if the file does not exist, you should create it):

zfs_load="YES" vfs.root.mountfrom="zfs:zroot"

If you would have chosen the "Live CD" option, you should have edited the file.

ZFS settings
Side note: If you followed the above descriptions, there is no need to change anything settings of your ZFS file system anymore. If you nevertheless want to set some changes and get this error:

internal error: failed to initialize ZFS library
 * 1) zfs set mountpoint=/ zroot

This may be an indication that you are running in a chrooted jail. If you want to make changes to ZFS, make sure you choose the "Live CD" option, and not the "Shell" option, as explained in the previous section.

Troubleshooting
After a reboot, your machine should come up with a running FreeBSD system. If there are some errors in your settings, you may end up at the mountroot prompt.

After reboot I end up at the mount prompt:

mountroot>

And you can not boot ZFS from here:

mountroot> zfs:zroot Trying to mount root from zfs:zroot []... Mounting from zfs:zroot failed with error 2: unknown file system

This likely means that the bootcode you installed (with the  command earlier) worked fine, but it was not able to find the ZFS mounts.

The most likely cause is that your did forgot to add these lines to.

vfs.root.mountfrom="zfs:zroot" zfs_load="YES"

In order to fix this, you should reboot from the installer CD, and choose the "Live CD" option. You first need to mount the zpool under /mnt. First list the available pools:

no pools available pool: zroot id: 128742958409357345692345 state: ONLINE action: The pool can be imported using its name of numeric identifier. config: zroot      ONLINE ada1p2   ONLINE
 * 1) zpool list
 * 1) zpool import

Now import the zroot pool, and mount under :
 * 1) zpool import -R /mnt zroot

You can now edit.

If you want to mount the ZFS pool as read-only, use:

zpool import -R /mnt -o readonly=on zroot