Difference between revisions of "FreeBSD 9 with ZFS bootdisk"

Latest revision as of 23:49, 30 October 2015

A large part of this tutorial is based on the excellent tutorials by George Kontostanos.

Preparation

First of all, create a USB memory stick with the FreeBSD installer, as explained in the FreeBSD handbook in the section on Prepare the Installation Media.

Run the FreeBSD installer and select "install" to start and installation procedure.

After setting the keyboard settings, hostname and selecting the components to install, you will reach the partitioning stage.

Create the ZFS partition

At the partitioning stage, select "shell". (Thus not "Guided" or "Manual".)

You first get a helpful message:

Use this shell to set up partitions for the new system. When finished, mount the system at /mnt and place an fstab file for the new system at /tmp/bsdinstall_etc/fstab. Then type 'exit'. You can also enter the partition editor at any time by entering 'bsdinstall partedit'.

First, determine the names of your disks:

# camcontrol devlist
<Corsair Force 3 SSD 5.02>         at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD30EZRX-00MMMB0 80.00A80>    at scbus3 target 0 lun 0 (ada1,pass1)
<Imation Atom PMAP>                at scbus7 target 0 lun 0 (da0,pass2)

For more info, you can search dmesg:

# grep 'ada0' /var/run/dmesg.boot
# egrep '[ac]?da[0-9]' /var/run/dmesg.boot

In this case, the USB key with the installer was located at /dev/da0, and the SSD which would contain the new system at /dev/ada0. The 3 TByte HDD at /dev/ada1 is not of importance yet (it will be configured later).

Once determined, use gpart to list the existing partitions (this list was engineered to show a few examples):

# gpart show
=>      0  7821312  da0  BSD  (3.7G)
        0  1095504    1  freebsd-ufs  (534M)
  1095504  6725808       - free -  (3.2G)

=>      34  30932925  ada0  GPT  (14G)
        34         6        - free -  (3.0K)
        40       984     1  freebsd-boot  (492k)
      1024  30932797     2  freebsd-zfs  (14G)

=>      34  30932925  ada1  GPT  (14G)
        34       500     1  freebsd-boot  (250k)
       534  10485760     2  freebsd-ufs  (5.0G)
  10485922  16777216     3  freebsd-ufs  (8.0G)
  27263138   3667968     4  freebsd-swap  (1.8G)
  30931106      1853        - free -  (926k)

If a disk has no known recognised partition table, it is not listed, or gpart gives an error:

# gpart show ada0
gpart: No such geom: ada0

If the disk already has partitions (and you no longer want the data on it), you delete all but the first partition. There is no need to delete the "freebsd-boot" partition, as you will need that anyway.

# gpart delete -i 4 ada0
# gpart delete -i 3 ada0
# gpart delete -i 2 ada0

If the disk does not yet have a partition table or boot partition, you should create that first:

# gpart create -s gpt ada0
# gpart add -b 40 -s 984 -t freebsd-boot ada0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

Previously, a freebsd-boot partition of 94 blocks was enough; this is no longer true for FreeBSD 9.2. The listed 984 blocks should be more than enough. Unfortunately, gpart does not seem to print an error in case the bootcode is too large for the partition.

Modern disks have a block size of 4 kiByte, but the GPT block size is only 512 bytes. For performance, it's best to align partitions with 4k block sizes. You either make sure each start (-b) and size (-s) size is dividable by 8, or simply specify -a 4k.

Next, create the ZFS partition and give it a label (in this case, I choose the name 'ssd0'). If you want the ZFS partition to use all disk space (which is a sane default), there is no need to specify the size parameter:

# gpart add -t freebsd-zfs -a 4k -l ssd0 ada0

However, I wanted to leave the last 4 GByte of the disk for two ZFS Intend Logs (ZIL) to speed up ZFS on other disks. In addition, I decided to use a ZFS device for swapping instead of a zvol.

# gpart add -t freebsd-swap -l swap0 -a 4k -s 4G ada0
# gpart add -t freebsd-zfs -l ssd0 -a 4k -s 100453096 ada0
# gpart add -t freebsd-zfs -l zil0 -a 4k -s 4194304 ada0
# gpart add -t freebsd-zfs -l zil1 -a 4k -s 4194304 ada0

The result is:

# gpart show -p ada0
=>      34   117231341     ada0  GPT  (14G)
        34           6           - free -  (3.0K)
        40         984   ada0p1  freebsd-boot  (492k)
      1024     8388608   ada0p2  freebsd-swap  (4.0G)
   8389632   100453096   ada0p3  freebsd-zfs  (47.9G)
 108842728     4194304   ada0p4  freebsd-zfs  (2.0G)
 113037032     4194304   ada0p5  freebsd-zfs  (2.0G)
 117231336           5           - free -  (2.5K)

Create the ZFS file system

Now it's time to create the ZFS file system. We will call it zroot, but you can pick any name you like. The device name is /dev/gpt/ followed by the name you gave to the freebsd-zfs partition above. Mount it at /mnt as requested by the FreeBSD Installer:

# zpool create -R /mnt zroot /dev/gpt/ssd0

If you want to create a mirror of two disks, the command is:

# zpool create -R /mnt zroot mirror /dev/gpt/ssd0 /dev/gpt/ssd1

This will mount the zroot partition at /mnt, as can be confirmed with mount:

% mount
zroot on /mnt (zfs, local, nfsv4acls)
% zpool list
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  51.5G   104K  51.5G     0%  1.00x  ONLINE  /mnt
~% zfs list  
NAME            USED  AVAIL  REFER  MOUNTPOINT
zroot           104K  51.3G    31K  /mnt

Note: please use the -R /mnt option, not the -m /mnt option. The -R option temporarily sets the mount point of the zpool (the altroot) to /mnt, exactly what we want. The -m option permanently sets the mount point of the root dataset to /mnt, which is not what we want. In case you accidentally used the -m option, you should correct with:

# zfs set mountpoint=/ zroot
# zfs set cachefile=none zroot

ZFS configuration

Now proceed to configure the create zroot pool:

# zpool set bootfs=zroot zroot
# zfs set checksum=fletcher4 zroot

Now create datasets. Datasets are similar to partitions in other file systems. Each dataset can have some attributes. For example, you can specify that /tmp may not exceed a certain size.

# zfs create zroot/var
# zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
# zfs create -o setuid=off zroot/tmp
# chmod 1777 /mnt/tmp

Optionally disable keeping track of the access time of file (this reduces the number of writes to a SSD):

zfs set atime=off zroot

If you have not create a swap device, you can create a swap dataset:

# zfs create -V8G zroot/swap
# zfs set org.freebsd:swap=on zroot/swap
# zfs set checksum=off zroot/swap

Otherwise, make sure to mount your swap device(s):

# cat << EOF > /tmp/bsdinstall_etc/fstab
# Device                       Mountpoint              FStype  Options         Dump    Pass#
/dev/gpt/swap0                 none                    swap    sw              0       0
/dev/gpt/swap1                 none                    swap    sw              0       0
EOF

You may create multiple mount points with ZFS if desired.

Installation of FreeBSD

If you're done, exit the shell to proceed with the regular installation:

# exit

Alternatively, you can do the installation manually:

# cd /usr/freebsd-dist/
# tar --unlink -xpJf base.txz -C /mnt
# tar --unlink -xpJf kernel.txz -C /mnt
# tar --unlink -xpJf ports.txz -C /mnt

and edit /mnt/etc/rc.conf to your liking.

Configuration of Boot Parameters

At the end of the installation procedure, you must first configure some boot parameters before you can reboot. Otherwise, booting will fail because the ZFS driver is not yet loaded.

You must make these changes from the command line, and have two changes of dropping in the shell after the installation process.

1. You can select "Yes" when asked "The installation is now finished. Before exiting the installer, would you like to open a shell in the new system to make any final manual modifications?"
2. Select "No" to the previous question, but select "Live CD" when asked "Installation of FreeBSD is complete! Would you like to reboot into the installed system now?" You can log in as root (with an empty password).

The difference between the two is that in the first case, you end up in chrooted jail, while you end up in the full system in the second case. Let's assume you choose the first option, than you need to edit the file /boot/loader.conf and add these two lines (if the file does not exist, you should create it):

zfs_load="YES"
vfs.root.mountfrom="zfs:zroot" 

If you would have chosen the "Live CD" option, you should have edited the file /mnt/boot/loader.conf.

ZFS settings

Side note: If you followed the above descriptions, there is no need to change anything settings of your ZFS file system anymore. If you nevertheless want to set some changes and get this error:

# zfs set mountpoint=/ zroot
internal error: failed to initialize ZFS library

This may be an indication that you are running in a chrooted jail. If you want to make changes to ZFS, make sure you choose the "Live CD" option, and not the "Shell" option, as explained in the previous section.

Troubleshooting

After a reboot, your machine should come up with a running FreeBSD system. If there are some errors in your settings, you may end up at the mountroot prompt.

After reboot I end up at the mount prompt:

mountroot>

And you can not boot ZFS from here:

mountroot> zfs:zroot
Trying to mount root from zfs:zroot []...
Mounting from zfs:zroot failed with error 2: unknown file system

This likely means that the bootcode you installed (with the gpart bootcode command earlier) worked fine, but it was not able to find the ZFS mounts.

The most likely cause is that your did forgot to add these lines to /boot/loader.conf.

vfs.root.mountfrom="zfs:zroot" 
zfs_load="YES"

In order to fix this, you should reboot from the installer CD, and choose the "Live CD" option. You first need to mount the zpool under /mnt. First list the available pools:

# zpool list
no pools available
# zpool import
  pool: zroot
    id: 128742958409357345692345
 state: ONLINE
action: The pool can be imported using its name of numeric identifier.
config:
        zroot       ONLINE
          ada1p2    ONLINE

Now import the zroot pool, and mount under /mnt:

# zpool import -R /mnt zroot

You can now edit /mnt/boot/loader.conf.

If you want to mount the ZFS pool as read-only, use:

zpool import -R /mnt -o readonly=on zroot