FreeBSD 9 on Soekris net6501

This is an installation guide for installing FreeBSD 8 or FreeBSD 9 on a Soekris net6501 (it has been tested with 8.2 and 9.0RC3). It uses a bootable USB stick to install FreeBSD on another drive in the Soekris (in this case, on a build-in solid state disk).

Serial Interface
As a first step, connect the Soekris to your computer with a serial cable. The second step is to set the serial speed to either 9600 or 19200 baud. By default, the Soekris BIOS communicates at 19200 baud, and most operating systems communicate at 9600 baud. Since it is inconvenient to switch speeds on your terminal, I recommend to use the same speed for all communications (BIOS, bootloader and final OS).

The net6501 manual recommends to set the connected ANSI/VT100 terminal or terminal emulator to 19200 baud, 8 databits, no parity, 1 stop bit, no flow control. I also had to set key emulation to use carriage return (CR) as the line ending.

The rest of this guide describes how to set the communications speed of the OS to 19200 baud. This is less common, but makes the serial interface faster. If you see garbage on your screen, your communication speed is wrong.

If you instead wish to change the BIOS speed to 9600 baud, press control-P to jump into the comBIOS monitor and set the speed: > set ConSpeed=9600 > reboot

Boot Sequence
Before you continue, check for known issues or firmware updates. comBIOS version 1.41 or higher is required for USB booting (only on the Soekris net6501; for the Soekris net55xx and net45xx, it will be supported in a future release)

Write down your disk settings. In my case I had both a 16GB Transcend SSD disk in port-00 and a 4GB SanDisk Compact Flash (CF) stick:

SATA AHCI BIOS ver. 0.6 20110902 Copyright (C) 2003-2011 Intel Corporation Controller Bus#02, Device#06, Function#00: 02 Ports, 01 Devices Port-00: Hard Disk, TS16GMSA300 Port-01: No device detected Soekris USB Expansion ROM ver. 1.01 20111203 81: USB 01 SanDisk Cruzer Fit      Xlt 969-128-63  3910 Mbyte

Check the comBIOS settings, in particular the boot sequence:

> show BootDrive BootDrive = 80 81 F0 FF

The meaning of these numbers is as follows:
 * 0x80 = primary drive
 * 0x81 = secondary drive
 * 0x82 = tertiary drive
 * 0x83 = quaternary drive
 * 0x84 = quinary drive (etc.)
 * 0xF0 = PXE boot (netboot)
 * 0xFF = stop and go to comBIOS prompt

So the default setting is to first try the primary drive, if that fails the secondary drive, if that fails netbooting, and otherwise, present the comBIOS prompt.

The drive numbers are not fixed, but depend on the installed hardware. In order, the following hardware is checked, and if available, a driver number is assigned: (Note: I'm not sure about the order of the PCI Express and SATA ports)
 * 1) SATA Port 0 (J1)
 * 2) SATA Port 1 (J2)
 * 3) PCI Express connector (J3)
 * 4) PCI Express riser (J4)
 * 5) mSATA 0 socket (J5)
 * 6) mSATA 1 socket (J6)
 * 7) external USB connector (JP3)
 * 8) internal USB connector (JP5)

For example, I have a solid state disk (SSD) in mSATA socket 0, and a USB stick in the internal USB connector. The SSD is assigned boot number 0x80, and the USB sticks boot number 0x81. Only the 0x81 is reported by comBIOS:

SATA AHCI BIOS ver. 0.6 20110902 Copyright (C) 2003-2011 Intel Corporation Controller Bus#02, Device#06, Function#00: 02 Ports, 01 Devices Port-00: Hard Disk, TS16GMSA300 Port-01: No device detected Soekris USB Expansion ROM ver. 1.01 20111203 81: USB 01 SanDisk Cruzer Fit      Xlt 969-128-63  3910 Mbyte

If I would add another USB disk in the external connector, this would change to: 81: USB 01                         Xlt -2-32   Mbyte 82: USB 02 SanDisk Cruzer Fit      Xlt 969-128-63  3910 Mbyte

The driver number of the internal USB disk has changed. Also note that not all USB sticks works flawlessly. In this case, the capacity of a 8 GByte Imation USB sticks is not reported. I would recommend to use a USB sticks that is properly recognised for an installation procedure.

Decide on your Installation Method
There are three ways to install FreeBSD:
 * PXE booting:Run the installer on the Soekris using PXE (netbooting), and install on another drive in the Soekris. This is the traditional way, as described by Jeremy Chadwick and others.
 * External host:Take the destination drive out of the Soekris, and hook it to a computer, and run the installer on that computer. This is possible if the OS should be installed on a USB drive. This is described in Barry O'Donovan's guide.
 * USB stick installation:Create a bootable USB stick with installed, boot the Soekris with this USB stick, and install on another drive in the Soekris. This method is described in this guide.

The procedure here greatly borrows from Barry O'Donovan's guide (in particular his advice on VirtualBox), as well as the FreeBSD Handbook chapters on headless installation (without monitor or keyboard), serial console set up and (to a lesser extend) diskless operation.

Creating an Installer on USB Stick
FreeBSD 8 and higher have a memstick distribution. Download this ISO file. E.g. wget ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/ISO-IMAGES/8.2/FreeBSD-8.2-RELEASE-i386-memstick.img or wget ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/9.0/FreeBSD-9.0-RELEASE-i386-memstick.img

Plug in an empty USB stick in your computer, and determine the device identifier (e.g. /dev/da0 on FreeBSD, /dev/disk2 on Mac OS X, /dev/sda1 on Linux, etc.)

On my Mac, I used diskutil to find the correct device name, and to unmount the whole disk (Mac OS X somehow ejecting it from the Finder was not good enough): diskutil list diskutil unmountDisk /dev/disk2

copy the disk image like this (use the correct device name for the output): dd if=FreeBSD-9.0-RELEASE-i386-memstick.img of=/dev/da0 bs=4k

Make sure that the USB sticks is larger than the disk image. Larger is not a problem. FreeBSD will complain about non-matching media size, but work just fine. GEOM: da0: media size does not match label.

Headless Booting Configuration
The USB stick now contain a bootable installer disk, but must be modified before it can be used to install on a machine without monitor (headless booting).

Insert the USB stick in a machine that can read the UFS2 file system. If you do not have such a system, you must create one. On my Mac, I used VirtualBox to create a FreeBSD installation. If you have problems mounting the USB stick in Virtualbox, see Mounting a USB stick with VirtualBox.

Mount the USB stick: mkdir -p /mnt/stick mount /dev/da0a /mnt/stick

Check if the USB stick is successfully mounted ls /mnt/stick

Make adjustmenst to USB key, as described in the FreeBSD Handbook chapter on Installing FreeBSD on a System without a Monitor or Keyboard:

The boot loader and kernel must be told to use a serial console at 19200 baud (replace 19200 with 9600 if you use 9600 baud). This is specified in Edit /boot/loader.conf. If this file does not exist, create it. (ee is a simple editor, replace with emacs or vim if you prefer.)

ee /mnt/stick/boot/loader.conf and add the lines: console="comconsole" comconsole_speed="19200"

While you are there, I also add the following lines (the last one is commented out since it is not necessary for the installer disk). The legacy aliases disables old-fashioned ATA (/dev/ad0) naming convention and always uses modern AHCI naming (/dev/ada0) for disks. kern.cam.ada.legacy_aliases=0 zfs_load="YES"
 * 1) vfs.root.mountfrom="zfs:zroot"

The FreeBSD handbook chapter on Setting Up the Serial Console suggests to improve security by setting the console to use vt100, and turn on password login. I personally didn't bother, since if an attacker has physical access to my device, he has access to my device and network and a password won't stop him from gaining access.

ee /mnt/stick/etc/ttys and editing the ttyu0 line: ttyu0  "/usr/libexec/getty std.19200"   vt100   on secure

(again, replace 19200 with 9600 if you use 9600 baud for your serial connection).

Boot the FreeBSD Installer
Plug in the USB stick in your Soekris and boot.

Boot Soerkis and jump into comBIOS monitor (with control-P)

The default Boot drive is: > show BootDrive = 80 81 F0 FF

But the USB key is at device 81:

81: USB 01 SanDisk Cruzer Fit      Xlt 969-128-63  3910 Mbyte

So tell the Soekris to boot this device:

> boot 81

You should now see a spinning line on the user interface. Enjoy, since it is Soekris bootstrapping FreeBSD.

If you see garbage like this, your console settings are wrong (in this case, it seems that the line break is wrong): -.\.|./.-.\.|./.-.\.|./.-.\.|./.-.\.|./.-.\.|./.-.\.|./.-.\.|./.-.\.|./.-.\.|./.. [6n.[02;02H C.[02;03H o.[02;04H n.[02;05H s.[02;06H o.[02;07H l.[02;08H e.[02;09H s. [02;10H :.[02;11H .[02;12H i.[02;13H n.[02;14H t.[02;15H e.[02;16H r.[02;17H n.

Wait for half a minute. At FreeBSD, I had this garbage, but once the bootloader took over, things improved. If it does not improve, fix your serial console settings. Pay attention to line break and speed.

Install FreeBSD
Install FreeBSD by following the instructions in the FreeBSD installer. For detailed instructions how to create a bootable ZFS disk, see FreeBSD 9 with ZFS bootdisk.

One issue to be aware of is that at the end of the installation process, you must alter the /boot/loader.conf and /etc/ttys just as list above under. Since the boot disk is mounted under /mnt</tt> by the installer, you might expect them to be available at /mnt/boot/loader.conf</tt> and /mnt/etc/ttys</tt>. This is true if you select the Live CD option at the end of the installation sequence. However, if you choose the default Reboot, you will end up in a chrooted jail. In that case, the files are available as /boot/loader.conf</tt> and /etc/ttys</tt>. What's more, running the zfs set mountpoint=/ zroot</tt> command fails in a chrooted jail with internal error: failed to initialize ZFS library</tt>. The solution is to choose the Live CD option at the end of the installation sequence.