Installing FreeBSD 10.0 NanoBSD on a Soekris net6501

I recently built a NanoBSD configuration of FreeBSD 10.0 for a Soekris net6501 box, in a VirtualBox build environment on my mac.

I had to piece this procedure together using information from a variety of sources, so I’m documenting it here in the (longshot) hope that it might be useful to someone else somewhere/someday.

What You Need

  • Familiarity with FreeBSD. This is not a UNIX tutorial.
  • VirtualBox on your mac. If you are new to VirtualBox read up on the basics on your own. I’ll outline my approach and give details on one tricky part: accessing a USB stick from a virtual machine.
  • Some familiarity with NanoBSD. Again, this is not a tutorial but I will talk about the configuration options I used for the Soekris.
  • A Soekris 6501. I’m using the -50 model (1GB memory, 1GHz).
  • A Mac with lots of free disk space. Probably 32GB minimum.
  • A null-modem cable
  • An ancient vt100-ish terminal or a USB/Serial adapter on your mac. I’m using this Keyspan product and “screen” (built into OS X).
  • A low-profile USB stick (to fit into the Soekris case) such as the Sandisk Cruzer Fit. I’m using a 16GB stick.

What You’ll End Up With

Your Soekris box will be running NanoBSD booting off the USB stick. You get a wonderful environment that runs read-only from the USB and is a solid environment for hosting embedded applications. You will (presumably) use the serial console terminal just enough to configure network access and other items specific to your project and then work via ssh or whatever after that.

First: Get the terminal working

Connect your terminal (or emulator) to the 6501 and power on. The factory delivers the Soekris set to 19200 on its serial port. Here’s my command for accessing the console on my mac:

screen /dev/ttySOMETHING 19200

where “SOMETHING” depends on your particular adapter. Mine is /dev/tty.USA19H142P1.1 (nice, huh?). An old real terminal would also work (set to 19200).

You should see the power on self test output and eventually see a “> ” prompt. Type show and press return. If you see something like this then everything is working:

> show

ConSpeed = 19200
ConLock = Enabled
ConMute = Disabled
BIOSentry = Enabled
PCIROMS = Enabled
PXEBoot = Enabled
FLASH = Primary
BootDelay = 5
FastBoot = Disabled
BootPartition = Disabled
BootDrive = 80 81 F0 FF 
ShowPCI = Enabled
Reset = Hard
CpuSpeed = Default

FreeBSD runs the console at 9600, so change the Soekris BIOS to also use 9600 on the serial port. This is the path of least resistance (vs changing FreeBSD). Type this:

set conspeed=9600

Reboot the box and restart your terminal emulator at 9600 and verify you still have connectivity.

VirtualBox FreeBSD

We’ll make a virtual machine environment using VirtualBox and use that to install FreeBSD and build the Soekris installation.

I imagine that the outline of this procedure would work just as well for other OS variants you might want to build/install on the Soekris, though of course the kernel configuration details would differ.

Go to the FreeBSD site and download the amd64 disc1 CD .iso file.

Create a machine in VirtualBox but do not start it yet. I used these parameters:

  • Type: BSD
  • Version FreeBSD (64bit)
  • Memory Size 1024MB (not sure how much this matters)
  • Create Virtual drive. I used 64GB. You need space to hold the build tree and the eventual USB stick image. In my case I appear to be using about 10GB of the virtual drive I created, so you could certainly get by with less than 64GB if you are short on disk space.

Before starting the virtual machine, attach the .iso file CD image to it as a CD in the drive. To do this select the virtual machine, click the settings gear, go to the storage pane and click around until you figure out how to put the .iso file into the virtual CD drive. Also go to the System pane and verify that it is booting off the CD drive before the hard drive.

Now you can start the machine. It should boot from the FreeBSD “CD” (.iso file) and launch you into the FreeBSD install dialog. Follow the instructions. IMPORTANT: don’t forget to select the option to install the source code when you are asked about optional components (it’s not installed by default).

When it is done it will reboot, which of course will launch you right back into the CD install menu again. So before you let it reboot take the FreeBSD .iso image out of the virtual CD drive (click on the CD icon at the bottom of the VirtualBox window for your machine). Now let the machine reboot and you should be presented with a login prompt (password as however you set it during install).

You now have a virtualized FreeBSD environment running on your mac. Now we’ll use that environment to build the NanoBSD disk image that will eventually boot on the Soekris.

Configuring a net6501 kernel

We need to build a custom kernel because the GENERIC FreeBSD kernel won’t run on the Soekris box; you need to add “device mptable” to it to make it work. So do that. On your FreeBSD virtual machine go to /usr/src/sys/amd64/conf and copy the GENERIC config file to SK6501 (or whatever you want to call it). Edit this file and add “device mptable” and (for good housekeeping) change the ident from GENERIC to something else. Here’s the totality of a diff from the GENERIC config to my config:

< ident		GENERIC 
> ident		SK6501
> device	mptable	  # for Soekris

I put the device line right before the plethora of standard options “SCHED_ULE”, “PREEMPTION” etc.

You also need to insert a delay during booting because the USB stick doesn’t always become “ready” in time to mount the root file system. There are two ways to do this. Either put

options CAM_BOOT_DELAY=10000

into the kernel config, or else you need to get


put into your /boot/loader.conf boot parameters file. You need to do that via a NanoBSD customization command. See the NanoBSD docs for an example and you can follow the “nobeastie” example (see below) as a template.

Without either of these your system may not always boot reliably; it will sometimes drop you into the mountroot prompt on the console after failing to mount the / filesystem. The issue, as far as I understand it (which is not very well) has to do with the USB stick being “ready” in time for the mounting of / at boot time. The delay (that’s 10 seconds) makes it work reliably; I did not spend any time trying to determine the minimal delay necessary.

Of the two methods I prefer building the delay into my kernel configuration because that way it is always “there” regardless of the boot parameters file.

Creating a NanoBSD configuration

Next we need to customize a few NanoBSD options, including telling NanoBSD to use our modified kernel configuration. The NanoBSD tools are in /usr/src/tools/tools/nanobsd (yes, “tools” twice).

Create a file “sk.conf” with your NanoBSD particulars. Some of this is up to your specific application, but some of it is important for getting the Soekris to work. Here’s what I think is the minimum set of requirements:

NANO_MEDIASIZE=`expr -e 6442450944 / 512`
cust_nobeastie() (
	touch ${NANO_WORLDDIR}/boot/loader.conf
	echo "beastie_disable=\"YES\"" >> ${NANO_WORLDDIR}/boot/loader.conf

customize_cmd cust_nobeastie
customize_cmd cust_comconsole

The “SK6501” for NANO_KERNEL must match whatever name you picked for your kernel config file. The “NANO_NAME” can be whatever you want, that will control where the build goes (it will show up later in a “nanobsd.sk6501” directory if you set the NANO_NAME to “sk6501”). For MEDIASIZE pick something that makes sense for your application. My choice results in two 3GB partitions (active and backup) and of course the standard tiny /cfg partition (read up on NanoBSD for details on all this). The extra space on my 16GB USB sticks is going to be used for another partition in my application. You’ll likely want to adjust your MEDIASIZE for your own purposes.

The nobeastie thing simplifies the boot menu; painting the BSD daemon in ASCII art at 9600 isn’t worth waiting for. In case your web browser has mangled that code, note that the “echo” command line needs to be all on one line.

The comconsole function (built in to NanoBSD environment) alters the /etc/ttys file so that the serial port console will have a getty running on it when the system boots. The default is to look for a VGA console which isn’t there on the Soekris.

You will likely want/need to read up on more NanoBSD options because you’ll probably eventually want to change more than just these parameters. I’m just showing how to get to the point where you can build a working system and boot it and you can iterate from there.

Building NanoBSD

When you’ve got your NanoBSD config file “sk.conf” (or whatever you called it) all ready, it’s time to build the world:

# cd /usr/src/tools/tools/nanobsd
# sh -c sk.conf

I suggest redirecting that output to a logfile and running in the background in whatever your favorite UNIX idiom for that is.

It took a long time on my mac, come back in about two hours YMMV.

If you are lucky, you’ll end up with an exit 0 and a complete build.

There was a bug in earlier releases of FreeBSD10 that caused me to write the next few paragraphs that are now all indented and italicized. The bug is fixed and so you shouldn’t have to hack out the “conv=sparse” as described below. For now I’m leaving this information here for reference but you really should ignore it unless you for some reason are trying an older FreeBSD10:

I ran into a problem that seems to be a bug in FreeBSD10 and is triggered by the use of “conv=sparse” in a dd command in the script. You can read a thread about this and patch your kernel, OR you can do the much simpler thing and edit the script to remove the “conv=sparse” from the dd command that is creating the secondary (backup) partition on the NanoBSD image.

If your build fails with an error during the dd of the secondary partition, then look for this line in

dd conv=sparse if=/dev/${MD}s1 of=/dev/${MD}s2 bs=64k

and simply delete the “conv=sparse” part. This will work-around the bug. Restart your nanobsd build using the “-b” option (to skip the building of the world and the kernel; you don’t need to rebuild from scratch at this point) and hopefully this time it will make your disk image correctly. Whether you hit the bug or not depends on a bunch of things so your mileage may vary.

The conv=sparse thing just saves disk space in your build environment and therefore on your mac; it has no semantic effect on the disk image. Indeed the script in FreeBSD9.2 didn’t specify this in the dd command; it’s new in the version 10 release.

When you are done you will be rewarded with a bunch of files in /usr/obj/nanobsd.sk6501 (where “sk6501” is whatever NANO_NAME you set in the .conf file you made).

There should be a file _.disk.full there. This is what you are now going to transfer to your USB stick in the next step.

Shut down (“# shutdown -h now”) your virtual FreeBSD environment and use VirtualBox to power off the virtual machine. We’re going to be editing its parameters next (which can only be done when it is shut down).

USB Stick Magic

Now we need to get that _.disk.full image file onto a USB stick. There might be a better way to do this, but what I did was make the physical USB stick show up as a disk drive in the FreeBSD virtual machine, so I could then just dd the disk image directly onto the stick.

THIS PROCESS IS FRAUGHT WITH PERIL. If you get the device name wrong you might wipe your mac. That’s your problem, not mine. Pay attention to what you are doing!

Put your target USB stick into a USB port on your mac. Pull up a shell window and type “diskutil list”. From that you should be able to tell which disk device your USB stick is showing up as. It is /dev/disk3 on my mac. YOU NEED TO GET THIS RIGHT so be sure.

For the rest of this writeup I will use /dev/disk999 as a placeholder for whatever /dev/diskN device is appropriate for your configuration.

If there are filesystems on this device that your mac recognized you need to unmount them:

MyMac:$ diskutil unmountdisk /dev/disk999

You need read/write access to this device as an ordinary user.  On my mac the device is only read/write by root, so I did:

MyMac:$ sudo chmod 666 /dev/disk999

You will likely have to repeat one or both of these actions every time your mac reasserts dominion over the USB stick (e.g., if you reinsert it)

At this point we’re ready to create a “VMDK” file which VirtualBox will use as a descriptor/pointer to the physical device. There is a VirtualBox command line tool for this, VBoxManage. Go to some suitable directory (you’ll need to remember this later) and use this command (all on one line of course):

MyMac:$ VBoxManage internalcommands 
 createrawvmdk -filename stick.vmdk 
 -rawdisk /dev/disk999

The file (“stick.vmdk”) this creates is a text file – you can examine it and see that the device name (/dev/disk999 or whatever is right on your system) is in there. You could edit this later if that changes for some reason.

Now you can attach this to your FreeBSD virtual machine. Go back into the VirtualBox GUI. Your virtual machine should be powered off at this point (“# shutdown -h now” and then power it virtually off). Go into its settings and use the add disk icon on the storage pane. You are “adding an existing disk” and choose the .vmdk file you just added.

Now your physical USB stick will show up as a virtual raw device in your FreeBSD virtual machine. Start the FreeBSD virtual machine and:

# cd /usr/obj/nanobsd.sk6501
# ls -l _.disk.full

You should see a large file – this is the drive image that NanoBSD created. Your virtualized USB stick drive should show up in your FreeBSD environment as /dev/ada1. Verify that that device exists and then you are ready for:

# dd if=_.disk.full of=/dev/ada1 bs=1m

On my mac this took almost an hour for a 6GB image. I don’t know if there is a better faster way. Of course this is the step where if you specified the wrong /dev/diskN in your VMDK file you are now obliterating *some* device on your mac somewhere instead of writing to the USB stick. Don’t do that; be very sure you got all the device names correct in the VMDK setup.

Booting the Soekris

Now the easy and fun part! Shut down your FreeBSD virtual machine again (“# shutdown -h now” and then power it off in VirtualBox) and pull the USB stick. Put the USB stick into your Soekris. Start your terminal emulator up again and power up the Soekris.

You should be rewarded with a login prompt on the console. Log in as root (no password) and start configuring your system. Be sure you understand how NanoBSD works and how to save configuration changes so they’ll persist to the next boot; the rest of the process from here is beyond the scope of this write-up and obviously depends a lot on what you are trying to do.

Going back and forth

Once you’ve reached this point, you can put the USB stick back into your mac too and access it (mount it) as a regular filesystem from your VirtualBox FreeBSD environment. That can sometimes be a useful way to get stuff onto the Soekris. Just be careful to be sure you’ve got the correct /dev/diskN configuration every time you re-insert the USB stick into your mac.

Revisit your NanoBSD conf file

I do recommend that after you’ve configured your system the usual “trial and error and iteration” way on the Soekris itself that you go back to the NanoBSD build environment and automate all your configuration changes into the build process itself. Between chroot hacks and just general cleverness (and taking advantage of all the infrastructure already provides) you should be able to automate whatever it is you’ve done interactively to set the box up. The advantage of building those changes into the build process is that they will happen automatically the next time you need to update from source for a new BSD release or any other reason.

Don’t forget to save away your .conf files and everything else you put into the VirtualBox environment!


One Reply to “Installing FreeBSD 10.0 NanoBSD on a Soekris net6501”

Comments are closed.