I've been away from FreeBSD for a few years but I wanted some more functionality on my home network that I was able to configure with my Synology NAS and router. Specifically, I wanted:
- a configurable caching name server that would serve up authoritative private names on my LAN and also validates responses with DNSSEC.
- a more configurable DHCP server so I could make the server assign specific IPs to specific MAC addresses.
- more compute power for transcoding videos for Plex.
Running FreeBSD 11 on an Intel NUC seemed like an ideal solution to keep my closet tidy. As of this week, $406.63 on Amazon buys a last generation i3 Intel NUC mini PC (NUC5I3RYH), with 8GB of RAM and 128GB of SSD storage. This was the first model I tried since I found reports of others using this with FreeBSD online, but I was also able to get it working on the newer generation i5 based NUC6i5SYK with 16GB of RAM and 256GB of SSD. The major issue with these NUCs is that the Intel wireless driver is not supported in FreeBSD. I am not doing anything graphical with these boxes so I don't know how well the graphics work, but they are great little network compute nodes.
Installation
I downloaded the FreeBSD 11 memory stick images, and was pleased to see that the device booted fine off the memory stick without any BIOS configuration required. However, my installation failed trying to mount root ("Mounting from ufs:/dev/ufs/FreeBSD_Install failed with error 19."). Installation from an external USB DVD drive and over the network with PXE both proved more successful at getting me into bsdinstaller to complete the installation.
I partitioned the 128GB SSD device with 8GB of swap and the rest for the root partition (UFS, Journaled and Soft Updates). After installation I edited /etc/fstab to add a tmpfs(5) mount for /tmp. The dmesg output for this host is available in a Gist on Github.
Warren Block's article on SSD on FreeBSD and the various chapters of the FreeBSD Handbook were helpful. There were a couple of tools that were also useful in probing the performance of the SSD with my FreeBSD workload:
- The smartctl tool in the sysutils/smartmontools package allows one to read detailed diagnostic information from the SSD, including wear patterns.
- The basic benchmark built into diskinfo -t reports that the SSD is transferring 503-510MB/second.
Rough Benchmarks
This post isn't meant to report a comprehensive suite of FreeBSD benchmarks, but I did run some basic tests to understand how suitable these low power NUCs perform in practice. To start with, I downloaded the 11-stable source from Subversion and measured the build times to understand performance of the new system. All builds were done with a minimal 2 line make.conf:
MALLOC_PRODUCTION=yes CPUTYPE?=core2
Build Speed
| Build Command | Environment | Real Times |
|---|---|---|
| make -j4 buildkernel | /usr/src and /usr/obj on SSD | 10.06 minutes |
| make -j4 buildkernel | /usr/src on SSD, /usr/obj on tmpfs | 9.65 minutes |
| make -j4 buildworld | /usr/src and /usr/obj on SSD | 1.27 hours |
| make buildworld | /usr/src and /urs/obj on SSD | 3.76 hours |
Bonnie
In addition to the build times, I also wanted to look more directly at the performance reading from flash and reading from the NFS mounted home directories on my 4-drive NAS. I first tried Bonnie++, but then ran into a 13-year old bug in the NFS client of FreeBSD. After switching to Bonnie, I was able to gather some reasonable numbers. I had to use really large file sizes for the random write test to eliminate most of the caching that was artificially inflating the results. For those that haven't seen it, Brendan Gregg's excellent blog post highlights some of the issues of file system benchmarks like Bonnie.| Configuration | Random I/O | Block Input | Block Output | |||
|---|---|---|---|---|---|---|
| Seeks/Sec | CPU Utilization | Reads/sec | CPU Utilization | Writes/sec | CPU Utilization | |
| NFS | 99.2 | 0.9 | 106505 | 4.8 | 89966 | 7.5 |
| SSD | 8809 | 13.5 | 538671 | 25.33 | 160917 | 11.3 |
The block input rates from my bonnie benchmarks on the SSD were within 5% of the value provided by the much quick and dirtier diskinfo -t test.
Running Bonnie with less than 40GB file size yielded unreliable benchmarks due to caching at the VM layer. The following boxplot shows the random seek performance during 3 runs each at 24, 32, and 40GB file sizes. Performance starts to even off at this level but with smaller file sizes the reported random seek performance is much higher.
Open Issues
As mentioned earlier, I liked the performance I got with running FreeBSD on a 2015-era i3 NUC5I3RYH so much that I bought a newer, more powerful second device for my network. The 2016-era i5 NUC 6i5SYK is also running great. There are just a few minor issues I've encountered so far:
- There is no FreeBSD driver for the Intel Wireless chip included with this NUC. Code for other platforms exists but has not been ported to FreeBSD.
- The memory stick booting issue described in the installation section. It is not clear if it didn't like my USB stick for some reason, or the port I was plugging into, or if additional boot parameters would have solved the issue. Documentation and/or code needs to be updated to make this clearer.
- Similarly, the PXE Install instructions were a bit scattered. The PXE section of the Handbook isn't specifically targetting new manual installations into bsdinstall. There are a few extra things you can run into that aren't documented well or could be streamlined.
- Graphics / X11 are outside of the scope of my needs. The NUCs have VESA mounts so you can easily tuck them behind an LCD monitor, but it is not clear to me how well they perform in that role.
