Tuesday, October 17, 2023

Launching FreeBSD VMs in Azure with Bicep Templates

I have followed Colin Percival's excellent work on getting FreeBSD images supported on the public cloud provider infrastructure since 2010. I spent some time this weekend looking at the latest images and writing an Azure Bicep template to launch a FreeBSD VM. Microsoft now has an article about launching FreeBSD VM images with their relatively excellent free azure documentation and training. That doc makes a great starting point, but it doesn't show how to find the latest FreeBSD images or use Bicep templates. As of this writing, FreeBSD 13.2 is the latest image in the Azure MarketPlace, and VMs can be launched in Azure at least half a dozen different ways:

  • az command-line tool
  • Start-AzVM PowerShell
  • Bicep template engine
  • ARM Templates
  • Terraform
  • Pulumi
  • etc..

I'll show the az command-line needed to quickly launch a VM, but will focus mostly on Bicep templates, as this appears to be the current Microsoft-recommended Azure-specific template engine for automating Infrastructure as Code deployments on Azure. Before experimenting with this you need:

  • An active Azure subscription.
  • A user with owner/contributor roles.
  • A resource group.
These can all easily be created at the Azure Portal if this is your first time working with Azure.

Command Line VM Creation:

The easiest method to create a new VM is simply to use the az command. I created a resource group called "FreeBSD" and then looked for the latest FreeBSD image names:

To find a list of publishers of FreeBSD images you can use something like:

# az vm image list-publishers --location westus --output table|grep thefreebsdfoundation
westus      thefreebsdfoundation

To find "offers" and "skus" from this publisher:

# az vm image list-offers --location westus --publisher thefreebsdfoundation --output table
#westus      freebsd-13_2
# az vm image list-skus --location westus --publisher thefreebsdfoundation --offer freebsd-13_2 --output table
#westus      13_2-release
#az vm image list \
#    --location westus \
#    --publisher thefreebsdfoundation \
#    --offer freebsd-13_2 \
#    --sku 13_2-release \
#    --all --output table
#x64             freebsd-13_2  thefreebsdfoundation  13_2-release  thefreebsdfoundation:freebsd-13_2:13_2-release:13.2.0  13.2.0

Once we have identified a specific FreeBSD image string, you can create a VM instance with:

az vm create --name MyFreeBSD13 --resource-group FreeBSD --image thefreebsdfoundation:freebsd-13_2:13_2-release:13.2.0 --admin-username murray --generate-ssh-keys
ResourceGroup    PowerState    PublicIpAddress    Fqdns    PrivateIpAddress    MacAddress         Location    Zones
---------------  ------------  -----------------  -------  ------------------  -----------------  ----------  -------
FreeBSD          VM running               00-0D-3A-8B-22-FD  eastus

Behind the scenes, Azure is creating a number of resources on my behalf in addition to the VM instance:

  • A virtual network (VNET)
  • A virtual network interface (VNIC)
  • A public IPv4 address
  • A network security group (stateful level 4 firewall rules)
  • A persistent disk for the OS.
  • The virtual machine

All of these are created in my specified "FreeBSD" resource group, and I can ssh directly into the new VM, start installing packages or configuring it with Ansible, and otherwise prepare it for my use case. However, using a template engine can allow us to provide more configuration details about the VM to control costs and control how the resources are deployed.

Windows Azure Linux Agent

One thing to notice about the VM is that there are some Python 3.9 processes running by default. These are for the Windows Azure Linux Agent running from /usr/local/sbin/waagent and is outside the scope of this post.

Bicep Automation

The recommended way to author and deploy Bicep configurations is with Visual Studio Code and the Bicep extension. This extension allows you to visualize the resource graph that will be created by the config, handle autocompletion of long Azure resource names, and lets you deploy a configuration.

@description('Location for all resources.')
param location string = resourceGroup().location

@description('Prefix to use for VM names')
param vmNamePrefix string = 'BackendVM'

@description('Size of the virtual machines')
param vmSize string = 'Standard_D2s_v3'

// Unique DNS Name for the Public IP used to access the Virtual Machine.
@description('DNS Label Prefix for Public IP used by Azure to access the VM')
param dnsLabelPrefix string = toLower('FreeBSD-${uniqueString(resourceGroup().id)}')

var publicIPAddressName = 'FreeBSDPublicIP'
var virtualNetworkName = 'FreeBSDVnet'
var subnet1Name = 'FreeBSD-subnet-1'
var subnetRef1 = resourceId('Microsoft.Network/virtualNetworks/subnets', virtualNetworkName, subnet1Name)
var networkInterfaceName = 'nic'

//var numberOfInstances = 2

resource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-05-01' = {
  name: virtualNetworkName
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
    subnets: [
        name: subnet1Name
        properties: {
          addressPrefix: ''

  resource subnet1 'subnets' existing = {
    name: subnet1Name


output subnet1ResourceId string = virtualNetwork::subnet1.id

// public IP Address
resource publicIP 'Microsoft.Network/publicIPAddresses@2020-06-01' = {
  name: publicIPAddressName
  location: location
  properties: {
    publicIPAllocationMethod: 'Dynamic'
    publicIPAddressVersion: 'IPv4'
    dnsSettings: {
      domainNameLabel: dnsLabelPrefix
    idleTimeoutInMinutes: 4
  sku: {
    name: 'Basic'

resource networkInterface1 'Microsoft.Network/networkInterfaces@2021-05-01' = {
  name: '${networkInterfaceName}1'
  location: location
  properties: {
    ipConfigurations: [
        name: 'ipconfig1'
        properties: {
          privateIPAllocationMethod: 'Dynamic'
          subnet: {
            id: subnetRef1
          publicIPAddress: {
            id: publicIP.id
  dependsOn: [

resource vm1 'Microsoft.Compute/virtualMachines@2021-11-01' = {
  name: '${vmNamePrefix}1'
  location: location
  plan: {
    name: '13_2-release'
    product: 'freebsd-13_2'
    publisher: 'thefreebsdfoundation'
  properties: {
    hardwareProfile: {
      vmSize: vmSize
    osProfile: {
      computerName: '${vmNamePrefix}1'
      adminUsername: 'murray'
      adminPassword: 'XXXXXXX' // Use SSH keys instead, for testing with hard-coded password need 6+ chars, one lower, one upper, one special character
    storageProfile: {
      imageReference: {
        publisher: 'thefreebsdfoundation'
        offer: 'freebsd-13_2'
        sku: '13_2-release'
        version: '13.2.0'
      osDisk: {
        createOption: 'FromImage'
    networkProfile: {
      networkInterfaces: [
          id: networkInterface1.id
//    diagnosticsProfile: {
//      bootDiagnostics: {
//        enabled: true
//        storageUri: storageAccount.properties.primaryEndpoints.blob
//      }
//    }

Thats it! The Azure docs cover myriad configuration parameters to customize all aspects of the network and VM deployment, but this basic template should get you started with FreeBSD in Azure.

Wednesday, May 31, 2023

Triple Booting in 2023: FreeBSD, Windows, and Ubuntu Linux

Before this evening, I don't think I had setup a dual boot system in more than 20 years. In that time I've run mostly MacOS X, FreeBSD, and Linux desktops, and run Windows and some other Unix variants through Parallels on the Mac. However, I recently purchased a Dell Precision 3260 Compact workstation and mounted it under the desk in my home office. I wanted a compact machine to re-aquaint myself with modern Linux/FreeBSD Desktop environment, have a non-Virtualized copy of Windows, and a PCIe expansion slot for GPU, data acquisition tinkering, etc, and I think this fits the bill nicely. UEFI firmware makes this much easier than the last time I did this, but I still wanted to take a few notes in case I have to do this again (on my next PC laptop, most likely):


I bought the workstation without an OS, but I was able to download the Windows 10 installation media, write it to a bootable USB key, and then transfer an extra copy of a full Windows 10 license I have for an unneeded Parallels virtual machine. I opted to give Windows the first full SSD then split the second SSD between FreeBSD and Linux, since they will mount large home directories from my NAS over NFS. Installation was straightforward, I downloaded a few drivers from the Dell website, then did the free upgrade to Windows 11. I created a 550MB EFI System partition in the Windows partition editor, and then let Windows create 100MB, 16MB, and 465GB parititions for the rest of the install.


Second up was the FreeBSD install. I booted from a FreeBSD 14 memory stick and performed an install. The installer doesn't make it easy to split a disk with a ZFS partition between multiple operating systems. Rather than fiddle with it all manually, I opted for a UFS filesystem where I could specify half the ssd for FreeBSD and leave the remaining parts empty for the Linux install. The installer created a 260MB EFI partition, 233GB UFS root partition, and 8GB for swap. After a reboot I could boot into FreeBSD by pressing F12 and startup and manually selecting the second SSD to boot from, but the default would still boot into the Windows Boot Manager and load Windows 11.

Ubuntu Linux

Finally, I installed Ubuntu 22 LTS from a downloaded memory disk. The installer installed GRUB 2 and on first reboot it presented a menu with items for Windows Boot Manager and Ubuntu Linux, but no FreeBSD! I logged into Ubuntu and edited the /etc/grub.d/40_custom file to add the following:

  menuentry "FreeBSD" {
    set root=(hd1,1)
    chainloader (${root})/efi/boot/BOOTx64.efi

Then ran the command sudo update-grub. The output shows that it finds the FreeBSD 14 partition but for some reason wasn't creating a menu item until I manually added the line above.


With this change, I now get a nice GRUB2 menu of FreeBSD, Linux, or Windows to boot into, just like I had with my noisy tower desktop computers with multiple hard disks of the late 1990s. This is a secondary PC to my primary Mac Pro desktop, but I am happy to get the flexibility to jump into other bare metal operating systems when needed.

Some things to follow up on:

  • FreeBSD's install could make it easier to install FreeBSD on ZFS on only part of the disk/ssd, like it does for UFS by just asking you how much total space you want to occupy then handling all the other details.
  • Ubuntu's GRUB2 bootloader doesn't include a menu item for FreeBSD even though it was able to detect the FreeBSD partition. It would be more user friendly if the default config files built the FreeBSD menu item without requiring the 4 lines above to be added.
  • The steps above created three EFI system partitions of various sizes. This was easiest but wastes a bit of disk space.

Friday, May 5, 2023

Svelte Jails From Scratch

I recently upgraded one of my homelab servers to FreeBSD 14-CURRENT, and I updated my Ansible scripts to build up some needed Jails for various services. I have never relied on iocage, exjail, or the like and have instead typically built my jails from source. With the latest FreeBSD -CURRENT a make installworld / make distribution into the new jail DESTDIR created a 1.3GB installation. This is a large surface area for running just a single service such as nginx, and so I visited the src.conf build options to get the base system of each jail down to 85MB.

The largest parts of the default 14-CURRENT userland install is by far /usr/lib/debug at 724MB. Clang/LLVM/LLDB is another large chunk. The full src.conf to build a base userland without many unneeded bits is included below:


Note that WITHOUT_GH_BC is broken for installworld in -CURRENT at present, so I've commented it out. nginx, isc-dhcpd, and other packages I install my jails add about 40MB each, and so I'm pretty happy with an 85MB base system for each jail.

FreeBSD has a long history of projects to provide a minimal system for embedded devices and other use cases, such as PicoBSD (deprecated -- FreeBSD 3 on a single floppy!), NanoBSD, mfsBSD, and more. Please see these projects for more robust techniques to further minimize. This post is meant purely to see how small I can get an installed jail userland from the output of a default make buildworld by just setting build variables with make installworld.

Monday, May 1, 2023

New NUC13ANKi7 13th gen

I'd like to replace some of my older 2015-2016 era FreeBSD NUCs with something with more DRAM to support ZFS and more Jails, and a faster CPU to speed up my builds. After some research I settled on the NUC13ANKi7. This NUC fits a lot into the smaller short height form factor:

  • Intel Core i7-1360P
    • 12 cores (4 performance, 8 efficiency), 16 threads)
    • Performance cores turbo boost up to 5.0 GHz
    • Raptor Lake architecture
  • 64GB DDR4 DRAM.
  • 1TB PCIe x4 Gen4 NVMe
  • 2.5GB Ethernet, Wifi6E
  • UEFI with support for HTTP Boot.

The price from SimplyNUC is $1,189 plus tax and shipping. I would have preferred DDR5 DIMMs but otherwise was pretty happy with this, especially the smaller NUC form factor.

The machine builds and runs FreeBSD 13.2 great (see dmesg gist).

Buildworld Times

To test the suitability of this new machine as a build server, I ran 270 iterations of make buildworld over a week.
  • 30 -j parallelism options (1 to 30)
  • Three source and /usr/obj configurations:
    • /usr/src and /usr/obj on local SSD
    • /usr/src on SSD and /usr/obj on tmpfs
    • /usr/src and /usr/obj on tmpfs.
  • Three iterations of each test.

Median result of the three runs for the default ZFS /usr/src and /usr/obj runs are included in the plot below:

The fastest buildtimes occurred in under 26 minutes with -j21, but the performance of anything between -j16 and -j30 was within 1.3% of that elapsed build time. Moving src and obj to tmpfs led to less than a 4.1% delta in elapsed time.

SSD Performance

diskinfo -t nvd0 shows 3.9-4.35 MB/s transfer rates to the local SSD.

diskinfo -t nvd0

        512             # sectorsize
        1000204886016   # mediasize in bytes (932G)
        1953525168      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        PNY CS2140 1TB SSD      # Disk descr.
        PNY2201220106010B3A8    # Disk ident.
        nvme0           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Seek times:
        Full stroke:      250 iter in   0.004330 sec =    0.017 msec
        Half stroke:      250 iter in   0.019046 sec =    0.076 msec
        Quarter stroke:   500 iter in   0.013305 sec =    0.027 msec
        Short forward:    400 iter in   0.005383 sec =    0.013 msec
        Short backward:   400 iter in   0.005009 sec =    0.013 msec
        Seq outer:       2048 iter in   0.045332 sec =    0.022 msec
        Seq inner:       2048 iter in   0.029877 sec =    0.015 msec

Transfer rates:
        outside:       102400 kbytes in   0.025866 sec =  3958865 kbytes/sec
        middle:        102400 kbytes in   0.023508 sec =  4355964 kbytes/sec
        inside:        102400 kbytes in   0.024016 sec =  4263824 kbytes/sec


I also used IOZone benchmark to quickly gather some SSD stats with 4k reads:
MetricOutput in KBytes/sec
Initial write:1160931.00
Reverse Read1859190.25
Stride read568050.94
Random read143096.12
Mixed workload290691.06
Random write94226.63


One thing I wasn't expecting with this NUC is that it supports HTTP Boot and a UEFI shell. It seems more finicky than HTTPBoot on a Dell, and is possibly only looking at HTTPS URLs. Will try to follow up with another post in more detail about HTTP Boot with FreeBSD.