Svelte Jails From Scratch

I recently upgraded one of my homelab servers to FreeBSD 14-CURRENT, and I updated my Ansible scripts to build up some needed Jails for various services. I have never relied on iocage, exjail, or the like and have instead typically built my jails from source. With the latest FreeBSD -CURRENT a make installworld / make distribution into the new jail DESTDIR created a 1.3GB installation. This is a large surface area for running just a single service such as nginx, and so I visited the src.conf build options to get the base system of each jail down to 85MB.

The largest parts of the default 14-CURRENT userland install is by far /usr/lib/debug at 724MB. Clang/LLVM/LLDB is another large chunk. The full src.conf to build a base userland without many unneeded bits is included below:

WITHOUT_ACCT=true
WITHOUT_ACPI=true
WITHOUT_APM=true
WITHOUT_ATM=true
WITHOUT_AUTOFS=true
WITHOUT_BHYVE=true
WITHOUT_BLUETOOTH=true
WITHOUT_BOOT=true
WITHOUT_BOOTPARAMD=true
WITHOUT_BOOTPD=true
WITHOUT_BSDINSTALL=true
WITHOUT_BSNMP=true
WITHOUT_CLANG=true
WITHOUT_CXGBETOOL=true
WITHOUT_DEBUG_FILES=true
WITHOUT_DTRACE=true
WITHOUT_EFI=true
WITHOUT_EXAMPLES=true
WITHOUT_FINGER=true
WITHOUT_FLOPPY=true
WITHOUT_FREEBSD_UPDATE=true
WITHOUT_FTP=true
WITHOUT_GAMES=true
#WITHOUT_GH_BC=true
WITHOUT_GNU_DIFF=true
WITHOUT_HAST=true
WITHOUT_HTML=true
WITHOUT_HYPERV=true
WITHOUT_INCLUDES=true
WITHOUT_INSTALLLIB=true
WITHOUT_IPFILTER=true
WITHOUT_IPFW=true
WITHOUT_ISCSI=true
WITHOUT_JAIL=true
WITHOUT_LEGACY_CONSOLE=true
WITHOUT_LIB32=true
WITHOUT_LLDB=true
WITHOUT_LOCALES=true
WITHOUT_LPR=true
WITHOUT_MAN=true
WITHOUT_MANCOMPRESS=true
WITHOUT_MLX5TOOL=true
WITHOUT_NDIS=true
WITHOUT_NETCAT=true
WITHOUT_NIS=true
WITHOUT_NLS=true
WITHOUT_NTP=true
WITHOUT_OFED=ture
WITHOUT_OPENMP=true
WITHOUT_PF=true
WITHOUT_PMC=true
WITHOUT_PPP=true
WITHOUT_RADIUS_SUPPORT=true
WITHOUT_RBOOTD=true
WITHOUT_RESCUE=true
WITHOUT_ROUTED=true
WITHOUT_SENDMAIL=true
WITHOUT_SHAREDOCS=true
WITHOUT_SYSCONS=true
WITHOUT_TALK=true
WITHOUT_TESTS=true
WITHOUT_UNBOUND=true
WITHOUT_USB=true
WITHOUT_VT=true
WITHOUT_WIRELESS=true
WITHOUT_ZFS=true

Note that WITHOUT_GH_BC is broken for installworld in -CURRENT at present, so I've commented it out. nginx, isc-dhcpd, and other packages I install my jails add about 40MB each, and so I'm pretty happy with an 85MB base system for each jail.

FreeBSD has a long history of projects to provide a minimal system for embedded devices and other use cases, such as PicoBSD (deprecated -- FreeBSD 3 on a single floppy!), NanoBSD, mfsBSD, and more. Please see these projects for more robust techniques to further minimize. This post is meant purely to see how small I can get an installed jail userland from the output of a default make buildworld by just setting build variables with make installworld.