tag:blogger.com,1999:blog-1703739977686049052024-03-13T00:43:05.524-07:00Murray's FreeBSD NotesMurrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.comBlogger69125tag:blogger.com,1999:blog-170373997768604905.post-77999522063142856252023-10-17T16:15:00.001-07:002023-10-17T16:15:13.903-07:00Launching FreeBSD VMs in Azure with Bicep TemplatesI have followed <a href="https://www.daemonology.net">Colin Percival's</a> excellent work on getting FreeBSD images supported on the public cloud provider infrastructure since <a href="https://www.daemonology.net/blog/2010-12.html">2010</a>. I spent some time this weekend looking at the latest images and writing an Azure Bicep template to launch a FreeBSD VM.
Microsoft now has an <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/linux/freebsd-intro-on-azure">article about launching FreeBSD VM images</a> with their relatively excellent free azure documentation and training. That doc makes a great starting point, but it doesn't show how to find the latest FreeBSD images or use Bicep templates. As of this writing, <a href="https://portal.azure.com/#view/Microsoft_Azure_Marketplace/GalleryItemDetailsBladeNopdl/id/thefreebsdfoundation.freebsd-13_2/">FreeBSD 13.2</a> is the latest image in the Azure MarketPlace, and VMs can be launched in Azure at least half a dozen different ways:
<ul>
<li><b>az command-line tool</b></li>
<li>Start-AzVM PowerShell</li>
<li><b>Bicep template engine</b></li>
<li>ARM Templates</li>
<li>Terraform</li>
<li>Pulumi</li>
<li>etc..</li>
</ul>
<p>I'll show the az command-line needed to quickly launch a VM, but will focus mostly on Bicep templates, as this appears to be the current Microsoft-recommended Azure-specific template engine for automating Infrastructure as Code deployments on Azure. Before experimenting with this you need:</p>
<ul>
<li>An active Azure subscription.</li>
<li>A user with owner/contributor roles.</li>
<li>A resource group.</li>
</ul>
These can all easily be created at the <a href="http://portal.azure.com">Azure Portal</a> if this is your first time working with Azure.
<h2>Command Line VM Creation:</h2>
<p>The easiest method to create a new VM is simply to use the <tt>az</tt> command. I created a resource group called "FreeBSD" and then looked for the latest FreeBSD image names:</p>
<p>To find a list of publishers of FreeBSD images you can use something like:</p>
<pre># az vm image list-publishers --location westus --output table|grep thefreebsdfoundation
westus thefreebsdfoundation
</pre>
<p>To find "offers" and "skus" from this publisher:</p>
<pre>
# az vm image list-offers --location westus --publisher thefreebsdfoundation --output table
#westus freebsd-13_2
# az vm image list-skus --location westus --publisher thefreebsdfoundation --offer freebsd-13_2 --output table
#westus 13_2-release
#az vm image list \
# --location westus \
# --publisher thefreebsdfoundation \
# --offer freebsd-13_2 \
# --sku 13_2-release \
# --all --output table
#x64 freebsd-13_2 thefreebsdfoundation 13_2-release thefreebsdfoundation:freebsd-13_2:13_2-release:13.2.0 13.2.0
</pre>
<p>Once we have identified a specific FreeBSD image string, you can create a VM instance with:</p>
<tt>az vm create --name MyFreeBSD13 --resource-group FreeBSD --image thefreebsdfoundation:freebsd-13_2:13_2-release:13.2.0 --admin-username murray --generate-ssh-keys</tt>
<pre>
ResourceGroup PowerState PublicIpAddress Fqdns PrivateIpAddress MacAddress Location Zones
--------------- ------------ ----------------- ------- ------------------ ----------------- ---------- -------
FreeBSD VM running 172.191.154.200 10.0.0.4 00-0D-3A-8B-22-FD eastus
</pre>
<p>Behind the scenes, Azure is creating a number of resources on my behalf in addition to the VM instance:</p>
<ul>
<li>A virtual network (VNET)</li>
<li>A virtual network interface (VNIC)</li>
<li>A public IPv4 address</li>
<li>A network security group (stateful level 4 firewall rules)</li>
<li>A persistent disk for the OS.</li>
<li>The virtual machine</li>
</ul>
<p>All of these are created in my specified "FreeBSD" resource group, and I can ssh directly into the new VM, start installing packages or configuring it with Ansible, and otherwise prepare it for my use case. However, using a template engine can allow us to provide more configuration details about the VM to control costs and control how the resources are deployed.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixWjfIYLG7IZBeFbkNlGyyhZe94t-aPXdMjomOy2DXVUFtFkkyzG8tuFCd08pCRlgtlja3zV1BKQoXh_-eVSSnU14jFwMIxVu2QSnRTs43DgKC_buVIjCHOdwPa4f2Pfn89pVh8hsPNKy5evJkQTpFvGrtka20SJ0VvS0O8Atgei8LxDUkJYw1H6mVMrWe/s1600/Screenshot%202023-10-14%20at%201.53.49%E2%80%AFAM.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" data-original-height="704" data-original-width="1536" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixWjfIYLG7IZBeFbkNlGyyhZe94t-aPXdMjomOy2DXVUFtFkkyzG8tuFCd08pCRlgtlja3zV1BKQoXh_-eVSSnU14jFwMIxVu2QSnRTs43DgKC_buVIjCHOdwPa4f2Pfn89pVh8hsPNKy5evJkQTpFvGrtka20SJ0VvS0O8Atgei8LxDUkJYw1H6mVMrWe/s1600/Screenshot%202023-10-14%20at%201.53.49%E2%80%AFAM.png"/></a></div>
<h3>Windows Azure Linux Agent</h3>
<p>One thing to notice about the VM is that there are some Python 3.9 processes running by default. These are for the Windows Azure Linux Agent running from <tt>/usr/local/sbin/waagent</tt> and is outside the scope of this post.</p>
<h2>Bicep Automation</h2>
<p>The recommended way to author and deploy Bicep configurations is with Visual Studio Code and the Bicep extension. This extension allows you to visualize the resource graph that will be created by the config, handle autocompletion of long Azure resource names, and lets you deploy a configuration.</p>
<pre>
@description('Location for all resources.')
param location string = resourceGroup().location
@description('Prefix to use for VM names')
param vmNamePrefix string = 'BackendVM'
@description('Size of the virtual machines')
param vmSize string = 'Standard_D2s_v3'
// Unique DNS Name for the Public IP used to access the Virtual Machine.
@description('DNS Label Prefix for Public IP used by Azure to access the VM')
param dnsLabelPrefix string = toLower('FreeBSD-${uniqueString(resourceGroup().id)}')
var publicIPAddressName = 'FreeBSDPublicIP'
var virtualNetworkName = 'FreeBSDVnet'
var subnet1Name = 'FreeBSD-subnet-1'
var subnetRef1 = resourceId('Microsoft.Network/virtualNetworks/subnets', virtualNetworkName, subnet1Name)
var networkInterfaceName = 'nic'
//var numberOfInstances = 2
resource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-05-01' = {
name: virtualNetworkName
location: location
properties: {
addressSpace: {
addressPrefixes: [
'10.0.0.0/16'
]
}
subnets: [
{
name: subnet1Name
properties: {
addressPrefix: '10.0.0.0/24'
}
}
]
}
resource subnet1 'subnets' existing = {
name: subnet1Name
}
}
output subnet1ResourceId string = virtualNetwork::subnet1.id
// public IP Address
resource publicIP 'Microsoft.Network/publicIPAddresses@2020-06-01' = {
name: publicIPAddressName
location: location
properties: {
publicIPAllocationMethod: 'Dynamic'
publicIPAddressVersion: 'IPv4'
dnsSettings: {
domainNameLabel: dnsLabelPrefix
}
idleTimeoutInMinutes: 4
}
sku: {
name: 'Basic'
}
}
resource networkInterface1 'Microsoft.Network/networkInterfaces@2021-05-01' = {
name: '${networkInterfaceName}1'
location: location
properties: {
ipConfigurations: [
{
name: 'ipconfig1'
properties: {
privateIPAllocationMethod: 'Dynamic'
subnet: {
id: subnetRef1
}
publicIPAddress: {
id: publicIP.id
}
}
}
]
}
dependsOn: [
virtualNetwork
]
}
resource vm1 'Microsoft.Compute/virtualMachines@2021-11-01' = {
name: '${vmNamePrefix}1'
location: location
plan: {
name: '13_2-release'
product: 'freebsd-13_2'
publisher: 'thefreebsdfoundation'
}
properties: {
hardwareProfile: {
vmSize: vmSize
}
osProfile: {
computerName: '${vmNamePrefix}1'
adminUsername: 'murray'
adminPassword: 'XXXXXXX' // Use SSH keys instead, for testing with hard-coded password need 6+ chars, one lower, one upper, one special character
}
storageProfile: {
imageReference: {
publisher: 'thefreebsdfoundation'
offer: 'freebsd-13_2'
sku: '13_2-release'
version: '13.2.0'
}
osDisk: {
createOption: 'FromImage'
}
}
networkProfile: {
networkInterfaces: [
{
id: networkInterface1.id
}
]
}
// diagnosticsProfile: {
// bootDiagnostics: {
// enabled: true
// storageUri: storageAccount.properties.primaryEndpoints.blob
// }
// }
}
}
</pre>
<p>Thats it! The Azure docs cover myriad configuration parameters to customize all aspects of the network and VM deployment, but this basic template should get you started with FreeBSD in Azure.</p>
Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-24448751960683337192023-05-31T20:07:00.001-07:002023-05-31T20:13:33.271-07:00Triple Booting in 2023: FreeBSD, Windows, and Ubuntu Linux<p>Before this evening, I don't think I had setup a dual boot system in more than 20 years. In that time I've run mostly MacOS X, FreeBSD, and Linux desktops, and run Windows and some other Unix variants through Parallels on the Mac. However, I recently purchased a <a href="https://www.dell.com/en-us/shop/cty/precision-3260-compact/spd/precision-3260-workstation">Dell Precision 3260 Compact</a> workstation and mounted it under the desk in my home office. I wanted a compact machine to re-aquaint myself with modern Linux/FreeBSD Desktop environment, have a non-Virtualized copy of Windows, and a PCIe expansion slot for GPU, data acquisition tinkering, etc, and I think this fits the bill nicely. UEFI firmware makes this much easier than the last time I did this, but I still wanted to take a few notes in case I have to do this again (on my next PC laptop, most likely):</p>
<h2>Windows</h2>
<p>I bought the workstation without an OS, but I was able to download the Windows 10 installation media, write it to a bootable USB key, and then transfer an extra copy of a full Windows 10 license I have for an unneeded Parallels virtual machine. I opted to give Windows the first full SSD then split the second SSD between FreeBSD and Linux, since they will mount large home directories from my NAS over NFS. Installation was straightforward, I downloaded a few drivers from the Dell website, then did the free upgrade to Windows 11. I created a 550MB EFI System partition in the <a href="https://superuser.com/questions/1308324/create-efi-partition-before-installing-windows-10">Windows partition editor</a>, and then let Windows create 100MB, 16MB, and 465GB parititions for the rest of the install.</p>
<h2>FreeBSD</h2>
<p>Second up was the FreeBSD install. I booted from a FreeBSD 14 memory stick and performed an install. The installer doesn't make it easy to split a disk with a ZFS partition between multiple operating systems. Rather than fiddle with it all manually, I opted for a UFS filesystem where I could specify half the ssd for FreeBSD and leave the remaining parts empty for the Linux install. The installer created a 260MB EFI partition, 233GB UFS root partition, and 8GB for swap. After a reboot I could boot into FreeBSD by pressing F12 and startup and manually selecting the second SSD to boot from, but the default would still boot into the Windows Boot Manager and load Windows 11.</p>
<h2>Ubuntu Linux</h2>
<p>Finally, I installed Ubuntu 22 LTS from a downloaded memory disk. The installer installed <a href="https://www.gnu.org/software/grub/">GRUB 2</a> and on first reboot it presented a menu with items for Windows Boot Manager and Ubuntu Linux, but no FreeBSD! I logged into Ubuntu and edited the <tt>/etc/grub.d/40_custom</tt> file to add the following:
<pre>
menuentry "FreeBSD" {
set root=(hd1,1)
chainloader (${root})/efi/boot/BOOTx64.efi
}
</pre>
<p>Then ran the command <tt>sudo update-grub</tt>. The output shows that it finds the FreeBSD 14 partition but for some reason wasn't creating a menu item until I manually added the line above.</p>
<h2>Conclusion</h2>
<p>With this change, I now get a nice GRUB2 menu of FreeBSD, Linux, or Windows to boot into, just like I had with my noisy tower desktop computers with multiple hard disks of the late 1990s. This is a secondary PC to my primary Mac Pro desktop, but I am happy to get the flexibility to jump into other bare metal operating systems when needed.</p>
<p>Some things to follow up on:</p>
<ul><li>FreeBSD's install could make it easier to install FreeBSD on ZFS on only part of the disk/ssd, like it does for UFS by just asking you how much total space you want to occupy then handling all the other details.</li>
<li>Ubuntu's GRUB2 bootloader doesn't include a menu item for FreeBSD even though it was able to detect the FreeBSD partition. It would be more user friendly if the default config files built the FreeBSD menu item without requiring the 4 lines above to be added.</li>
<li>The steps above created three EFI system partitions of various sizes. This was easiest but wastes a bit of disk space.</li>
</ul>Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-52321534173664429642023-05-05T01:46:00.000-07:002023-05-05T01:46:26.265-07:00Svelte Jails From Scratch<p>I recently upgraded one of my <a href="http://freebsd.stokely.org/2023/05/new-nuc13anki7-13th-gen.html">homelab servers</a> to FreeBSD 14-CURRENT, and I updated my Ansible scripts to build up some needed Jails for various services. I have never relied on iocage, exjail, or the like and have instead typically built my jails from source. With the latest FreeBSD -CURRENT a <tt>make installworld</tt> / <tt>make distribution</tt> into the new jail DESTDIR created a 1.3GB installation. This is a large surface area for running just a single service such as nginx, and so I visited the <tt>src.conf</tt> build options to get the base system of each jail down to 85MB.</p>
<p>The largest parts of the default 14-CURRENT userland install is by far <tt>/usr/lib/debug</tt> at 724MB. Clang/LLVM/LLDB is another large chunk. The full <tt>src.conf</tt> to build a base userland without many unneeded bits is included below:</p>
<pre>
WITHOUT_ACCT=true
WITHOUT_ACPI=true
WITHOUT_APM=true
WITHOUT_ATM=true
WITHOUT_AUTOFS=true
WITHOUT_BHYVE=true
WITHOUT_BLUETOOTH=true
WITHOUT_BOOT=true
WITHOUT_BOOTPARAMD=true
WITHOUT_BOOTPD=true
WITHOUT_BSDINSTALL=true
WITHOUT_BSNMP=true
WITHOUT_CLANG=true
WITHOUT_CXGBETOOL=true
WITHOUT_DEBUG_FILES=true
WITHOUT_DTRACE=true
WITHOUT_EFI=true
WITHOUT_EXAMPLES=true
WITHOUT_FINGER=true
WITHOUT_FLOPPY=true
WITHOUT_FREEBSD_UPDATE=true
WITHOUT_FTP=true
WITHOUT_GAMES=true
#WITHOUT_GH_BC=true
WITHOUT_GNU_DIFF=true
WITHOUT_HAST=true
WITHOUT_HTML=true
WITHOUT_HYPERV=true
WITHOUT_INCLUDES=true
WITHOUT_INSTALLLIB=true
WITHOUT_IPFILTER=true
WITHOUT_IPFW=true
WITHOUT_ISCSI=true
WITHOUT_JAIL=true
WITHOUT_LEGACY_CONSOLE=true
WITHOUT_LIB32=true
WITHOUT_LLDB=true
WITHOUT_LOCALES=true
WITHOUT_LPR=true
WITHOUT_MAN=true
WITHOUT_MANCOMPRESS=true
WITHOUT_MLX5TOOL=true
WITHOUT_NDIS=true
WITHOUT_NETCAT=true
WITHOUT_NIS=true
WITHOUT_NLS=true
WITHOUT_NTP=true
WITHOUT_OFED=ture
WITHOUT_OPENMP=true
WITHOUT_PF=true
WITHOUT_PMC=true
WITHOUT_PPP=true
WITHOUT_RADIUS_SUPPORT=true
WITHOUT_RBOOTD=true
WITHOUT_RESCUE=true
WITHOUT_ROUTED=true
WITHOUT_SENDMAIL=true
WITHOUT_SHAREDOCS=true
WITHOUT_SYSCONS=true
WITHOUT_TALK=true
WITHOUT_TESTS=true
WITHOUT_UNBOUND=true
WITHOUT_USB=true
WITHOUT_VT=true
WITHOUT_WIRELESS=true
WITHOUT_ZFS=true
</pre>
<p>Note that <tt>WITHOUT_GH_BC</tt> is broken for installworld in -CURRENT at present, so I've commented it out. <tt>nginx</tt>, <tt>isc-dhcpd</tt>, and other packages I install my jails add about 40MB each, and so I'm pretty happy with an 85MB base system for each jail.</p>
<p>FreeBSD has a long history of projects to provide a minimal system for embedded devices and other use cases, such as <a href="https://people.freebsd.org/~picobsd/old/picobsd.html">PicoBSD</a> (deprecated -- FreeBSD 3 on a single floppy!), <a href="https://docs.freebsd.org/en/articles/nanobsd/">NanoBSD</a>, <a href="https://github.com/mmatuska/mfsbsd">mfsBSD</a>, and more. Please see these projects for more robust techniques to further minimize. This post is meant purely to see how small I can get an installed jail userland from the output of a default <tt>make buildworld</tt> by just setting build variables with <tt>make installworld</tt>.</p>Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-61900434516033330382023-05-01T01:38:00.003-07:002023-05-05T10:57:39.867-07:00New NUC13ANKi7 13th gen<p>I'd like to replace some of my older 2015-2016 era FreeBSD NUCs with something with more DRAM to support ZFS and more Jails, and a faster CPU to speed up my builds. After some research I settled on the NUC13ANKi7. This NUC fits a lot into the smaller short height form factor:</p>
<ul>
<li>Intel Core i7-1360P
<ul>
<li>12 cores (4 performance, 8 efficiency), 16 threads)</li>
<li>Performance cores turbo boost up to 5.0 GHz</li>
<li>Raptor Lake architecture</li>
</ul></li>
<li>64GB DDR4 DRAM.</li>
<li>1TB PCIe x4 Gen4 NVMe</li>
<li>2.5GB Ethernet, Wifi6E</li>
<li>UEFI with support for HTTP Boot.</li>
</ul>
<p>The price from SimplyNUC is $1,189 plus tax and shipping. I would have preferred DDR5 DIMMs but otherwise was pretty happy with this, especially the smaller NUC form factor.</p>
<p>The machine builds and runs FreeBSD 13.2 great (<a href="https://gist.github.com/murraystokely/43ef2e4a332c4cd17a96af7d9b1f8a9b">see dmesg gist</a>).</p>
<h2>Buildworld Times</h2>
To test the suitability of this new machine as a build server, I ran 270 iterations of make buildworld over a week.
<ul>
<li>30 -j parallelism options (1 to 30)</li>
<li>Three source and /usr/obj configurations:
<ul>
<li><tt>/usr/src</tt> and <tt>/usr/obj</tt> on local SSD</li>
<li><tt>/usr/src</tt> on SSD and <tt>/usr/obj</tt> on tmpfs</li>
<li><tt>/usr/src</tt> and <tt>/usr/obj</tt> on tmpfs.</li>
</ul></li>
<li>Three iterations of each test.</li>
</ul>
<p>Median result of the three runs for the default ZFS /usr/src and /usr/obj runs are included in the plot below:</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRSch9yMxbHE4UyqZUDeSkz3fN4el1p2lMH0Eq_t9NMKAm0vrE5SppZhBYJGKQUiCKPWDU7XIJM3WaQGUG6-QTEwqAVG5dHYj6oXCsgSpewXOdbqrSizrYsw0xIt2qN4ICZH2d6mUqjxdnthCYoygLsl1uRoszRRjzgCST62FtKFTA8OohksctbkqaKw/s1274/000017.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="600" data-original-height="787" data-original-width="1274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRSch9yMxbHE4UyqZUDeSkz3fN4el1p2lMH0Eq_t9NMKAm0vrE5SppZhBYJGKQUiCKPWDU7XIJM3WaQGUG6-QTEwqAVG5dHYj6oXCsgSpewXOdbqrSizrYsw0xIt2qN4ICZH2d6mUqjxdnthCYoygLsl1uRoszRRjzgCST62FtKFTA8OohksctbkqaKw/s600/000017.png"/></a></div>
<p>The fastest buildtimes occurred in under 26 minutes with -j21, but the performance of anything between -j16 and -j30 was within 1.3% of that elapsed build time. Moving src and obj to tmpfs led to less than a 4.1% delta in elapsed time.</p>
<h2>SSD Performance</h2>
<p><tt>diskinfo -t nvd0<tt> shows 3.9-4.35 MB/s transfer rates to the local SSD.</p>
<h3>diskinfo -t nvd0</h3>
<pre>nvd0
512 # sectorsize
1000204886016 # mediasize in bytes (932G)
1953525168 # mediasize in sectors
0 # stripesize
0 # stripeoffset
PNY CS2140 1TB SSD # Disk descr.
PNY2201220106010B3A8 # Disk ident.
nvme0 # Attachment
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Seek times:
Full stroke: 250 iter in 0.004330 sec = 0.017 msec
Half stroke: 250 iter in 0.019046 sec = 0.076 msec
Quarter stroke: 500 iter in 0.013305 sec = 0.027 msec
Short forward: 400 iter in 0.005383 sec = 0.013 msec
Short backward: 400 iter in 0.005009 sec = 0.013 msec
Seq outer: 2048 iter in 0.045332 sec = 0.022 msec
Seq inner: 2048 iter in 0.029877 sec = 0.015 msec
Transfer rates:
outside: 102400 kbytes in 0.025866 sec = 3958865 kbytes/sec
middle: 102400 kbytes in 0.023508 sec = 4355964 kbytes/sec
inside: 102400 kbytes in 0.024016 sec = 4263824 kbytes/sec
</pre>
<h3>IOZone</h3>
I also used IOZone benchmark to quickly gather some SSD stats with 4k reads:
<table>
<tr><th>Metric</th><th>Output in KBytes/sec</th></tr>
<tr><th>Initial write:</th><td>1160931.00</td></tr>
<tr><th>Rewrite</th><td>1223939.88 </td></tr>
<tr><th>Read</th><td>1900560.25 </td></tr>
<tr><th>Re-read</th><td>3034837.25 </td></tr>
<tr><th>Reverse Read</th><td>1859190.25 </td></tr>
<tr><th>Stride read</th><td>568050.94 </td></tr>
<tr><th>Random read</th><td>143096.12 </td></tr>
<tr><th>Mixed workload</th><td>290691.06 </td></tr>
<tr><th>Random write</th><td>94226.63 </td></tr>
<tr><th>Pwrite</th><td>1184217.75 </td></tr>
<tr><th>Pread</th><td>3080741.50 </td></tr>
<tr><th>Fwrite</th><td>1211297.38 </td></tr>
<tr><th>Fread</th><td>1776154.00 </td></tr>
</table>
<h2>UEFI HTTP Boot</h2>
<p>One thing I wasn't expecting with this NUC is that it supports HTTP Boot and a UEFI shell. It seems more finicky than HTTPBoot on a Dell, and is possibly only looking at HTTPS URLs. Will try to follow up with another post in more detail about HTTP Boot with FreeBSD.</p>
Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-75518455268707284912016-10-24T20:27:00.000-07:002016-10-24T20:27:43.207-07:00FreeBSD on Intel NUCs<p>I've been away from FreeBSD for a few years but I wanted some more functionality on my home network that I was able to configure with my Synology NAS and router. Specifically, I wanted:</p>
<ul>
<li>a configurable caching name server that would serve up authoritative private names on my LAN and also validates responses with DNSSEC.
<li>a more configurable DHCP server so I could make the server assign specific IPs to specific MAC addresses.
<li>more compute power for transcoding videos for Plex.
</ul>
<p>Running FreeBSD 11 on an Intel NUC seemed like an ideal solution to keep my closet tidy. As of this week, $406.63 on Amazon buys a last generation i3 Intel NUC mini PC (NUC5I3RYH), with 8GB of RAM and 128GB of SSD storage. This was the first model I tried since I found reports of others using this with FreeBSD online, but I was also able to get it working on the newer generation i5 based NUC6i5SYK with 16GB of RAM and 256GB of SSD. The major issue with these NUCs is that the Intel wireless driver is not supported in FreeBSD. I am not doing anything graphical with these boxes so I don't know how well the graphics work, but they are great little network compute nodes.</p>
<center>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQFo_CLblPARp-KYuZW-FxD5dl84o1vr126hWcAO7a2NEhbYKlOvJJuKVmyR7GIQuqKq6-RVJe2U1axUhSG0bwLOUgscpgoVk_6ufmNg9fJSIpSWWtXc3wqk_elqYp7EhCHdHAg-VgVCIm/s1600/20161010_223726.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQFo_CLblPARp-KYuZW-FxD5dl84o1vr126hWcAO7a2NEhbYKlOvJJuKVmyR7GIQuqKq6-RVJe2U1axUhSG0bwLOUgscpgoVk_6ufmNg9fJSIpSWWtXc3wqk_elqYp7EhCHdHAg-VgVCIm/s320/20161010_223726.jpg" width="320" height="180" /></a>
</center>
<h2>Installation</h2>
<p>I downloaded the FreeBSD 11 memory stick images, and was pleased to see that the device booted fine off the memory stick without any BIOS configuration required. However, my installation failed trying to mount root ("Mounting from ufs:/dev/ufs/FreeBSD_Install failed with error 19."). Installation from an external USB DVD drive and over the network with PXE both proved more successful at getting me into <tt>bsdinstaller</tt> to complete the installation.</p>
<p>I partitioned the 128GB SSD device with 8GB of swap and the rest for the root partition (UFS, Journaled and Soft Updates). After installation I edited <tt>/etc/fstab</tt> to add a <tt>tmpfs(5)</tt> mount for <tt>/tmp</tt>. The dmesg output for this host is available in a <a href="https://gist.github.com/murraystokely/13f8eec8ef1c1e79b8de32ef0b8de03b">Gist on Github</a>.</p>
<p>Warren Block's article on <a href="http://www.wonkity.com/~wblock/docs/html/ssd.html">SSD on FreeBSD</a> and the various chapters of the FreeBSD Handbook were helpful. There were a couple of tools that were also useful in probing the performance of the SSD with my FreeBSD workload:</p>
<ul>
<li>The <tt>smartctl</tt> tool in the <tt>sysutils/smartmontools</tt> package allows one to read detailed diagnostic information from the SSD, including wear patterns.</li>
<li>The basic benchmark built into <a href="https://www.freebsd.org/cgi/man.cgi?query=diskinfo&sektion=8">diskinfo -t</a> reports that the SSD is transferring 503-510MB/second.</li>
</ul>
But how well does it perform in practice?
<h2>Rough Benchmarks</h2>
<p>This post isn't meant to report a comprehensive suite of <a href="https://wiki.freebsd.org/BenchmarkAdvice">FreeBSD benchmarks</a>, but I did run some basic tests to understand how suitable these low power NUCs perform in practice. To start with, I downloaded the 11-stable source from Subversion and measured the build times to understand performance of the new system. All builds were done with a minimal 2 line <tt>make.conf</tt>:
<pre>
MALLOC_PRODUCTION=yes
CPUTYPE?=core2
</pre>
<h3>Build Speed</h3>
<table>
<tr><th>Build Command</th><th>Environment</th><th>Real Times</th></tr>
<tr><td>make -j4 buildkernel</td><td>/usr/src and /usr/obj on SSD</td><td>10.06 minutes</td></tr>
<tr><td>make -j4 buildkernel</td><td>/usr/src on SSD, /usr/obj on tmpfs</td><td>9.65 minutes</td></tr>
<tr><td>make -j4 buildworld</td><td>/usr/src and /usr/obj on SSD</td><td>1.27 hours</td></tr>
<tr><td>make buildworld</td><td>/usr/src and /urs/obj on SSD</td><td>3.76 hours</td>
</table>
<h3>Bonnie</h3>
In addition to the build times, I also wanted to look more directly at the performance reading from flash and reading from the NFS mounted home directories on my 4-drive NAS. I first tried Bonnie++, but then ran into a <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=57696">13-year old bug</a> in the NFS client of FreeBSD. After switching to Bonnie, I was able to gather some reasonable numbers. I had to use really large file sizes for the random write test to eliminate most of the caching that was artificially inflating the results. For those that haven't seen it, <a href="http://www.brendangregg.com/ActiveBenchmarking/bonnie++.html">Brendan Gregg's excellent blog post</a> highlights some of the issues of file system benchmarks like Bonnie.</p>
<hr>
<center>
<table>
<caption>Average of 3 bonnie runs with 40GB block size</caption>
<tr><th rowspan="2">Configuration</th><th colspan="2">Random I/O</th><th colspan="2">Block Input</th><th colspan="2">Block Output</th></tr>
</tr>
<tr><th>Seeks/Sec</th><th>CPU Utilization</th><th>Reads/sec</th><th>CPU Utilization</th><th>Writes/sec</th><th>CPU Utilization</th></tr>
<tr><td>NFS</td><td>99.2</td><td>0.9</td><td>106505</td><td>4.8</td><td>89966</td><td>7.5</td></tr>
<tr><td>SSD</td><td>8809</td><td>13.5</td><td>538671</td><td>25.33</td><td>160917</td><td>11.3</td></tr>
</table>
</center>
<hr>
<p>The block input rates from my bonnie benchmarks on the SSD were within 5% of the value provided by the much quick and dirtier <tt>diskinfo -t</tt> test.</p>
<p>Running Bonnie with less than 40GB file size yielded unreliable benchmarks due to caching at the VM layer. The following boxplot shows the random seek performance during 3 runs each at 24, 32, and 40GB file sizes. Performance starts to even off at this level but with smaller file sizes the reported random seek performance is much higher.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggVolUp0mEMI_o3WSaQ7hIsU-2aKGoCxqoiIwxJRH18TyqS7n2paG2Tixgv5mMVpYeqHxSK3-D6YlgfWrrFGbSXvqR5nTbQE0lRBiK1deGl1pHL2KVaXJPgUkVak6M2BRcbjHz80V3tXtH/s1600/nuc-randomseeks.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggVolUp0mEMI_o3WSaQ7hIsU-2aKGoCxqoiIwxJRH18TyqS7n2paG2Tixgv5mMVpYeqHxSK3-D6YlgfWrrFGbSXvqR5nTbQE0lRBiK1deGl1pHL2KVaXJPgUkVak6M2BRcbjHz80V3tXtH/s320/nuc-randomseeks.png" width="320" height="221" /></a></div>
<h2>Open Issues</h2>
<p>As mentioned earlier, I liked the performance I got with running FreeBSD on a 2015-era i3 NUC5I3RYH so much that I bought a newer, more powerful second device for my network. The 2016-era i5 NUC 6i5SYK is also running great. There are just a few minor issues I've encountered so far:</p>
<ul>
<li>There is no FreeBSD driver for the Intel Wireless chip included with this NUC. Code for other platforms exists but has not been ported to FreeBSD.</li>
<li>The memory stick booting issue described in the installation section. It is not clear if it didn't like my USB stick for some reason, or the port I was plugging into, or if additional boot parameters would have solved the issue. Documentation and/or code needs to be updated to make this clearer.</li>
<li>Similarly, the PXE Install instructions were a bit scattered. The PXE section of the Handbook isn't specifically targetting new manual installations into bsdinstall. There are a few extra things you can run into that aren't documented well or could be streamlined.</li>
<li>Graphics / X11 are outside of the scope of my needs. The NUCs have VESA mounts so you can easily tuck them behind an LCD monitor, but it is not clear to me how well they perform in that role.</li>
</ul>
Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-9878439875188452862015-01-07T15:22:00.000-08:002015-01-07T15:22:51.404-08:00AsiaBSDCon 2014 Videos Posted (6 years of BSDConferences on YouTube)
<p>Sato-san has once created a <a href="https://www.youtube.com/playlist?list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">playlist</a> of videos from <a href="http://www.asiabsdcon.org">AsiaBSDCon</a>. There were 20 videos from the conference held March 15-16, 2014 and papers can be found <a href="http://2014.asiabsdcon.org/papers/index.html">here</a>. Congrats to the organizers for running another successful conference in Tokyo. A full list of videos is included below. <a href="http://freebsd.stokely.org/2008/12/new-channel-on-youtube-for-bsd.html">Six years ago</a> when I first created this channel videos longer than 10 minutes couldn't normally be uploaded to YouTube and we had to create a special partner channel for the content. It is great to see how the availability of technical video content about FreeBSD has grown in the last six years.</p>
<iframe width="560" height="315" src="//www.youtube.com/embed/videoseries?list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL" frameborder="0" allowfullscreen></iframe>
<ul>
<li><a href="https://www.youtube.com/watch?v=8T9Rh-46jhI&index=1&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Matthew Ahrens, OpenZFS ensures the continued excellence of ZFS on FreeBSD, Linux, and illumos</a></li>
<li><a href="https://www.youtube.com/watch?v=o2dmreSy76Q&index=2&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Eric Allman, Bambi Meets Godzilla: They Elope - Open Source Meets the Commercial World</a></li>
<li><a href="https://www.youtube.com/watch?v=4sZZN8Szh14&index=3&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Scott Long, Modifying the FreeBSD kernel Netflix streaming servers</a></li>
<li><a href="https://www.youtube.com/watch?v=E04LxKiu79I&index=4&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Dr. Marshall Kirk McKusick, An Overview of Security in the FreeBSD Kernel</a></li>
<li><a href="https://www.youtube.com/watch?v=fWgbBUPMsVw&index=5&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Luba Tang, Bold, fast optimizing linker for BSD (P1A, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=NLqDAclXMMU&index=6&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">David Chisnall, The Future of LLVM in the FreeBSD Toolchain (P2A, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=w-0PlAVSg5U&index=7&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Kris Moore, Snapshots, Replication, and Boot Environments (P3A, AsiaBSDCon 2014</a></li>
<li><a href="https://www.youtube.com/watch?v=z5apZFFvx4k&index=8&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Dru Lavigne, ZFS for the Masses: Management Tools Provided by the PC BSD and FreeNAS Projects (P4A)</a></li>
<li><a href="https://www.youtube.com/watch?v=iwF82aep-l8&index=9&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">John Hixson, Introduction to FreeNAS development (P5A, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=wptkUxJSNMY&index=10&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Peter Grehan, Nested Paging in Bhyve (P6A, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=2KLXcyLZ_RE&index=11&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Zbigniew Bodek, Transparent Superpages for FreeBSD on ARM (P8A, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=nb8jB5x0OX4&index=12&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Martin Matuška , Deploying FreeBSD systems with Foreman and mfsBSD (P7A, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=6eKMLuzsTbY&index=13&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">James P. Brown, Analysys of BSD Associate Exam Results (P10A, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=rjNg1eQ7uAk&index=14&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Michael Dexter, Visualizing Unix: Graphing bhyve, ZFS and PF with Graphite (P1B, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=cgBh0iC9WhM&index=15&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Mindaugas Rasiukevicius, NPF - progress and perspective (P2B, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=nW8iHgOL9y4&index=16&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Luigi Rizzo, Netmap as a core networking technology (P3B, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=ufeEP_hzFN0&index=17&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Reyk Floeter, VXLAN(4) and Cloud based networking with OpenBSD (P5B, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=ApruZrU5fVs&index=18&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Masanobu Saitoh, Developing CPE Routers based on NetBSD: Fifteen Years of SEIL (P6B)</a></li>
<li><a href="https://www.youtube.com/watch?v=vh-TjLUj6os&index=19&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Pierre Pronchery, Guillaume Lasmayous, Carve your NetBSD (P8B, AsiaBSDCon 2014)</a></li>
<li><a href="https://www.youtube.com/watch?v=gOPfRQgTjNo&index=20&list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL">Jos Jansen, Adapting OSX to the enterprise (P9B, AsiaBSDCon 2014)</a></li>
</ul>Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-3386475832170090592013-11-21T23:36:00.000-08:002013-11-21T23:36:56.721-08:00AsiaBSDCon 2013 Videos Posted to YouTube<p>Sato-san has created a <a href="https://www.youtube.com/playlist?list=PLnTFqpZk5ebB1Phtixwfy46WS6LTAsuLI">playlist</a> of 15 videos from <a href="http://2013.asiabsdcon.org/">AsiaBSDCon 2013</a>. Congrats to the organizers for running another successful conference in Tokyo. Some of the FreeBSD-related videos are listed below:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=1NC0lQMWHoc&list=PLnTFqpZk5ebB1Phtixwfy46WS6LTAsuLI&index=2">John Hixson: FreeNAS Plugins everything you ever wanted to know (P1A, AsiaBSDCon 2013)</a></li>
<li><a href="https://www.youtube.com/watch?v=C7i_PJqqHlU&list=PLnTFqpZk5ebB1Phtixwfy46WS6LTAsuLI&index=6">Kris Moore: Automating the deployment of FreeBSD & PC BSD systems (P3A, AsiaBSDCon 2013)</a></li>
<li><a href="https://www.youtube.com/watch?v=8DoApGiY62w&list=PLnTFqpZk5ebB1Phtixwfy46WS6LTAsuLI&index=12">Davide Italiano: Calloutng: a new infrastructure for timer facilities in the FreeBSD kernel (P6A)</a></li>
<li><a href="https://www.youtube.com/watch?v=fyKNMJsZACM&list=PLnTFqpZk5ebB1Phtixwfy46WS6LTAsuLI&index=8">Michael Dexter: Hands on bhyve, the BSD Hypervisor (P4A, AsiaBSDCon 2013)</a></li>
<li><a href="https://www.youtube.com/watch?v=rT0mbX6r-vI&list=PLnTFqpZk5ebB1Phtixwfy46WS6LTAsuLI&index=10">Takuya ASADA: Implements BIOS emulation support for BHyVe: A BSD Hypervisor (P5A, AsiaBSDCon 2013)</a></li>
</ul>
<p>There were also quite a few <a href="http://undeadly.org/cgi?action=article&sid=20131122040844">OpenBSD talks</a> this year. I had a great time at AsiaBSDCon several years back and hope to make it back again some day.</p>
<iframe width="560" height="315" src="//www.youtube.com/embed/videoseries?list=PLnTFqpZk5ebB1Phtixwfy46WS6LTAsuLI" frameborder="0" allowfullscreen></iframe>Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-80458733002402427422013-06-28T22:34:00.000-07:002013-06-28T22:34:40.943-07:00Trip Report from USENIX ATC 2013<p>I spent half of the week at USENIX ATC in San Jose. I previously attended in 2000, 2001, 2002, and 2004, and I have been to other more academic USENIX conferences in the intervening years such as <a href="https://www.usenix.org/legacy/events/fast11/index.html">FAST</a> and <a href="http://googleresearch.blogspot.com/2010/10/google-at-usenix-symposium-on-operating.html">OSDI</a>, but I have not made it back to Annual Tech in nearly a decade.</p>
<p>The conference is very familiar but has also definitely changed since '04 (no more terminal rooms and the BoF board was nearly empty!) I was very happy with the caliber of the accepted papers in the main conferences as well as in many of the workshops of <a href="https://www.usenix.org/conference/fcw13">Federated Conferences Week</a> (HotStorage, HotCloud, etc.). There is less industry and open-source participation now, but still a variety of really interesting talks about storage, networking, operating systems, virtualization, and more from academia and (<em>a smaller subset of</em>) industry.</p>
<p>As I've <a href="http://freebsd.stokely.org/2008/12/open-cfps-for-usenix-oscon-and-bsd.html">previously noted on this blog</a>, I think the BSD conferences are great, but that it is very important for the FreeBSD community to also present work at some of the broader open-source and academic systems conferences. I would be much more likely to attend <a href="http://2013.eurobsdcon.org/">EuroBSDCon</a> if it were held adjacent to <a href="http://eurosys2013.tudos.org/">EuroSys</a> or <a href="https://fosdem.org/2013/">FOSDEM</a>, for example. And would be more likely to attend a U.S.-based BSD conference if it were held adjacent to a USENIX or <a href="http://strataconf.com/">O'Reilly Strata</a> event.</p>
<p>On Wednesday my team presented one of our main projects of last year, <a href="http://research.google.com/pubs/pub41179.html">Janus: Optimal Flash Provisioning for Cloud Storage Workloads</a>. This work describes a method for automatically segregating hot and cold storage workloads in a large distributed filesystem, formulates an optimization problem to match the available flash to different workloads in such a way as to maximize the total reads going to flash, and then places that hot data on the distributed flash devices instead of distributed disk devices.</p>
<p>There were a number of other really interesting talks about flash, virtualization, and distributed storage systems, but I wanted to highlight two short-papers that I think would most appeal to the FreeBSD audience here:</p>
<ul>
<li><a href="https://www.usenix.org/conference/atc13/practical-and-effective-sandboxing-non-root-users">Practical and Effective Sandboxing for Non-root Users</a>, <em>Taesoo Kim and Nickolai Zeldovich, MIT CSAIL</em><br>
This was a nice practical short paper about interposing system calls, using unionfs in a clever way, and taking some revision control ideas for a nice little tool.</li>
<li><a href="https://www.usenix.org/conference/atc13/packetdrill-scriptable-network-stack-testing-sockets-packets">packetdrill: Scriptable Network Stack Testing, from Sockets to Packets</a>, <em>Neal Cardwell, Yuchung Cheng, Lawrence Brakmo, Matt Mathis, Barath Raghavan, Nandita Dukkipati, Hsiao-keng Jerry Chu, Andreas Terzis, and Tom Herbert, Google</em><br>
Another practical short paper about a portable tool, which works on FreeBSD or Linux, that enables testing the correctness and performance of entire TCP/UDP/IP network stack implementations, from the system call layer to the hardware network interface, for both IPv4 and IPv6. This tool was instrumental in identifying 10 bugs in the Linux network stack and enabling the development of three new features: TCP—Early Retransmit, <a href="http://freebsd.stokely.org/2012/02/updated-tcp-proposals-and-freebsd.html">Fast Open</a>, and Loss Probes.</li>
</ul>
<p>I'm not sure if I'll go to FAST, or USENIX ATC, or both next year, but it's likely I'll attend at least one. What other industry conferences outside of the BSDCan/EuroBSDCon circuit does the FreeBSD community congregate at these days? For folks that have been in industry 10+ years, do you go to more or less industry conferences now than in the past?</p>Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com1tag:blogger.com,1999:blog-170373997768604905.post-62453139476808294032012-02-04T15:07:00.001-08:002012-02-04T16:05:42.008-08:00Updated TCP Proposals and FreeBSDThere are a number of proposals for improving TCP performance coming out of Google that have some implications for FreeBSD. These proposals have taken the form of a group of IETF proposals, RFCs, patches to the Linux kernel, and research publications. A nice summary of the different initiatives is available from <a href="http://googlecode.blogspot.com/2012/01/lets-make-tcp-faster.html">Lets Make TCP Faster</a> on the <a href="http://googlecode.blogspot.com/">Google Code Blog</a>.<br /><br /><a href="http://research.google.com/pubs/pub37517.html">TCP Fast Open</a> by Radhakrishnan, Cheng, Chu, Jain, and Raghavan is based on the observation that modern web services are dominated by TCP flows so short that they terminate a few round trips after handshaking. This means that the 3-way TCP handshake is a significant source of latency for such flows, and they describe a new mechanism for secure data exchange during the initial handshake to reduce some of the round-trip network transmission and associated latency for such short TCP transfers. This work shares many goals and challenges with <a href="http://www.kohala.com/start/ttcp.html">T/TCP</a>, which was previously in FreeBSD but suffered from some security vulnerabilities.<br /><br /><a href="https://plus.google.com/102219142216209294629">David Malone</a> posted some thoughts on my <a href="https://plus.google.com/108173651282419595823/posts/HpamdjE2UTN">Google+ post</a> about how FreeBSD could implement the various changes. Maybe we could have some <a href="http://code.google.com/soc">Summer of Code</a> students work in this area this summer?Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com1tag:blogger.com,1999:blog-170373997768604905.post-68015832082971885552011-08-19T13:47:00.000-07:002011-08-21T21:22:04.251-07:00Two New Videos: SuperPages and NanoBSDThanks to Kirk McKusick, I'm happy to announce two new fully edited high quality videos from <a href="http://www.bsdcan.org/2011">BSDCan 2011</a> in the <a href="http://www.youtube.com/bsdconferences">BSD Conferences</a> YouTube channel. I've also created a new <a href="http://www.youtube.com/playlist?list=PL95329F8B660F86C5">playlist</a> for the BSDCan 2011</a> videos.
<br />
<br />The first talk is <a href="http://www.bsdcan.org/2011/schedule/events/219.en.html">"Superpages in FreeBSD"</a> by McKusick, and it describes the addition of superpage support to the FreeBSD 8 kernel on the Intel PC architecture. Superpages aggregate together standard-sized hardware pages into much larger "superpages". Each superpage requires only one entry in the page table replacing the numerous entries used by the standard-sized hardware pages.
<br />
<br /><iframe width="560" height="345" src="http://www.youtube.com/embed/0wIxny-n_Mg" frameborder="0" allowfullscreen></iframe>
<br />
<br />The second talk is <a href="http://www.bsdcan.org/2011/schedule/events/239.en.html">"Updates from NanoBSD: FreeNAS drives NanoBSD development"</a> from Warner Losh, and it describes the basics of NanoBSD and how FreeNAS moved over to NanoBSD.
<br />
<br /><iframe width="560" height="345" src="http://www.youtube.com/embed/5qCaOMQ3ZnQ" frameborder="0" allowfullscreen></iframe>
<br />
<br />We now have 108 high-quality videos in the <a href="http://www.youtube.com/bsdconferences">BSD Conferences</a> channel. These videos have been watched in aggregate over 400,000 times, and our most popular video remains <a href="http://freebsd.stokely.org/2009/01/freebsd-kernel-internals-lecture-posted.html">McKusick's FreeBSD Kernel Internals Lecture</a>.
<br />
<br />As a reminder, this channel was setup specifically for the BSD technical community and does not have the standard limitations on video size for other types of YouTube uploads. If you have additional video content from a conference, presentation, or class about BSD Unix please get in touch and I'd be happy to help you publish the content here.
<br />Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com2tag:blogger.com,1999:blog-170373997768604905.post-34176457288810235322011-08-18T21:20:00.000-07:002011-08-18T21:30:14.489-07:00Luigi Talks Netmap on Google Tech Talks ChannelLast week, Luigi Rizzo visited Google and gave a talk on high-speed networking with <a href="http://info.iet.unipi.it/~luigi/netmap/">Netmap</a>.
<br />
<br /><iframe width="560" height="345" src="http://www.youtube.com/embed/SPtoXNW9yEQ" frameborder="0" allowfullscreen></iframe>
<br />
<br />This was Luigi's second talk at Google and the third talk about FreeBSD in the <a href="http://www.youtube.com/user/GoogleTechTalks">Google Tech Talks</a> YouTube channel. We also have more than 100 videos available in the <a href="http://www.youtube.com/bsdconferences">BSD Conferences</a> channel.Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-11679063356558802162010-10-24T21:31:00.000-07:002010-10-24T22:26:48.633-07:00FreeBSD at GSoC Mentor SummitAs in <a href="http://freebsd.stokely.org/2008/10/strong-bsd-showing-at-gsoc-mentor.html">previous</a> <a href="http://freebsd.stokely.org/2007/10/summer-of-code-mentor-summit.html">years</a>, Google held a "Mentor Summit" to bring together representatives from the open source organizations that participated in the <A href="http://code.google.com/soc/">Google Summer of Code</a> to share experiences of what worked, what didn't, and generally learn from each other about shepherding students through the program. The mentor summit is always run <a href="http://en.wikipedia.org/wiki/Unconference">Unconference-style</a> and it is a great opportunity to meet, learn, and socialize with the many other open source organizations.<br /><br />In addition to several hours of face to face FreeBSD-related catch-up with Brooks Davis over pizza and beer, I particularly enjoyed catching up with old colleagues and learning about the current state of a variety of other open source projects I use such as <a href="http://cran.r-project.org/">R</a>, <a href="http://www.boost.org/">Boost</a>, <a href="http://www.ntp.org/">NTP</a>, and <a href="http://code.google.com/p/ganeti/">Ganeti</a>.<br /><br />This weekend Brooks and I were the only FreeBSD representatives. Given that I'm local and Google sponsors the travel of 2 representatives from each open source organization it's quite unfortunate we couldn't get another FreeBSD mentor here this year. I would strongly encourage some of the other mentors that have never participated in this forum to volunteer to represent FreeBSD next year. This program has funded approximately <a href="http://google-opensource.blogspot.com/2010/09/freebsds-summer-highlights.html">117 students</a> to work on FreeBSD over the past 5 years and the mentor summit is best way I know of to improve the experience for students and open source projects next year.<br /><br />Thanks again to all the FreeBSD mentors that worked with students this summer and hope to see some of you at the post-GSoC Mentor Summit next year...Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-56229284689592331772010-09-15T10:25:00.000-07:002010-09-15T10:34:33.147-07:00FreeBSD Summer of Code Students Highlighted on Google BlogAs in <a href="http://google-opensource.blogspot.com/2008/10/freebsds-fourth-google-summer-of-code.html">previous</a> <a href="http://googlesummerofcode.blogspot.com/2007/09/updates-from-freebsd.html">years</a>, I've posted a <a href="http://google-opensource.blogspot.com/2010/09/freebsds-summer-highlights.html">summary of FreeBSD Project participation</a> in <a href="http://code.google.com/soc">Google Summer of Code</a> on the <a href="http://google-opensource.blogspot.com/2010/09/freebsds-summer-highlights.html">Google Open Source Blog</a>.<br /><br />By my count we have now mentored at least 117 students on FreeBSD development through this program. As in previous years it was tough to identify a few student projects to highlight given how much cool work is going on here. My list is certainly not complete but at least a few other people mentioned that Efstratios Karatzas, Zheng Liu, and David Forsythe had done a lot of excellent work this summer. Hats off to them, all the students and mentors this summer, and Brooks and Robert for serving as administrators of this whole thing for us.Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-67987782970087925422010-08-20T08:08:00.000-07:002010-08-20T08:22:03.544-07:00MeetBSD 2010 Videos in HD on YouTubeThanks to Tomasz Dudzisz we now have 14 videos from <a href="http://www.meetbsd.org">MeetBSD 2010</a> in Poland. Tomasz has created a <a href="http://www.youtube.com/my_playlists?p=B6717D506CEADEA6">playlist</a> for the <a href="http://www.youtube.com/bsdconferences">BSD Conferences</a> YouTube channel, or you can select a video directly from the list below. These videos are all in HD quality and most are in English and a few are in Polish. Thanks Tomasz and the MeetBSD organizers for being so organized with the video recordings again this year.<br /><br /><li><a href="http://www.youtube.com/watch?v=I_vm37L9mjY&p=B6717D506CEADEA6&index=0">Dru Lavigne - Update on BSD Certification</a><br /><li><a href="http://www.youtube.com/watch?v=QGF2FJbiJ7A&p=B6717D506CEADEA6&index=1">Hans Peter Selasky - The new USB stack in FreeBSD</a><br /><li><a href="http://www.youtube.com/watch?v=Up61bmmnc3Q&p=B6717D506CEADEA6&index=2">Jakub Klama - FreeBSD on DaVinci DMSoC (polish)</a><br /><li><a href="http://www.youtube.com/watch?v=Qmm7z7MSqho&p=B6717D506CEADEA6&index=3">Jan Srzednicki - What ideas can FreeBSD borrow from AIX?</a><br /><li><a href="http://www.youtube.com/watch?v=SpS7Ajnx9Q8&p=B6717D506CEADEA6&index=4">Attilio Rao - The VFS/vnode interface in the FreeBSD kernel</a><br /><li><a href="http://www.youtube.com/watch?v=wh09MirPd5Y&p=B6717D506CEADEA6&index=5">Marko Zec - Network emulation using the virtualized network stack in FreeBSD</a><br /><li><a href="http://www.youtube.com/watch?v=i0KFFsdebpM&p=B6717D506CEADEA6&index=6">Paweł Jakub Dawidek - HAST -- Highly Available storage for FreeBSD (polish)</a><br /><li><a href="http://www.youtube.com/watch?v=rz6INTPM1Ac&p=B6717D506CEADEA6&index=7">Paweł Jakub Dawidek - HAST -- Highly Available storage for FreeBSD (questions, polish)</a><br /><li><a href="http://www.youtube.com/watch?v=J-7Llk5uOns&p=B6717D506CEADEA6&index=8">Nikolay Aleksandrov - FreeBSD-based solution for Internet traffic management</a><br />Sławek Żak - NoSQL<br /><li><a href="http://www.youtube.com/watch?v=bj24jHtRkZs&p=B6717D506CEADEA6&index=10">Ramon Tancinco - meetBSD 2010 Welcome Intro</a><br /><li><a href="http://www.youtube.com/watch?v=ebhxZt7BbbA&p=B6717D506CEADEA6&index=11">Martin Matuska - mfsBSD</a><br /><li><a href="http://www.youtube.com/watch?v=J-FLykTaHRo&p=B6717D506CEADEA6&index=12">Dmitri Epshtein - Advances in Embedded ARM processors, for performance driven applications</a><br /><li><a href="http://www.youtube.com/watch?v=66enuplpZxw&p=B6717D506CEADEA6&index=13">Warner Losh - Using FreeBSD in a Commercial Setting</a><br /><br /><object width="640" height="385"><param name="movie" value="http://www.youtube.com/v/i0KFFsdebpM?fs=1&hl=en_US"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/i0KFFsdebpM?fs=1&hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="385"></embed></object>Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-74088816484800501892010-08-10T23:28:00.000-07:002010-08-10T23:39:34.737-07:00BSDCan on Google's Open Source BlogA coworker of mine, <a href="http://www.employees.org/~kirk/">Kirk Russell</a>, just posted an excellent summary of <a href="http://google-opensource.blogspot.com/2010/08/bsdcan-through-years.html">BSDCan through the years</a> on the <a href="http://google-opensource.blogspot.com">Google Open Source Blog</a>.<br /><br />I wasn't able to make it to <a href="http://www.bsdcan.org">BSDCan</a> this year due to family commitments, but I did make it to another <a href="http://user2010.org/">open source conference</a> later this summer that I also <a href="http://google-opensource.blogspot.com/2010/07/notes-from-user-2010.html">wrote about</a> on Google's open source blog.<br /><br />Kirk and I haven't worked closely together but we both do our best at evangelizing BSD and open source inside our respective corners of the company. It's great to see his post about all the excellent work happening in the BSD community on a corporate blog.Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-85771992903393367632010-05-30T00:40:00.000-07:002010-05-30T00:49:46.379-07:00Kirk McKusick on Journaling Soft Updates in FreeBSDDr. Kirk McKusick has produced a high quality recording of his talk on <a href="http://www.youtube.com/watch?v=_NuhRkiInvA">Journaled Soft-Updates</a> at <a href="http://www.bsdcan.org">BSDCan 2010</a>. This is the 92nd BSD conference video in the <a href="http://www.youtube.com/bsdconferences">BSD Conferences</a> YouTube channel.<br /><br /><object width="640" height="385"><param name="movie" value="http://www.youtube.com/v/_NuhRkiInvA&hl=en_US&fs=1&"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/_NuhRkiInvA&hl=en_US&fs=1&" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="385"></embed></object>Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-55583325793573840032010-05-29T23:38:00.000-07:002010-05-29T23:57:54.264-07:00AsiaBSDCon 2010 VideosThe videos from <a href="http://2010.asiabsdcon.org/">AsiaBSDCon 2010</a> are now available on the <a href="http://www.youtube.com/bsdconferences">BSD Conferences</a> YouTube channel. The full list of 17 AsiaBSDCon videos includes:<br /><br /><ul><br /><li><a href="http://www.youtube.com/watch?v=lofmFRY-yjs">George Neville-Neil: Hardware Performance Monitoring Counters on non-X86 Architectures</a><br /><li><a href="http://www.youtube.com/watch?v=34l1-YQvoos">Massimiliano Stucchi: BSD in the routing industry</a><br /><li><a href="http://www.youtube.com/watch?v=UiaVrlD1S7M">Marco Peereboom: Softraid: OpenBSD's virtual HBA, with benefits</a><br /><li><a href="http://www.youtube.com/watch?v=lMaNlyqUiJA">Brooks Davis: Porting HPC Tools to FreeBSD</a><br /><li><a href="http://www.youtube.com/watch?v=GbUz5Zamt7I">Marco Peereboom: Epitome2: dedup for the masses</a><br /><li><a href="http://www.youtube.com/watch?v=ZL30z1uI-JI">Rui Paulo: Wireless Mesh Networks under FreeBSD</a><br /><li><a href="http://www.youtube.com/watch?v=3B0SHPWaT58">Constantine A. Murenin: Quiet Computing with BSD</a><br /><li><a href="http://www.youtube.com/watch?v=pV1QZTzwf6c">Ryan McBride: What's wrong with PF</a><br /><li><a href="http://www.youtube.com/watch?v=c7PTL-fOM2E">Takuya ASADA: SMP Implementation for OpenBSD/sgi</a><br /><li><a href="http://www.youtube.com/watch?v=3AJNxa33pzk">Antti Kantee: Rump Device Drivers: Shine On You Kernel Diamond</a><br /><li><a href="http://www.youtube.com/watch?v=1g3Pdz1qF5E">Simon Perreault: Ecdysis: Open-Source DNS64 and NAT64</a><br /><li><a href="http://www.youtube.com/watch?v=LT85JTFJGxM">Paul Schenkeveld: Minimizing service windows on servers using NanoBSD + ZFS + jails</a><br /><li><a href="http://www.youtube.com/watch?v=LRi9swVQ_5A">Peter Losher: Closing the DNS Security Loop with DNSSEC</a><br /><li><a href="http://www.youtube.com/watch?v=9SPM1y-HynQ">Claudio Jeker: vscsi(4) and iscsid -- iSCSI initiator the OpenBSD way</a><br /><li><a href="http://www.youtube.com/watch?v=3ATCw_03jf8">Ana Kukec: Native SeND kernel API for *BSD</a><br /><li><a href="http://www.youtube.com/watch?v=Wcro3UliMZ4">Alexandre Ratchov: OpenBSD audio & MIDI framework for music and desktop applications</a><br /><li><a href="http://www.youtube.com/watch?v=Z0yOuDelXUU">Kris Moore: PC-SYSINSTALL - A new system installer backend for PC-BSD and FreeBSD</a><br /></ul><br /><br />Thanks Hiroki Sato and the other organizers of AsiaBSDCon for running a successful conference and uploading these videos. Some of these videos were previously available on ustream but are not currently accessible there. The YouTube channel provides automatic machine generated captions in ~50 languages, fast streaming, and a total of over 90 videos from conferences over the past ~3 years.Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-36405500284326631492010-04-06T23:29:00.000-07:002010-04-11T21:43:38.287-07:00FreeBSD Tech Talk @ GoogleLong time FreeBSD developer <a href="http://info.iet.unipi.it/~luigi/">Luigi Rizzo</a> from the University of Pisa came to Google last week to visit with Sam Leffler and me, and he agreed to give a talk about some of his work on link emulation and packet scheduling.<br /><br /><object width="640" height="385"><param name="movie" value="http://www.youtube.com/v/r8vBmybeKlE&hl=en_US&fs=1&"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/r8vBmybeKlE&hl=en_US&fs=1&" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="385"></embed></object><br /><br />This marks the second FreeBSD video in the <a href="http://www.youtube.com/user/GoogleTechTalks">Google Tech Talks</a> channel in addition to the 70+ videos in the <a href="http://www.youtube.com/bsdconferences">BSD Conferences</a> channel. Enjoy.Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-28592558876085017142010-02-20T22:29:00.000-08:002010-02-22T10:46:27.425-08:00Improved Conference Captions from Amazon Mechanical Turk (2)<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDhH8fgWLb_J1iB46nse_YMBug4PqqT9cDS7uFm61GmgpZWb8pjIrGT8libK561ei57BLUSFGIi03eHEokdWWxQkb6QhXYlQ0ujBDowb41ZKy5fj4VQLwpX6303pHfvYwYonumx3Z2UBv2/s1600-h/Picture+4.png"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;width: 400px; height: 268px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDhH8fgWLb_J1iB46nse_YMBug4PqqT9cDS7uFm61GmgpZWb8pjIrGT8libK561ei57BLUSFGIi03eHEokdWWxQkb6QhXYlQ0ujBDowb41ZKy5fj4VQLwpX6303pHfvYwYonumx3Z2UBv2/s400/Picture+4.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5440583051905561778" /></a><br />After my initial <a href="http://freebsd.stokely.org/2010/01/improved-conference-captions-from.html">experiments</a> last month, I applied to the <a href-"http://www.freebsdfoundation.org">FreeBSD Foundation</a> for funds to pay for additional human editing of the YouTube machine generated transcripts. The screenshot on the left shows an example <em>HIT</em> (Human Intelligence Task) available on <a href="http://www.mturk.com">Amazon Mechanical Turk</a>.<br /><br />The task description on the left is based on a template I created with three variables: <b>$VIDEO_URL</b>, <b>$VIDEO_TITLE</b>, and <b>$CAPTIONS_URL</b>. New HITs are then created by uploading a CSV file with three columns for each of those variables, e.g.<br /><br /><pre>VIDEO_URL,VIDEO_TITLE,CAPTIONS_URL<br />http://www.youtube.com/watch?v=mMmbjJI5su0,"BSD v. GPL, Jason Dixon, NYCBSDCon 2008",http://people.FreeBSD.org/~murray/improved-captions-bsdvsgpl.sbv<br />http://www.youtube.com/watch?v=Pe8LdJpBGJ4,"Isolating Cluster Jobs for Performance and Predictability, Brooks Davis (DCBSDCon 2009",http://people.FreeBSD.org/~murray/improved-captions-isolatingcluster.sbv</pre><br /><br />Using this method I created 12 HITs for the first pass of editing for which I offered between $9 and $14 per video. A slightly modified template with the same three variables was used to pay ~$7 per video for a second pass to further improve the transcripts improved in the first pass. <br /><br />The template has gotten more detailed over the past month in response to all of the minor ways that workers submitted less than perfect transcripts. The actual SBV file format used by YouTube captions is not formally specified anywhere as far as I can tell, but the 60 character maximum width and simple format can be verified in submitted transcripts with a few emacs macros.<br /><br />The transcript files have been checked into the <a href="http://www.freebsd.org/cgi/cvsweb.cgi/doc/en_US.ISO8859-1/captions/">FreeBSD Doc CVS Repository</a>. The full list of videos with human-edited English language transcripts is:<br /><ul><br /><li><a href="http://www.youtube.com/watch?v=DF_YqNIiU-o">"M. Warner Losh, An Overview of FreeBSD/mips, AsiaBSDCon2009"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-imp-mips.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=G3T05n9Y0S0">"AsiaBSDCon 2009: Internet Mail — Past, Present, and (a bit of) the Future"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-internetmail.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=a3XLROUjXic">"A. Rao: The Locking Infrastructure in the FreeBSD kernel #1"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-kernellocking-1.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=MApy8-lN49A">"A. Rao: The Locking Infrastructure in the FreeBSD kernel #2"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-kernellocking-2.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=N0q37X-MJzY">"PC-BSD, Matt Olander, AsiaBSDCon 2008"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-pcbsdolander.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=OwBh8ro7xHQ">"FreeBSD, Protecting Privacy with Tor"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-torprivacy.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=Pe8LdJpBGJ4">"Isolating Cluster Jobs for Performance and Predictability, Brooks Davis (DCBSDCon 2009"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-isolatingcluster.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=UM4ZrsOjmNQ">"Richard Bejtlich, Network Security Monitoring Using FreeBSD"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-networksecurity.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=YzXLwhkBe80">"Jason Dixon Closing Remarks of DCBSDCon - BSD is Still Dying"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-dcbsdcon09-closing.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=ds77e3aO9nA">"A Narrative History of BSD, Dr. Kirk McKusick"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-historyofbsd.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=g7tvI6JCXD0">"BSD is Dying, Jason Dixon, NYCBSDCon 2007"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-bsdisdying.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=mMmbjJI5su0">"BSD v. GPL, Jason Dixon, NYCBSDCon 2008"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-bsdvsgpl.sbv">captions</a>)</em><br /><li><a href="http://www.youtube.com/watch?v=nwbqBdghh6E">"FreeBSD Kernel Internals, Dr. Marshall Kirk McKusick"</a> <em>(<a href="http://people.FreeBSD.org/~murray/improved2-captions-freebsdkernelinternals.sbv">captions</a>)</em><br /></ul>Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-80615469246919385962010-01-10T17:38:00.000-08:002010-01-10T18:20:29.305-08:00Improved Conference Captions from Amazon Mechanical TurkJust wanted to send a quick note that three of the popular videos from the <a href="http://www.youtube.com/bsdconferences">BSD Conferences</a> YouTube channel have been updated with human-edited English language caption files. These offer a significant improvement over the <a href="http://freebsd.stokely.org/2009/12/machine-generated-captions-for-bsd.html">machine generated captions</a> I wrote about last month.<br /><br />The following videos have been updated:<ul><li><a href="http://www.youtube.com/watch?v=N0q37X-MJzY">PC-BSD, Matt Olander, AsiaBSDCon 2008</a> (<em><a href="http://people.FreeBSD.org/~murray/improved-captions-pcbsdolander.sbv">transcript text</a></em>)<br /><li><a href="http://www.youtube.com/watch?v=nwbqBdghh6E">FreeBSD Kernel Internals, Dr. Marshall Kirk McKusick</a> (<em><a href="http://people.FreeBSD.org/~murray/improved-captions-kernelinternals.sbv">transcript text</a></em>)<br /><li><a href="http://www.youtube.com/watch?v=ds77e3aO9nA">A Narrative History of BSD, Dr. Kirk McKusick</a> (<em><a href="http://people.FreeBSD.org/~murray/improved-captions-historyofbsd.sbv">transcript text</a></em>)<br /></ul><br />I've also posted three simple captions text files which provide the times and text in a very simple ascii format in case anyone wants to provide a diff to improve any remaining mistakes in the captions.<br /><br />The transcriptions were done with the help of the industrious workers behind <a href="http://www.mturk.com">Amazon Mechanical Turk</a>. The three transcripts above, representing at least 6 person hours of work, but easily twice that much time, were completed for less than $50 by leveraging the timing information from <a href="http://googleblog.blogspot.com/2009/11/automatic-captions-in-youtube.html">free machine generated captions</a> and mechanical turk for the editing. This is less than 1/10th of the cost of a commercial transcription service.<br /><br />What is the quality of these captions in other languages when automatically translated with YouTube? Are there any other videos for which captions would particularly be useful?<br /><br /><a href="http://2010.asiabsdcon.org/">AsiaBSDCon</a> is coming up in March, and I hope to have things streamlined by then such that videos with both Japanese and English captions can be added to the channel shortly after the conference.Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-78748747381906922372009-12-22T13:35:00.000-08:002009-12-22T14:17:30.831-08:00Machine generated captions for BSD conference videosOne of the most frequent requests I've received, since <a href="http://freebsd.stokely.org/2008/12/new-channel-on-youtube-for-bsd.html">Launching the BSD Conferences YouTube channel</a> last year, has been for captions in Spanish, Russian, Chinese, and other languages. I was excited last month when Google announced <a href="http://googleblog.blogspot.com/2009/11/automatic-captions-in-youtube.html">automatic captions</a> for Youtube videos using machine translation. This feature is still highly experimental but I am happy to report that it has been enabled for the <a href="http://www.youtube.com/bsdconferences">BSD Conferences</a> channel. In combination with the much more mature automatic translation feature, this means that captions are now available in over 50 languages from Afrikaans to Vietnamese for most of the 73 videos in the BSD Conferences channel.<br /><br />The automatic captions are still highly experimental and the quality of transcription for highly technical content spoken by a diverse set of international speakers is a significant challenge to get right. If you are interested in helping to correct any of the English transcripts I would be happy to provide you a simple text file of the transcription, with each line offering the start and end time for the caption to be displayed, and the caption text. One advantage of the machine translation is that the most time consuming part of manually creating captions, synchronizing the timing of the text with the speech, has been done automatically. Even when the technical words are mangled, the timing information in the automatic captions files can be leveraged to make the process of manually improving the captioning much easier.<br /><br />The experimental automatic captions are only available directly from the video watch pages, and not from channel pages or other views. For example, visit <a href="http://www.youtube.com/watch?v=nwbqBdghh6E">www.youtube.com/watch?v=nwbqBdghh6E</a> to see one of our most popular videos, Kirk McKusick speaking on FreeBSD Kernel Internals. Hover over the triangle at the bottom right of the video, then over the CC submenu and select "Transcribe Audio". You can then choose to "Translate Captions" into a different language as well.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-SKCWcr3hNvJKrPBpxY4ql4iE22yQytfgzpKR-XBqAdM3skS3D6GYSpBnQB21iasrT1irx3hybGESUf-Q6H8UkJPuCJYelMDRVChdxipPQhnzWWcA1-kEigyrfHuVGKLlTqyNJdQcMpfd/s1600-h/Picture+2.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 182px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-SKCWcr3hNvJKrPBpxY4ql4iE22yQytfgzpKR-XBqAdM3skS3D6GYSpBnQB21iasrT1irx3hybGESUf-Q6H8UkJPuCJYelMDRVChdxipPQhnzWWcA1-kEigyrfHuVGKLlTqyNJdQcMpfd/s320/Picture+2.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5418185660027808322" /></a>Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-33300286046719537742009-07-26T22:32:00.001-07:002013-06-28T20:01:43.528-07:00The Slashdot Effect<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOqh0RWr3jE9unjBq7PCzQN_JX7CTYd1wKSHKVfErVsrQqa2L-RYV6VABwiWs7HqDU4avEiUjDtkJ4Z6yaViPxSvYHoTuJE3DMsQ5rkYTitWMdvUZRlQWWjiNW2dLe1RFw2pr-hxTpmYZI/s1600-h/Picture+2.png"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 320px; height: 146px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOqh0RWr3jE9unjBq7PCzQN_JX7CTYd1wKSHKVfErVsrQqa2L-RYV6VABwiWs7HqDU4avEiUjDtkJ4Z6yaViPxSvYHoTuJE3DMsQ5rkYTitWMdvUZRlQWWjiNW2dLe1RFw2pr-hxTpmYZI/s320/Picture+2.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5363008781144161730" /></a>After 8 months, 66 videos uploaded, and 141,676 views, the <a href="http://www.youtube.com/bsdconferences">BSD Conferences YouTube Channel</a> was slashdotted for the first time last week. Specifically, Theo's <a href="http://www.youtube.com/watch?v=i7pkyDUX5uM">OpenBSD Release Engineering</a> talk was linked from <a href="http://tech.slashdot.org/story/09/07/16/2322203/Why-OpenBSDs-Release-Process-Works?from=rss">this slashdot post</a>. Views of the video spiked to nearly 8,000 a day after the Slashdot post, which dwarfs the previous highs of around 1,500 videos a day after I posted about Kirk McKusick's <a href="http://www.youtube.com/watch?v=nwbqBdghh6E">FreeBSD Kernel Internals</a> lecture.<br /><br />I think this is an excellent reminder of the power that forums like <a href="http://www.slashdot.org">Slashdot</a> still have in directing traffic among those seeking technical content online. I would encourage anyone interested in seeing more BSD related content online to install browser bookmarklets, toolbars, or other shortcuts to more easily share and promote FreeBSD content on <a href="http://www.digg.com">Digg</a>, <a href="http://www.delicious.com">Del.icio.us</a>, <a href="http://www.slashdot.org">Slashdot</a>, etc..Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-24468244978827284022009-07-18T23:28:00.000-07:002009-07-19T03:15:24.803-07:00FreeBSD Code Metrics Now on Ohloh.netI've <a href="http://freebsd.stokely.org/2008/04/thoughts-on-wwwfreebsdorg-2-of-2.html">written previously</a> about <a href="http://www.ohloh.net">Ohloh.net</a> and how I'd like to see more of the dynamic code metrics calculated there available on the <a href="http://www.FreeBSD.org">FreeBSD web site</a>. I am happy to report that today I noticed after several years of attempts, the ohloh repository import servers have finally managed to get through the entire FreeBSD source repository. Their software setup previously had difficulties dealing with a project with as long of a history as FreeBSD.<br /><br />You can now view the top level <a href="https://www.ohloh.net/p/freebsd/analyses/latest">code metrics</a> about FreeBSD from the <a href="https://www.ohloh.net/p/freebsd">FreeBSD Project Page</a> on Ohloh.net. This page indicates that there are over 10 million lines of code, that more files are licensed under GPLv2 than any other license.<br /><br />The committer totals do not quite match up with <a href="http://people.freebsd.org/~peter/commits.html">Peter's Commit Counters</a>. Even after accounting for the fact that Peter's system could potentially double count a commit that touches both sys and non-sys parts of the source tree, the numbers from Ohloh are still lower for some committers. Unlike the numbers on <a href="http://www.cia.vc">cia.vc</a>, the FreeBSD project on ohloh.net only contains the source repository. We are currently lacking the anonymous cvs access to our doc repository necessary to add the doc project to ohloh.net.<br /><br />How do the numbers reported on ohloh.net compare to code metrics others have reported for FreeBSD? Does this match expectations or are there any major problems with this data? How can we use this information on our website? Would a badge on the front page showing "Last improvement made X minutes ago" be useful? A list of most active committers in the past week on one of the developer pages? Other ideas for utilizing the work the Ohloh project is doing?Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-38509443627246893202009-07-12T05:22:00.001-07:002009-07-12T05:53:57.916-07:00Open Source in RecessionsIn general, recessions can be <a href="http://www.economist.com/businessfinance/displayStory.cfm?STORY_ID=13743278">really good</a> for open source. Large businesses look to cut back on IT budgets and this often involves re-evaluating whether proprietary software and maintenance contracts are necessary, given high quality open source alternatives. Companies may dedicate more internal resources to open source projects, and also the surplus of underemployed engineering talent in the market may be available for more open source development work.<br /><br />Unfortunately, there are also some significant downsides, and one that I would like to highlight is the plight of the small open source shops around the world. We've seen pleas earlier this month to <a href="http://www.freebsdnews.net/2009/07/03/save-bsd-magazine/">Save BSD Magazine</a>, and in recent years some other smaller open source companies such as <a href="http://www.daemonnews.org">Daemonnews</a>, and the Japanese publications <a href="http://www.freebsd.org/publish.html#magazines">FreeBSD Press</a> and BSD Magazine have exited the business. In the current environment I would like to take the unusual step of plugging a company that has been selling, marketing, legally defending, and supporting FreeBSD from the very beginning.<br /><br /><a href="http://www.FreeBSDMall.com">FreeBSD Mall</a> has been selling FreeBSD CDs since 1.0 in 1993 and is still selling and supporting CDs, DVDs, books, branded apparel, and more. The <a href="http://www.freebsdmall.com/cgi-bin/fm/pcbsddvd7.1">PC-BSD live-dvds</a> make an excellent introduction to FreeBSD for new users and the complete <a href="http://www.freebsdmall.com/cgi-bin/fm/bsddvd7.2">FreeBSD DVD</a> distributions are quite handy to have. Consider spending a few dollars at the FreeBSD Mall site, buy a BSD Magazine subscription, or otherwise spend some money to encourage the small commercial FreeBSD ecosystem and perhaps contribute to more funds being available to exploit the many disruptive opportunities (netbooks, cloud computing, etc..) that could be very good for open source and FreeBSD during this recession.Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0tag:blogger.com,1999:blog-170373997768604905.post-91760922867828219312009-05-17T23:56:00.000-07:002009-05-18T00:05:02.745-07:00Remaining AsiaBSDCon 2009 Videos PostedThe remaining 9 videos from <a href="http://2009.asiabsdcon.org">AsiaBSDCon 2009</a> have been posted. The new videos include talks by Theo de Raadt, Eric Allman, Kris Moore, Mohamad Fauzie, Brooks Davis, Atillio Rao, A. Kantee, and the Works In Progress Sessions.<ul><br /><li><a href="http://www.youtube.com/watch?v=i7pkyDUX5uM">AsiaBSDCon 2009:The OpenBSD Release Process: A Success Story, Theo de Raadt</a><br /><li><a href="http://www.youtube.com/watch?v=G3T05n9Y0S0">AsiaBSDCon 2009: Internet Mail — Past, Present, and (a bit of) the Future, Eric Allman</a><br /><li><a href="http://www.youtube.com/watch?v=7VYyQvzdD9g">Kris Moore: PC-BSD - Making FreeBSD on the Desktop a reality</a><br /><li><a href="http://www.youtube.com/watch?v=HWPPhMmuDT8">Mohamad Dikshie Fauzie: FreeBSD and SOI-Asia Project</a><br /><li><a href="http://www.youtube.com/watch?v=oL0JSs1DnUU">Isolating Cluster Users for Performance and Predictability, Brooks Davis</a><br /><li><a href="http://www.youtube.com/watch?v=MApy8-lN49A">A. Rao: The Locking Infrastructure in the FreeBSD kernel #2</a><br /><li><a href="http://www.youtube.com/watch?v=a3XLROUjXic">A. Rao: The Locking Infrastructure in the FreeBSD kernel #1</a><br /><li><a href="http://www.youtube.com/watch?v=RxFctq8A0WI">A. Kantee: Environmental Independence: BSD Kernel TCP/IP</a><br /><li><a href="http://www.youtube.com/watch?v=fv-AQJqUzRI">Work-in-Progress Session in AsiaBSDCon 2009</a><br /></ul>Thanks again to Hiroki Sato for posting the videos and organizing 3 consecutive years of successful <a href="http://www.asiabsdcon.org">AsiaBSDCons</a>. Sato-san has also created two separate YouTube playlists for the <a href="http://www.youtube.com/view_play_list?p=38B796B3DFE05469">AsiaBSDCon 2008</a> and <a href="http://www.youtube.com/view_play_list?p=EEBDF3C5853C513D">AsiaBSDCon 2009</a> videos. These playlists make it easier to find the newest videos from among the 66 videos now in the <a href="http://www.youtube.com/bsdconferences">BSDConferences</a> YouTube channel.Murrayhttp://www.blogger.com/profile/05615584529128840992noreply@blogger.com0