BSDCan Trip Report

BSDCan wrapped up yesterday and I'm back in the SF Bay Area. As usual, Dan did a great job organizing everything. A scheduling conflict prevented me from attending the developer summit before the conference, but I was mostly able to sync up with those working in areas of interest. Some of my highlights from this year's conference include :

• Having lunch with Bosko. I hadn't seen him since USENIX 2004 and he hasn't been around the FreeBSD development community much recently but previously did a lot of work on TCP and it was great to catch up with him.


• Discussing finstall with Ivan. I reviewed his latest alpha4 ISO image and provided additional feedback in person about what I think the priorities should be for getting this to the point of being the default installer for FreeBSD 8.0. Downloaded the code from sourceforge and discussed PyGTK interface for selecting/installing packages. I think I volunteered to implement a few menus, doh!


• Talking with Denise (iXsystems), Leslie (Google), and others about doing a MeetBSD Conference at Google in Mountain View in November. More details to follow once facilities are confirmed. Open source gatherings at the googleplex usually work pretty well, with free food, wireless, and copious conference rooms available.


• Dinner with Mike Silbersack, Doug Rabson, and Zach Loafman (Isilon) about Kerberized NFS, NFSv4, Microsoft, and general kvetching.


• Recording a BSDTalk podcast with the rest of the FreeBSD Core Team in attendance (not yet posted, audio coming soon).


• Talking with Deb from the FreeBSD Foundation about fundraising and more prominent links to the foundation website from www.freebsd.org.


• Discussing the pending Subversion conversion of the FreeBSD CVS repository and the necessary updates to our committer guide.


• Drinking a very nice single malt scotch with Brooks Davis and others in the hacker lounge late into the night, teasing Kip Macy, and learning all about some fine San Francisco establishments from George Neville-Neil.


• And of course, in addition to the hallway track there were actually lots of interesting talks such as Chris Lattner on BSD Licensed C++ compiler, LLVM, John Baldwin on kernel debugging, and Leslie Hawthorne on Summer of Code.

finstall alpha4 review

finstall is Ivan Voras's installer project for FreeBSD. His goal is to create a more user friendly, extensible, and maintainable replacement for the venerable sysinstall program. Replacing sysinstall is something that has been near and dear to my heart since I was first granted a commit bit to work on sysinstall over 8 years ago.

In the past decade a number of projects (libh, anyone?) have been initiated with the goal of replacing sysinstall and none have succeeded. Ivan began his 2007 Summer of Code project with a survey of the existing BSD/Linux installer landscape and his finstall project aims to provide functionality not possible with the alternatives while avoiding some of the pitfalls.

Today Ivan gave a presentation about finstall at the FreeBSD Developer Summit preceding BSDCan but work is keeping me in San Francisco until Friday. Since I'm missing the in person discussions about finstall and FreeBSD 8.0 I thought I'd at least download the latest ISO and publish my findings here. The latest code for finstall is available on SourceForge but Ivan also provides ISO images on his blog.

The first thing you'll notice is that the installer is LiveCD based. Sysinstall was traditionally installed from boot floppy disks or as floppy images on El-Torrito bootable CDs. This led to the use of crunchgen(1) and other techniques for keeping the size of the installation environment within the confines of 1.44MB disks.

Freed from this anachronism, finstall boots into FreeBSD 7.0 and provides a login prompt where users can login as root to use the disk as a LiveCD / recovery CD, or as install to complete an installation onto local disk.

Immediately after logging in as install, the user is prompted to choose a keymap and timezone using the same text libdialog(3) based tools that sysinstall users are familiar with. There is no back button if you accidentally choose the wrong keymap, and no easy way to intuitively scroll through the long list of options in each menu. Thankfully, these are the only two parts of the system that still rely on libdialog(3).

After the timezone is selected, X Windows starts up and the user is dropped into an XFce desktop with a 'FreeBSD Install' icon. Selecting this icon launches the graphical installer. Only the novice (minimum interaction) menu item is available, but standard and advanced modes which allow for RAID configuration are being worked on.

The novice installation process sets up a default partition layout based on the available space on the drive selected by the user, without allowing the user to change the partitions in any way. On my 4 GB VMWare image it chose 512MB /, 409MB /var, 256MB swap, 2048MB /usr, and 865MB /home. The user is then given a choice of file systems to use (UFS+SU, UFS+GJ, ZFS, ext2). finstall refuses to install on a disk with less than 2.3GB of space. This seems excessive until you realize that the current version isn't going to give you the option of pruning down the installed package set -- everything on the ISO will be installed.

After file system selection, the installation begins with the format/partitioning of the disk, installation of the base system, and installation of all the packages on the LiveCD. During installation a number of helpful system tips are provided to the user beneath the status bar ("BSD stands for ...", pointers to the FreeBSD Handbook, etc..)

After installation is completed the user is asked to enter a root password, create a user account, setup network interfaces, and enable system daemons. These configuration menus are pretty standard but are much more user friendly than the sysinstall equivalents.

After installation, the user is left at the XFce desktop. It would probably be more useful at this point to actually reboot into the newly installed system. Perhaps the last message of the installer could say something like "Click here to reboot into your new FreeBSD 7.0 system."

This post is far longer than I expected so I'll end here, and follow up tomorrow with some specific suggestions that I'd like to see added to finstall. In the mean time, by all means check out the code on SourceForge and send Ivan your patches.

Summer of Code Students in Community Bonding Period

We're over halfway through the community bonding period and in less than 2 weeks it will be time for our Summer of Code students to start coding.

At this point we've created Perforce accounts for our students and over half of them have already started submitting to their project directory. Ten of our students have started wikis about their projects over on wiki.FreeBSD.org and 8 students have setup blogs which are now aggregated on Planet FreeBSD SoC 2008. Many students sent introduction emails to the freebsd-hackers list over the last 3 weeks.

We ended up getting one extra student in the final allocation for a total of 21 Summer of Code students this year. Two of our students are previous summer of code participants (Gábor Kövesdán from SoC 2006, and 2007 and Rui Paulo from SoC 2007). We also have a former student, Attilio Rao (SoC 2007), serving as a mentor this year.

The graphs in this post present a breakdown of the current country of residence/study for the FreeBSD Summer of Code students that have shared this information. The countries with the most participating students are represented by darker shades of red on the world map above. The statistics for the entire Google Summer of Code program are available here. If you haven't already, please take a moment to send a welcome mail to our students in response to one of the introductions on hackers@ or use the forums above to get in contact with any students working in areas you are particularly interested in.

Update 3/14/2008 - We have a student from Portugal who was misclassified on the original graphs.

Technical BSD Content on YouTube

As I mentioned earlier I think we could do a much better job of presenting FreeBSD content online. One of the things I'd especially like to see is more technical talks available on YouTube.

The next time you register for a BSD related conference make sure to ask the organizer about video recordings. Many BSD conferences have cameras in the sessions, but there is a lot of additional work required to edit the videos, encode them, and upload them to popular video sharing sites. Personally, I think that conference organizers should be soliciting sponsors for this work and paying professionals to handle it as the last 9 years of BSD conferences haven't been very successful at capturing video with volunteer labor. The potential audience for these videos is huge, and some of the videos below have been watched over 10,000 times with basically no marketing or links from official sites anywhere. This could be a very effective way at spreading BSD technologies beyond the very limited set of people that attend the conferences or read the limited distribution conference proceedings.

Some of the more interesting BSD related videos I've seen available online include:

• FreeBSD wifi development
  Sam Leffler


• How the FreeBSD Project Works
  Robert Watson, June 20, 2007


• FreeBSD vs Linux on ScreenSavers, Tech TV (Part 2)
  Matt Olander and yours truly


• FreeBSD Clusters, ScreenSavers, TechTV
  Brooks Davis and Matt Olander


• DomainKeys Identified Mail Technology
  Eric Allman, December 7, 2007


• Launchd: One Program to Rule them All
  Dave Zarzycki, July 20, 2007


• Open Source Developers at Google: Amit Singh on MacFuse
  Amit Singh


• Theo de Raadt OpenBSD Interview
  Theo de Raadt, January 6, 2008

Thanks to Julian Elischer for taping and organizing some of the BAFUG tech talks in recent years (.mov files available here). It would be great if more FreeBSD user groups contributed video of interesting talks. Some other open source communities, such as Python, are really good at this. If anyone is interested in coming to Google and giving a talk about a BSD related project they are working on, please let me know.

Call for Open Source Awards 2008 Nominations

O'Reilly and Google have announced that for the fourth year running they will present a set of open source awards at the O'Reilly Open Source Conference this summer. The nomination process closes May 15, 2008 and submissions along with the name of the proposed award (Best Hacker, Best Community Builder, etc..) can be sent to osawards AT oreilly DOT com.

Itojun-san and Sam Leffler have been mentioned as potential strong candidates from the BSD community for impressive lifetime contributions. Send your thoughts to [email protected] to discuss or send your submission directly to osawards AT oreilly DOT com. Hubert Feyrer and Jeremy Reed have also been discussing contributions from the NetBSD side.

Thoughts on www.FreeBSD.org (2 of 2)

In the last post I identified some of the strengths and weaknesses of the current www.FreeBSD.org website. As promised, I've collected some specific suggestions that I think would improve the site:

• Better integration with CVSWeb, P4Web, ohloh, and cia.vc. A number of sites offering third party open source metrics have become available in recent years such as ohloh.net and cia.vc. We should ensure that our content is available on those sites and better utilize some of the features they provide. For example, customizable RSS feeds of code changes in specific subtrees should be made available, perhaps even only those changes that match complex search queries. Dynamic lists of the most active developers, or the parts of the source tree that are changing most rapidly could be displayed.


• Better utilization of our own structured data. We have geographic information about usergroups and events, and so it is natural to display the information to the user as a map rather than an extremely long list on one giant HTML page. It would be even better if this could be displayed as an image map or with javascript popups or integrated to one of the large online maps services.


• Enabling comments and feedback on items posted to the site. Readers of the site should have the opportunity to comment on newsflash entries and vote on development project ideas/priorities. Requiring updaters of the site to manually edit news.xml may not be the best way to handle this. Perhaps some kind of blogging software could be used which is then scraped into a static newsflash.html file while providing links back to the official blog to facilitate user comments.


• Integrate with other web sites. Since we have structured data representing events and other content on the site, we could construct links to videos or photos tagged as 'freebsd' on Flickr or YouTube during the days of each conference. There is no excuse for the technical content from any recent BSD conference not ending up there. Likewise, we should automatically update public Google/Yahoo calendars, and allow anyone visiting the site to instantly add an event to the calendar software of their choice.

Some of the particular areas that I'm interested in working on include utilizing the Google Charts API to provide visual representation of our structured data in XML files (events.xml, usergroups.xml, etc..) when appropriate. I wrote some XSLT code to generate some basic maps for the events and usergroup pages this weekend, but further improvements are needed.


I've also written a prototype web application to replace the Ideas database (ideas.xml). This application allows users to add new development project ideas, to comment or vote on existing ideas, to search the database, and to subscribe to RSS feeds of a specific search or the comments and activity on a specific idea.

It's possible to implement most all of these features on our existing xml/xslt infrastructure as long as we are willing to use web services provided by others (such as external blogs/commment feeds, links to search and graphing apis, etc.) Although not strictly required, I think it's probably time to start allowing Javascript on our pages as long as it degrades gracefully and the functionality provided is compelling enough to warrant it.

Thoughts? Comments? Better ideas for improving www.FreeBSD.org? Which open source organizations have particularly useful websites?

Thoughts on www.FreeBSD.org (1 of 2)

The FreeBSD website has been based on XML/SGML structured content since index.sgml was first checked into CVS in 1996. Since then we've realized a number of benefits of this emphasis on structured content, but there is a cost to this setup and I'm not convinced we're getting the full benefit of our infrastructure. First, consider what we gain by our current setup:

• XSLT stylesheets and CSS to separate content from presentation, so new technical pages can be added quickly in the same look and feel as the rest of the website.


• Dynamic generation of HTML pages from XML files of events and other structured data, so that we can always have a list of "upcoming" and "past" events that doesn't need to be manually updated once the event has occurred.


• Text based changes can be kept in the same revision control system we use for code. This helps our developers by lowering the barrier for them to make website contributions, and it helps our translators to easily see the textual differences of the english language pages that need to be translated.


• Complete change history allowing us to borrow content from previous revisions for cyclical activity like releases and summer of code.


• RSS feeds automatically generated for most XML content on the site, allowing the content to be syndicated to other sites or read more conveniently through feedreaders or mobile devices.


• Easy distribution of the source for the website allowing mirrors all over the world to host copies of the content.

Despite these advantages, there are also costs associated with this system.

• Communication is entirely uni-directional. There is no way for users of the website to add comments or additions to newsflash stories, to attach photos to events on the events page, or to comment on the utility of feature requests on the ideas page.


• Manual editing of XML files is tedious and error prone compared to using a web based content management system. The XML files and especially the stylesheets can be daunting to those that haven't spent significant time working with these technologies, which presents a barrier to more frequent updates and more regular improvements to the site.

I've been thinking a bit recently about these deficiencies and how we can address them while still making the most out of our existing presence on the web. I will follow up to this post with some specific ideas.