Once More unto the Breach

Patterns and Practices for Open Source Software Success

How do you create a successful free or open source software project?

There are two parts to “success”:

• Deployment growth: One publishes software to see it used. Even from the minimalist view point of, “It was useful to me; it might be useful to you.” But as the software is used, it reflects the dynamic nature of software, and as a solution to a problem, it gets used in new ways to solve new problems. This leads to the second part of the success formula -- contributions.
• Contribution flow: A free or open source software project is at it’s simplest a discussion in software, and without contributions the conversation fades and fails. From a more complex community perspective, a FOSS project is about the economics of collaborative innovation and development. Without a continuous contribution flow, the dynamic aspect of a software project will become static and brittle and lose its relevancy.

As the software at the core of the project is the dynamic thing (the “meme” to replicate), we need developers capable of contributing to the project. Developers that might contribute are people that probably started as users of the software project. They found the software, used it, perhaps fixed a bug, perhaps added a feature, or repurposed the software to a new problem. At this point they are in a position to contribute, but there’s work to be done to encourage and allow that contribution.

So there are three onramps that a FOSS project’s committer(s) need to build and relentlessly improve:

• Attract lots of users: The software needs to be easy to install/configure/use (and this applies to developer tools, libraries, and frameworks as much as to user space apps and tools).
• Out of this user base, attract developers: The software needs to be easy to download, and build and test to a known state. This allows potential developers to experiment/fix/extend the software project FOR THEMSELVES and their own selfish needs.
• Out of this pool of developers, attract contributors because there's actually a well thought out pipeline and communications about what, where, and how to contribute.  Contributions DO come from users, for example bug reports. But a project still needs to communicate with users to enable them to contribute good bug reports.

We have seen these pipelines anecdotally for years.

• In the early days of PC-based software (in the early '90s) there was the 5-Minute Rule (or 10-Minute Rule depending on who you consulted) that said if the software didn't install and do something useful [very] quickly it became shelfware.   Today, how many people download an open source project and never get it working and walk away? How much time are you willing to invest installing, configuring, building, testing, working through the forums, before you declare defeat and move on. How many potential contributors do you leave feeling frustrated and stupid?
• There was an idea expressed by core committers in some open source communities 10-15 years ago that said it's orders of magnitude between communities and contributions, i.e. for every 1000 users, you might see 100 bug reports, out of which 10 will send code that purports to fix said bug, out of which 1 read your coding guidelines and really did fix the bug.  I heard this from a couple of communities (BIG = sendmail, small = graphics drivers) and have "tested" the statement on many core committers since who all stop, think, and agree.  (Caveat lector: This is purely anecdotal. I’m thinking about ways to hunt through the data streams for proof along the lines of Donnie Berkholz’s excellent recent investigation.)
• We see it in statements in their talks from the likes of Jono Bacon ("We want to turn drive by contributors into regular contributors,"), Kohsuke Kawaguchi ("potential contributors start as visitors to your website,"), and Donnie Berkholz ("your future potential contributors start by lurking on your project's communications channels").
• Tim O’Reilly wrote “The Architecture of Participation” in 2004 describing the nature of systems that are designed for user contribution and extending Mitch Kapor’s meme on “architecture is politics.”

There are patterns and practices successful FOSS projects use to grow and evolve the community (users + developers + contributors), and thereby ultimately the software itself. Each group in the community needs an onramp. These activities can be arranged along two axes:

• Software construction: The patterns and practices that make the software itself easy to install, configure, run, build, test to a known state.
• Community development: The patterns and practices that encourage the participation of the people that will grow the user base.

So if we look at the onramps in light of the axes, we get the following diagrams.

Once a project begins to grow, there will be commercial interest. Companies will want to use the software in interesting ways in their own products and services, or provide services around the project to other users. At this point, there are an additional set of activities a project needs to undertake for community growth and contribution flow. (These activities make up their own axis so really there are three axes):

• IP management: The patterns and practices that encourage commercial participation that allows an ecosystem to take hold.

So driving open source project success is an exercise in building onramps for communities of users, developers, and contributors, and eventually commercial contributors to enable an project ecosystem.

I’ve written and presented before now on the role of foundations enabling growth. Many of the IP management issues are handled by free and open source software foundations, as they create neutral non-profit spaces for IP sharing and collaboration. Looking at the history of the Apache Software Foundation, with its incubation program to drive certain software construction and community development activities, and the role of the foundation IP structure itself, one can see the practices and patterns for open source software project success. The “Apache Way” is a reflection of the patterns and practices.

So too, with the Eclipse Foundation, its rigorous IP management and development practices and community support, and the Linux Foundation support for the Linux ecosystem. One can see this as far back as the Free Software Foundation and gcc and the enablement of a C-compiler ecosystem for the likes of Cygnus Software, and the Outercurve Foundation continues the tradition of supporting the three axes of activities for FOSS project success.

So, to summarize:

• Successful FOSS projects grow their communities outward to drive contribution to the core project.
• To build that community, a project needs to develop three onramps for software users, developers, and contributors, and ultimately commercial contributors.
• The patterns and practices involved in the onramps can be arranged along three axes of software construction activities, community development activities, and IP management activities.
• Build the onramps, and one builds the potential for the ecosystem.

I've written about many of these ideas over the past year, but without the framework. I presented the framework of patterns and practices for the first time at the Outercurve Foundation Open Source Software Conference in May this year [video below]. I'm looking forward to continuing the discussion through this next week at the Community Leadership Summit and OSCON 2013.

On Leaving the Outercurve Foundation

I’ve not written one of these posts for some time. I left the Outercurve Foundation effective the beginning of July. The Outercurve Foundation is in the process of restructuring its services and membership structure to meet a broader audience of developers and to deliver hopefully more value to its existing projects.   (Stay tuned for announcements in that space.) Part of this effort will require an increased focus on technology to facilitate more automated services, rather than "staff intensive" services.   To that end, I have left as Outercurve’s Technical Director.

It has been a great three years. The technical non-profit consortia of my experience were all launched with a collection of a dozen CEOs on stage explaining to customers the strategic significance of the collaboration to their businesses. This anchors the initial membership and acts as the inbound vector for new sponsors and members. The Outercurve Foundation (originally called the Codeplex Foundation) launched differently.

Outercurve has always had much more of a start-up feel to it. The interim board hired Paula Hunter as executive director in Feb 2010, who then hired me in May of 2010. Paula and I defined a business model for our “start-up”, iterating over the value foundations provide their free and open source software projects, and why they’re important for the growth of their projects. A lot of the thinking has gone into the various presentations we’ve given, and culminated in the recently published International FOSS Law Review article.

It’s been interesting to define the Outercurve business against the other key foundations as they each evolved around their key projects. Foundations provide IP management, a neutral non-profit space for projects to grow as commercial interest in participation grows, and experience to guide new projects. We worked at Outercurve to define a light weight IP policy while remaining rigorous. Likewise, we developed a mentorship program instead of insisting projects survive an incubation process. Our efforts at education have grown to include hosting the first modest conference for our projects with an agenda that included the likes of Jono Bacon, Scott Guthrie, Donnie Berkholz, Ross Gardler, Kohsuke Kawaguchi, and several of our more experienced project leaders.

Outercurve has grown to 28+ projects, with hundreds of contributors, and millions of lines of code across three gallery “collections”, with shining stars in each gallery. Website Panel, Chronozoom, Orchard, and NuGet continue to grow and thrive. (NuGet is now embedded in Visual Studio demonstrating that even Microsoft product teams are fully taking onboard how to live in a co-joined open source-enabled proprietary product world.) There are more, varied and interesting projects in the pipeline in discussions with the Foundation.

Working with Paula has been a pleasure. I’ve learned an enormous amount about non-profits as businesses, and the start-up as non-profit. She remains the Operational Goddess. I’ve also had the privilege of making many new acquaintances and friends this past three years. But the Board is shifting its business model evolution, it’s a new fiscal year at the Foundation, and it’s time for me to move on.

There are a couple of projects I’m chasing presently.

• At the Outercurve Open Source Software Conference I presented the start of work on patterns and practices for open source success [slides, video]. I believe there are certain activities that software development teams do that allow them to scale successfully, regardless of whether they’re building collaborative liberally-licensed projects (a.k.a FOSS), closed proprietary product for sale, or an internal IT program. I’ll be continuing to evolve this work. I feel it’s important. I also feel it has repercussions for how software engineering is taught (or not) in a Github world. Software is amazingly dynamic and if the project/product can’t scale, it’s useless. I’d like to see the equivalent of the Google Summer-of-Code re-invented into university programs to teach the next group of CompSci and software engineering students how software is constructed in the real world.
• Donnie Berholz did some great statistical work with the Ohloh dataset to demonstrate that active FOSS projects grow more active. I want to build on that work to see if there are ways to demonstrate that the patterns and practices [above] manifest themselves in more active and successful projects. The joy of datasets like Ohloh is that they demonstrate what developers actually do, and not what they say they do.
• “Open hardware” is coming. 30+ years may not be a long time to work, but in our industry it is long enough to see waves of innovation arrive that obliterate the previous model. I started work in the world of minicomputers, saw the arrival of the “toy” we called the PC, the rise of networking and client-server UNIX (now Linux) computing, and now the state of ubiquitous networking and handheld devices based on a smart phone model that didn’t exist six years ago. Embedded devices (software+hardware) are everywhere. The amateur historian and amateur economist in me is always looking for patterns in the rise and collapse of successive waves of tech and the companies that come and go. I think the rise of the Raspberry Pi, Arduino, and open hardware at places like SolderPad will become something fascinating.

All that said, I love to build teams and products that excite customers, so I’m absolutely looking for interesting work. I remain fascinated with the state of FOSS in China. [LinkedIn profile]