Once More unto the Breach

Why Openstack is Different from Other Open Source Projects

The Openstack project feels different from other open source projects to me. Let me try to explain.  Henrik Ingo did an excellent analysis of open source project size versus governance structure a few years ago.  Essentially, the nine largest most vibrant open source communities were anchored inside non-profit foundations. The tenth largest was an order of magnitude smaller and existed inside a corporation. Henrik was a good engineer and provided his data and listed his assumptions.  He was not suggesting that the growth was causal, simply that there was a strong correlation.  He rightly observed that it will be fascinating to see what it all means using Openstack as an example when he presented his findings Summer 2011. (He wrote an excellent follow-up post comparing cloud projects later.)

Henrik’s analysis and observations presumes certain criteria about how the open source licensed project must be functioning in and of itself.  It has to be a “well run project” before getting to the foundation amplifier. Lots of people have described essentials of successful communities.  Some of my favorites include Dave Neary, Donny Berholz, and Kohsuke Kawaguchi, and of course there’s Jono Bacon’s essential reference.  I tried to capture and collect together the patterns and practices of successful open source projects in a blog post a while ago. 

Paula Hunter and I offered rationale for the foundation amplification when we were executive and technical directors at the Outercurve Foundation. What might be the cause of the amplification?  We believe it comes from the one thing ALL foundations do for their stakeholders around their projects. Foundations exist at the behest of their stakeholders to make clear the IP management requirements and provide IP risk mitigation. The project community must be well run from engineering and governance perspectives, but once a foundation exists, corporate participants have a clear on ramp for participation and the investment in the community can grow substantially. 

So Solid Engineering Practices + Strong Community Governance + Clear IP Management enables Growth. So far so good. 

But Openstack is somewhat unique as an open source project. It was 2010. Amazon had enormous early mover momentum in providing cloud services [2006]. Eucalyptus was open source-licensed but controlled by a single vendor (née 2009). Cloudstack was just published under an open source license [May 2010] but still closely held by Cloud.com. The time was ripe for a vendor neutral option.  Openstack was announced into existence by then Rackspace CEO, Lew Moorman, and NASA CTO, Chris Kemp, at OSCON 2010. 

A large collection of vendors jumped early and hard onto a simple infrastructure to evolve it rapidly forward into a marketplace of potential customers.  The project governance was created and people started gathering at summits. Openstack was correctly forced into a neutral, non-profit foundation within two years to clarify IP ownership. (Everyone had MySQL as an example on their mind, as it had been bought by Sun Microsystems [2008], and then bought and ransomed by Oracle [2010].)

But here’s where Openstack begins to break patterns. There was relatively little code at the start. It was created from the start to engineer forward.  Openstack is going through forced growth in a time frame that is a 20%-25% of the time of other large-scale successful open source-licensed infrastructure projects.  Instead of 20 years, the effort is becoming enormous in four, already driving serious vendor-lead products to market. 

Project	Started (age)	LoC Mid-point**	LoC Today	Contributors Mid-point	Contributors Today	Re-Arch	Foundation
Linux kernel	1991 (23)	4,000,000	17,000,000	186	1000	2002	2000
Apache (httpd)	1995 (19)	980,000	1,700,000	17	17	2002	1999
gcc	1987 (27)	1,150,000	6,900,000	50	94	1998	1987*
Openstack	2010 (4)	500,000	2,300,000	174	575	???	2012

[* gcc started in the Free Software Foundation; ** All data pulled from Ohloh.net, Wikipedia, and interviews]

Openstack didn’t evolve in simple use cases with the experimental and experiential use that other infrastructure open source-licensed projects saw (e.g. Linux, Apache) before large vendors got involved.  Openstack continues to demonstrate enormous growth and participation. As vendors begin to develop cloud delivery products and services out of the various Openstack projects, they are discovering holes in functionality requiring them rapidly to create new Openstack projects to fill the gaps.  Vendors are also starting to discover that Openstack itself may not scale to the needs of certain industry desires (e.g. Telco desires around NFV).  As well, all core infrastructure open source projects hit points in their organic histories where a reset was required in the architectural design to account for the stresses of new use and deployment.  Linux, Apache, and gcc were each re-architected at points in their histories to accommodate the organic growth of the project into new deployments and use.  

So interesting questions:

• When do the developers at the heart of Openstack projects collectively re-architect/re-factor/re-write core Openstack components to scale to the real world workloads that people are discovering customers need to manage over the next 5 years?
• How does the Openstack Foundation capture those consumer requirements in a fiercely competitive landscape of competitors?
• How will the Openstack Foundation adapt to the needs of its vendor, user, and consumer stakeholders? 

Open source communities are amazingly adaptable organisms.  It will be fascinating to watch Openstack as a community of projects and as a foundation continue to grow and evolve to meet the challenges of the cloud marketplace. 

 

"The Internet was Secured by Two Guys Named Steve"

Jim Zemlin has always been quotable. His keynote at this year’s Linux Collaboration Summit provided a great summary (as always) of the growth of the Linux ecosystem, but also focused on the serious problems in the security of the Internet in an era when key breaches have their own branding and logos. [Think Heartbleed and Shellshock.] He ran through some scary facts:

• OpenSSL secures most of the Internet and the small team at the OpenSSL Foundation (including the “two Steves”, Steve Henson and Steve Marquess) were only pulling in about US$2000 in donations per year. 
• Theo de Raadt supports OpenSSH part-time. 
• Harlan Stenn “runs the clocks on the Internet”, i.e. he’s responsible for ntpd, and until recently was earning about US$25K/year. 
• Werner Koch maintains GnuPG, which secures a lot of email and provides the confirmation that a software package is what it says it is. According to a recent interview, he was going broke. 

Before readers unversed in open source software get concerned with the security of open source software, let us remember this is a software problem, not an open source problem. Closed proprietary products have their share of exploits, etc.  But with open source licensed software, the broad community can do things. 

It is perplexing that if Linus’s Law is true, “Given enough eyeballs, all bugs are shallow", then such security problems persist.  Jim suggested as he gave the above examples that “there just aren’t enough eyes.” I’d offer a corollary. I think vibrant projects live a culture of review before code gets committed.  I think this is because the developers have perspective and context that can never be built into a static analysis tool. Tools can find obvious portability breakage, or some security related issues (e.g. buffer overflow problems), so issues that are likely syntactically based, but a human can understand the semantic content of the code in front of them.

There's even research to back this up:

• Walcélio Melo, Forrest Shull, Guilherme Horta Travassos, "Software Review Guidelines", Technical Report ES 556/01, COPPE/UFRJ, August 2001. "Code review is more effective than test because in review the faults in the code are found directly, while testing uncovers only the symptoms of problems, requiring debugging to find the direct cause. The seriousness of the wrong behavior by the system does not have a relation to the type of mistake, since even simple mistakes can cause complex behaviors."
• Reidar Conradi, Amarjit Singh Marjara, Øivind Hantho, Torbjørn Frotveit, and Børge Skåtevik, "A Study of Inspections and Testing at Ericsson, NO", a rewritten edition of the paper presented at PROFES'99 (Oulu, Finland, June 1999). From the Conclusion: "Software inspections are indeed cost-effective: They find about 70% of the recorded defects, take 6% to 9% of the development effort, and yield an estimated saving of 21% to 34%. i.e., finding and correcting defects before testing pays off, so indeed “quality is free”.” and: "Individual inspection reading and individual Code Reviews are the most cost-effective techniques to detect defects, while System Tests are the least cost-effective."

In the open source community bugs are found quickly, but this happens after the fact.  Vibrant projects live a culture of review before code gets committed so it’s much more like they find the bugs before they happen.  Many of the key projects that have had breaches are taken for granted and don’t necessarily have the vibrancy of a Linux, or an Openstack. These projects have simply become part of the fabric of the Internet. 

The Linux Foundation is stepping up to tackle these problems with the Core Infrastructure Initiative (CII).  The Foundation is in an excellent position at this point in time to be the centre of gravity for such industry action. Jim talked about the Initiative in his keynote.  A broad collection of players have banded together to provide a three pronged approach to the problem of securing the open source software that secures the Internet. 

• Invest in the people that are the most knowledgeable about key projects. 
• Provide deep audits for the key projects to work to prevent the next security breach. 
• Develop and disseminate best practices and guidelines for developing and deploying secure software. 

Jim’s big concerns are how not to perturb the market economics that drive open source software ecosystems and how to avoid creating an open source welfare state. He rightly used the example of I35W bridge collapse of an example of failing infrastructure that should have been fixed before a key transcontinental artery collapsed. I think that’s the right idea economically. 

Governments invest in infrastructure to enable economic growth. Support and investment for rights of way for railroads, deep port infrastructure, or interstate highway systems creates the transportation infrastructure that enables economic growth and free markets for all. All the projects Jim discussed are fundamental Internet infrastructure. If a project under consideration implements or secures an underlying universal communications protocol  or cryptographic algorithm then it is probably a good candidate for CII investigation.

Likewise, software projects that are not owned by corporations seem to be a necessary attribute. A database, even one as broadly deployed as MySQL, shouldn’t be a candidate. A fabulous engine for application deployment (node.js) is owned by a company. I’m pretty sure the investors would love the vendor community to invest in securing node.js “because it’s so hugely important going forward at enabling blah blah marketing blah.” Sorry — if a company controls the copyright, then you’re off the list. Private roads didn’t get government funded bridges. 

The Linux Foundation is obviously not a government, but it is a well-funded well-organized industry non-profit.  As such it provides an excellent place for the vendors that best benefit from the Internet infrastructure to collectively support the infrastructure on which they all depend. 

The Core Infrastructure Initiative efforts are fundamentally important.  A complete list of participants to date exists on the Linux Foundation site. Jim’s excellent keynote is up on Youtube, the slides will hopefully be up soon, and Jim’s blog post introducing CII is published on the Linux Foundation blogs. If your company isn’t supporting the initiative, it is well worth exploring how best to participate.