Once More unto the Breach

OPNFV: When Open Source and Standards Cultures Meet

I attended the Linux Collaboration Summit last week. There was a kick-off panel on the opening day (http://sched.co/28x6) and then a couple of days of working group style presentations around OPNFV. The work comes under the sponsorship of the Linux Foundation and hopes to establish a carrier-grade, integrated, open source reference platform that industry peers will build together to advance the evolution of Network Function Virtualization. (Or read here for the ETSI definition of NFV). There’s a really good description of the intended work and architecture on the OPNFV site. 

The panel was moderated by OpenDaylight exec director Neela Jacques, and participants included Eriksson, Red Hat, Cumulus Networks, and the community manager from Openstack.  I tracked the OPNFV work for its first half day of the follow-on workshop and spot checked the work for much of a second day. OPNFV started last Autumn. To develop a "reference platform" for NFV activities, the group wants to integrate the work across other open source projects (OpenDaylight, OpenVSwitch, Openstack, etc.), determining what is needed and working with the relevant upstream groups to contribute to each project community appropriately. 

What’s most interesting to me is the very different cultures that are exposed in the room.  Successful open source communities:

• build the one true implementation
• by collaborating through contributions upstream, and 
• meritocratic influence is driven by the contribution of code, infrastructure, and effort. 

Standards organizations on the other hand : 

• collaborate on interface specifications to enable multiple [communicating] implementations 
• by negotiating a compromise position amongst participants, and 
• democratic influence is gained by diplomacy and participation. 

Time frames are also different very different between such technical collaborations. Successful open source communities are building in real time, and larger complex projects have begun scheduling themselves around agile six month delivery cycles. Standards documents often need to support complex externally driven procurement and certification needs and are carefully [sometimes deliberately slowly] developed to ensure participants can meet those needs, and once agreed, change relatively carefully i.e. slowly and with rightly conservative process. (For the moment we’ll assume the open source project is mature enough to be hosted inside a foundation so IP management is understood, and that the IP management of the standard is equally well managed in the standards development organization.) 

NFV is a telecommunications thing and the telco world was historically driven by standards with large internationally-focused organizations (ISO/IEC, ETSI, etc.) at their core. Linux as a vibrant open source community has demonstrated its ability to deliver vendor-collaborative engineering value this past 25 years. Openstack is attempting to replicate that success and is the hub for much of the present Infrastructure-as-a-Service (IaaS) cloud work. As explorations of software defined networking (SDN) have begun in the open source community, projects such as OpenDaylight (also under the auspices of the Linux Foundation) and now OPNFV have begun. The telco community wants to invest in driving this sort of open source collaborative work forward to reap the benefits of reduced costs and delivery agility. They just don't necessarily know they need to participate differently.

The OPNFV working group provided a stark example of this lack of understanding. Dave Neary (Red Hat) gave an excellent presentation on open source collaboration to the room. Chris Wright (also Red Hat) led a number of discussions. But many of the comments from the telco audience participants tended to be of the nature:

• When are you going to have work done for us to review?
• Notes on <some topic> are missing from the OPNFV wiki.

Many of the standards-centric telco participants seem to not understand that work happens in an open source community WHEN they participate and by their very participation.

Chris Price (Ericsson) is a primary technical leader for OPNFV. He too is working hard to understand the sort of coordination and participation that will be required. The coordination efforts across the collection of projects is enormous. Chris Wright made reference to one piece of work he tried to negotiate that required changes to OpenDaylight, Openstack, OpenVSwitch, libvirt/qemu and a couple of other projects and how trying to communicate requirements and priorities across such diverse projects is difficult. Just because he had a coordinated set of patches for each project doesn't mean there's interest to accept such contributions "from a stranger" into a project focused on different priorities. It’s not simply about drive-by contributions. One needs to participate enough to be heard.  

Likewise, using terms like "reference platform" in a standards-centric audience is often an invitation (indeed a command) to fork code. There is an enormous engineering cost to living on a fork, and to not aggressively contributing small changes regularly, and it often takes vendors unfamiliar with open source participation a while to come to grips with this learning curve. 

Ultimately, the work is very important going forward. There are many vendors in the room (including HP where I work) with a wealth of open source experience.  Hopefully, any initial confusion is sorted quickly and the work will move forward accordingly. 

 

Microsoft Failing Its Own OOXML Standard (ISO/IEC 29500)

I was dismayed this morning to see Andy Updegrove's write-up on Alex Brown's post on Microsoft failing it's own Office Open XML (OOXML) standard (formally known as ISO/IEC 29500). It is the second anniversary of the ballot resolution and approval of the standard. While Andy was reporting from the trenches through the final ballot resolution, Alex was responsible for the negotiations that allowed the standard to pass. Essentially Microsoft seems to be breaking the promises it made to the international standards community to get the standard through the process.

If Microsoft ship Office 2010 to handle only the Transitional variant of ISO/IEC 29500 they should expect to be roundly condemned for breaking faith with the International Standards community. This is not the format “approved by ISO/IEC”, it is the format that was rejected.

Unfortunately it's unclear what condemnation will bring. Even were the EU Commission to involve itself as Alex later suggests, it is unlikely that this would effect shareholder value in any meaningful way.

Alex also wonders at how such bad implementation can be allowed to happen:

So why — given the awareness Microsoft has at the top, at the bottom, and round the edges [for standards] — does it still manage to behave as it does? Something, perhaps, is wrong at the centre — some kind of corporate dysfunction caused by a failure of executive oversight.

It's nothing sinister I suspect. Cynically — it was nobody's job, and by that I mean no development manager or program management manager (or test manager for that matter) of any reasonable authority or seniority likely had it as a primary rewardable objective on their annual review. If the bugs were even filed, they were likely never deemed sufficiently important to fix during bug triage on the road to release-to-manufacturing. I saw bug triage meetings circa 1999 on the road to Windows 2000 where non-critical bugs that were effecting 10,000 beta customers were ignored because there were other bugs effecting 100,000 users. When you ship a product that has a consumer base of tens of millions, you learn different skills in the triage process. I suspect it's similar on the Office side of the company.

And Alex is completely correct, there would be no executive oversight pushing down from above on the Office development organization. The vice-president that published the open letter two-years ago making the promises has probably either (a.) moved on to other responsibilities, or (b.) assumed he had made the promise and someone else was to carry it out. It won't be on his review objectives either so he's still being well paid. So too with any other exec in the pipeline two-years ago. Even the standards team that worked so hard to get it through the ECMA and ISO processes will have moved on to other standards (and certainly wouldn't be so naive in the product-centricity of Microsoft to have accepted such an objective so far out of their control). The marketing team got its talking points two years ago. This is a cultural problem. The development teams within the company (i.e. the revenue generation engine) with a few exceptions in a few Web-related product teams just aren't tuned to deal with standards in a serious manner the way certain other vendors do.

Until there are serious lost sales to do with non-conformance from large government organizations, and the field organization starts to seriously yell, there will be no understanding in Redmond that it really mattered and that certain government officials may even have bet their careers on such promises. Even then, the first question in Redmond will [cynically] be, "How much do we sell to the Ministry of [Big Issues] in [Name-of-small-northern-European-country]?"

Alex finishes with the quote:

In short, we find ourselves at a crossroads, and it seems to me that without a change of direction the entire OOXML project is now surely heading for failure.

Failure indeed. I used the photograph at the top of the post two-years ago when I cynically predicted how this was going to go down over time. I summarized my opinions on the battle between ODF and OOXML and how Microsoft should have played the war as one of the examples in a standards primer I wrote around the same time. It will be interesting to watch how Microsoft responds to Alex's post. The world is indeed watching.

OSCON 2008: Open Source Software Economics, Standards, and IP in One Lesson

I was fortunate enough to give a talk at OSCON 2008 in Portland. I realized on Tuesday (the day before the talk) that my business tutorial slides that I normally use as a level set when consulting were just NOT going to cut it. Here are the OSCON slides:

Open Source Software Economics, Standards, and IP in One Lesson

view presentation (tags: oscon2008 ip ipr open source)

Developing a Standards Office for Google

I've been thinking about this since I published the Standards Primer a month ago. In the next two to five years, Google will be challenged by a technology standards effort that it will need to encourage or manage in its mission to organize the world’s information and make it universally accessible and useful. Such efforts cannot be erected quickly (as demonstrated most recently by Microsoft), and preparations for that day should begin in the near future. Google is also in a unique position to turn the standards development process on its head.

Here are a number of ideas for how Google can plan for the future in the most flexible and cost effective manner, and contribute a unique and valuable re-think of the standards development process. Well organized, a standards function can be an order of magnitude more effective at growing or defending a company’s business.

A Standards Office for Google

There are a set of strategic questions to be answered by a standards function at Google: What standards should be developed or encouraged to reduce the friction of organizing the world’s information and making it universally accessible and useful, or to open up new areas for growth? Is Android a reference implementation? An open source community? Does it require a standard? Are there micro-format standards that would make indexing web sites, libraries, or health records easier? What standards could others be planning that need to be managed because they threaten Google customer engagements or limit growth? Is a search results standard good or bad for business?

Google has a perceived public culture around serving customers, aggressively innovating, doing no evil, and having fun. These attributes should be celebrated equally loudly in their standards function which for most companies is portrayed as staid, conservative and dull.

As with all business functions, success in the standards function is a matter of organization, preparation and execution. To succeed, you need to:

• Have a strategic planning function that knows how best to recommend using the company assets to develop and respond to standards and standards-like programs and communities.
• Be able to find, organize, and educate potential participants and practitioners within Google quickly as the need arises.
• Be in a position to influence external participants and organizations through successive layers of trust, i.e. you need to be participating and be visible.

The following is the start of some considerations for a standards office at Google:

• Organize the standards office to be a small number of standards strategy staff. This group would have the responsibility for education, evaluation, organization and co-ordination, and forward strategy planning.
• Individual technical staff are generally the first group of people an organization discovers in their own ranks participating in standards. They are the first line of trust and influence. They are in the worst position when it comes to IP risk or reward. Standards practitioners often develop out of the initial group of standards participants. These practitioners are people that have an interest in the success of the standards efforts at a more organizational level and begin to see cross-standards influences.
• Educating participants and practitioners is key to ensure they know how to best get things done, protect the company’s investments, and how to keep others within the company informed of status. The standards office should be responsible for developing and delivering the educational materials.
• Organizing and co-ordinating the efforts of participants and practitioners is important if a company is to get the best return on the investment of having them involved. The initial participants are already building trust in their respective organizations. They are the front line and need to be supported in their roles.
• Google has offices in 27 countries. This is an opportunity to build trust with 27 national member bodies in ISO. Some countries have rules about participation (you need to attend so many of the past meetings to be eligible to vote for example). Some countries are more relevant than others, because they hold particular influence as well. But Google is in a great position to begin here.
• The standards team should be responsible for co-ordination with legal and corporate affairs and the business teams (product marketing and development) for standards-related efforts.
• The standards office should also be responsible for monitoring ongoing activities in the standards development world to watch for particular efforts that should be joined or managed.
• The standards office should develop forward thinking strategy in the standards space. Based on current products and future opportunities, what standards efforts might be undertaken at Google at what cost to best enable customers and users?

The investment need not be great. Google is in a great position to begin. But there are even more interesting ideas in the standards arena and Google is in a unique position to fulfill them.

Turning Standards on their Head

Taking a typical Google viewpoint of what could be done with infinite cycles, storage, and bandwidth when thinking about standards opens up new opportunities in the standards engagement space that come back to Google’s mission to organize the world’s information and make it universally accessible.

Developing interoperability standards is a labour intensive process. A working group meets to debate and discuss the nuts and bolts of the specification, but that act and its attendant work have lots of friction in the system:

• Meetings need to organize and track input documents, minutes, and output documents that pertain to the development of the specification that will become a standard.
• Specification development is often as complex as developing complex software, where a collection of document sources (text, markup+metadata, diagrams, tables), need to come together into a single known identifiable “printable” document that is easily available, indexed and searchable.
• Draft specifications need to be managed and balloted and the votes and comments from the balloting process need to be tallied, tracked and managed against document drafts.
• Interpretations (bug reports) need to be managed against published standards.
• Amendments need to be managed against the history and evolution of the standard specification.
• Many organizations develop software reference implementations for the specification, and the source code needs to be managed as a software project as well as against the specification (and its drafts).
• Some organizations develop test suites for conformance verification and validation against a standard, and again both the software itself needs to be managed, as well as the link to the specifications.

While different standards organizations exist to fulfill different missions, and so have different structures, policies, and practices, this lack of underlying consistency leads to a diverging base of information rather than a converging one. Regardless of the differing policies and practices of the standards development organizations, the friction in the system is common and causes lots of problems.

• Every working group and organization tackles these problems in their own ways, cobbling together the tools and processes that meet the standards development organization’s (SDO) policies and procedures. It is labour intensive, repetitive, and best practices seldom surface across diverse tools and organizations.
• History and knowledge are lost, both in process and content.
• Participation in the process is limited.
• Promulgation of the work can likewise be limited.
• The resulting specifications that become standards are generally not accessible and searchable in a consistent way.
• The quality of the specifications can suffer due to a lack of infrastructure and support.

In the end, regardless of policy and procedure tied to specific SDO, they all need the same basic toolset to support the process. Google has the building blocks in http://code.google.com and “Google Apps for your Domain” to develop http://standards.google.com providing the tool infrastructure and best practices to solve many of the friction problems faced by standards developers.

Regardless of an SDO’s policies and procedures, individual working groups can more easily deliver standards that meet the needs of the SDO and its constituencies in ways that would make the standards more discoverable, accessible, and useable. Using this as the opening olive branch, while developing a strategic standards office would put Google in an excellent position quickly.

Google is in an excellent position to begin their standards strategy function. The effort should be undertaken soon to provide the best most effective platform for success in this strategic area going forward.

A Standards Primer

I have recently had several long discussions about the motivations and machinations that surround the development of technology interoperability standards. Over the past few years, I've also captured a lot of ideas and experience on the blog. I pulled it all together into one place in the following paper, "Understanding Technology Standardization Efforts" (PDF 86.2K).

For the record, I was a long term participant in the POSIX and UNIX standardization efforts. I was a working group participant, balloted many pieces of the standards and their amendments, and participated in the management of the standards effort at the IEEE as both an inaugural member of the Project Management Committee and a voting member of the Sponsor Executive Committee. I was an international participant at ISO, as document editor, and participated on behalf of three different national body delegations (Canada, U.S., UK) over a number of years. I began my participation in 1989 as a customer (working for EDS with GM and the U.S. government as their primary POSIX-interested customers), but quickly ended up as a vendor, working for MKS developing a conforming POSIX.2 implementation that formed the basis of implementations from IBM, DEC, HP, UNISYS and Sun. In 1995, I put my money where my mouth was on the importance of applications portability, standards and the coming juggernaut of NT and co-founded Softway Systems, implementing the POSIX and UNIX standards on NT to enable UNIX applications to be directly migrated to the platform. A large amount of free and open source software was incorporated into the product. Softway Systems was acquired by Microsoft in 1999, and I worked there for five years. Over the years I've been in regular contact with people standardizing C#/CLI, the Linux Standards Base, and ODF.

Several friends and colleagues from the standards world have reviewed the paper and provided excellent comments. The paper is much better for it. All mistakes obviously remain my own.

Microsoft Office 2007 and Open XML: Lies, Damned Lies, and Statistics

Last week Joe Wilcox (Microsoft Watch) observed that Microsoft Office 2007 apparently doesn't conform to the Open XML standard (ISO/IEC 29500) that Microsoft has rammed through the system. Alex Brown has the full test here. No surprise. I've argued for the past year that the product must have diverged from the standard under construction. It's a normal thing in the standards world as Joe and Alex observe. They each challenge Microsoft to declare itself with respect to the standard and the future of the product.

But here's the problem: Microsoft already has declared itself. Last August Microsoft commissioned a study from IDC on the adoption of document standards. The "study" names Office Open XML as the obvious favourite. "Among the XML-based document standards, Office Open XML seems to be creating the most traction in the market." In the PR push leading up to the September 2007 votes on ISO/IEC 29500, Microsoft was already equating the standard with Microsoft Office 2007. That's what the sales field will be telling customers, with graphs culled from the "report". [srw — If you really want to read the report, follow the link from Mary Jo Foley's editorial. I still refuse to give the paid report link cred, small as it may be.]

Here's more writing on the ISO adoption and next steps:
Microsoft Claims Success with ISO and Open XML Standard

Microsoft Claims Success with ISO and Open XML Standard

Copyright © 2007 by Kordite

"Another key factor is the fact that people recognize the broad use of Open XML in the market as seen by the hundreds of independent implementations of Ecma 376." [Jason Matusow, Microsoft Director of Standards]

Think of the confusion if we only partially implemented the HTML standard. Okay — bad example. What if we only partially implemented a railroad standard? The track gauge would be correct, but the rail width was incorrect, or there was only one rail? Or maybe the track stopped before reaching its destination. Microsoft continues to maintain the Rovian perspective that a standard with "support" (their language is improving to "implementations") rather than complete conformance is good news for the industry. In this particular case it even ignores the very conformance statement in their own standard. It's only good news for Microsoft. It means lots of people are encouraged to do partial things around documents produced by Microsoft Office 2008. The economics is in the vendor's favour, not the consumer's. It defeats the actual purpose of de jure standardization. [In the industry, we call it a vendor specification regardless of standards body imprimatur.]

We now enter the next phase of the dance. Customers will discover they don't get the benefits that they thought they bought. A customer of note [likely government] or a consortia will put together a conformance certification program around the standards in the space. Brands and certifications will be the rule of the day. Microsoft will discover it needs to actually ensure their own products adhere [formally] to the standards they produced. The Microsoft Office team will discover conformance testing to a specification is (i.) hard work, (ii.) different than normal product testing, and (iii) that their product is drifting off the very standard they launched. (The .NET runtime team learned this a few years ago and I'm betting there are still conformance bugs logged against the product as "won't fix".) Implementation conformance will become important.

"You keep using that word. I do not think it means what you think it means." — Inigo Montoya, in the Princess Bride

Other writing I've done in this space:

• Office Open XML Conformance (A Lesson in Claiming Standards Conformance) [31 August 2007]
• Conformance and Certification: The ODF Standard and Microsoft's Office Open XML Specification [7 January 2007]

The OOXML Ballot Resolution

[Update (2008-2-25 13:45): There's an excellent press release from ISO that outlines exact history and next steps and requirements for this ballot.]

I have long maintained that technology standardization is commercial diplomacy and the purpose of individual participants (as with all diplomats) is to expand one's area of economic influence while defending sovereign territory. This week a lot of people are gathering in Geneva for the ISO ballot resolution meeting for Office Open XML (OOXML), Microsoft's Office product specification. The debate no doubt will be contentious.

Microsoft had a perfect opportunity to participate in the Open Document Format (ODF) standard's development at OASIS. They ignored that opportunity. The best time for technology standardization arises when a problem space is well understood, with sufficient real implementation knowledge to discern what works and what doesn't. Microsoft had arguably the best experience to contribute. They chose not to participate. Standardized document formats with multiple product implementations posed a threat to their Office business.

That threat became real when the Commonwealth of Massachusetts chose ODF as a basis for product procurement to best serve its citizens. Microsoft's response was not to adopt the ODF standard that already existed with multiple implementations (and continues to act as a hub for alignment with other international work like China's UOF standard), but to rush their own product specification into the standardization process.

They have over the two year process done a remarkable amount of work to bring the specification through ECMA to ISO, and have made great gestures to enable others to support the Microsoft specification.

But there's a problem.

Microsoft is an adjudicated monopoly in the United States. The EU continues to investigate possible abuse of their market dominance. (Market leadership and innovation are not what's being punished, but rather the abuse of a dominant position.) Microsoft can complain all they want, but the practices that enabled their success continue to plague them. We cannot collectively rewrite history. Microsoft is indeed held to a different measure. They have forfeited some of the freedoms that other companies enjoy. In many ways, they have lost our trust.

One can not judge Microsoft's newly declared preference for "openness" against the work they've done promoting their own product specification, but against their continued refusal to adopt ODF. In the end, OOXML as an ISO standard (with its attendant market confusion) will best serve the needs of Microsoft over its customers, and that's a shame.

Andy Updegrove has an excellent essay on his blog as we go into this week's ballot resolution deliberations. He takes a different approach. In it he argues that a particular class of standards should be held to a higher bar for acceptance, because they enable fundamental technology access in the world going forward. He makes an compelling case for why OOXML should be flunked out of the ISO process.

This promises to be a fascinating week.

FTC Settlement on Patent Abuse and Standards (and Open Source Implications)

Andy updegrove posted great news this morning on his standards blog. The US Federal Trade Commission (FTC) announced its resolution that a patent licensing promise made by a patent holder in a standards setting process is binding on a future holder of the patent.

National Semiconductor participated in an IEEE standards effort to develop the 100 Mbps "Fast Ethernet" specification in 1994. Two key (pending) patents were under their control, and they licensed them clearly, cleanly, and cheaply for US$1000 flat one-time fee to all takers. The patents changed hands, first to a group (2002) that wanted to change the licensing deal, then to N-Data (2003), a patent troll that was aggressively pursuing a changed expensive license.

Andy sums it up best:

"[T]he reliance upon promises made with respect to patents is of concern not only in the standard setting context, but with respect to open source software as well. The details of the settlement will provide significant guidance as to how the regulators would view similar conduct in an open source setting. Moreover, in the case of N-Data, the FTC has acted aggressively while acknowledging that the actions at issue might not rise to the level of violating relevant antitrust laws. In doing so, the Commissioners provide strong assurance to participants in standard setting that the FTC recognizes the importance of standards in the modern world. Finally, the details of the actual settlement demonstrate a willingness on the part of the FTC to craft a detailed and savvy set of requirements that addresses the realities of actual licensor-licensee conduct in the marketplace."

This is great news in the context of patent promises made to open source developers from the likes of IBM and Sun Microsystems, and through mechanisms like the Open Invention Network and the Linux Foundation's Patent Commons Project. It removes FUD slung around with respect to patents and intellectual property in both the standards arena and open source project communities. Each is a collaborative effort with significant economic importance and impact. Each will hopefully see the intellectual property landscape a little more clearly now.

Full details on Andy's blog.

GOSCON Discussion on Open Document Formats

Deb Bryant is blogging, which is great news. Deb is of course the creator and executive director of the Government Open Source Conference (GOSCON) that is held each year in Portland, OR.

The closing session of last Fall's conference was an executive panel on open document formats that included representatives from Sun, Microsoft, IBM, and Adobe. Deb's latest post points to the video of the panel, as well as the ongoing GOSCON forum discussion between the panellists. If you're interested in either the open document standards debate or government involvement in free and open source software, I would encourage you to have a read.