Once More unto the Breach

Optaros Publishes its Free and Open Source Software Policy

Today we published the Optaros Free and Open Source Software Policy. This policy defines our expectations on how Optaros employees work with the open source community at large. Think of this as our analog to the Debian social contract both with our employees and the free and open source software community at large.  (I appreciate it doesn't go as far as the Debian social contract, however, as a consulting services company our customers may sometimes require us to do work in ways that we can't simply publish, and we need to keep a foot in both the free and the open source worlds.) 

While it reads in slightly heavy-handed legal English, the intent is to present it in terms any enterprise developer or enterprise lawyer can understand. There is a huge amount of intellectual property FUD cycling around the industry since the SCO Group started its legal dispute with IBM.  We wanted to reduce the discussion back to a very simple idea — as a business you have software assets for which you are responsible and what you choose to do with them for the best overall benefit to your stakeholders will depend upon the business you are in. We have a breadth of development experience and business experience in this space and wanted to give people a place to start having better discussions based on business pragmatism rather than fear.

We published the policy under a Creative Commons license. We want people to be able to create their own such policies in this space and re-use the parts of our policy that make sense to them without asking permission.

The one part of the policy that will be discussed a lot I'm sure is the section on our community commitment of time to community projects. First, we recognize that free and open source software forms the building blocks of the solutions Optaros develops for our clients. So we need to give back. Second, every consultant has "bench" time (a.k.a. "beach" time depending upon one's point of view) when they are not working for a client. From the business's point of view we would love to have everyone billable all the time, but it simply never works out that way. Likewise, we can't guarantee in advance that we know how much time a consultant can give back. So we have taken the approach that when an employee isn't working on a customer project, they will be working on free and open source software projects that interest them. Think of this as our Google 20% project.  The comments on Chris's post (and Joe's) are right — it's about culture and we want to set the culture at Optaros early.

I appreciate this policy may not go far enough for some in the community, but I think it's a great start as open source software spreads itself deeper into the enterprise, and I look forward to comments and discussions. 

Update (30-Jun-2005): Here are Dave Gynn's views on the policy.  (He also works at Optaros.)
Update (1-Jul-2005): Groklaw has kindly posted our policy and some commentary here.  There's an ongoing discussion at Groklaw. 

The Massachusetts Software Council OSS SIG

Last Friday I attended the first meeting of the Mass Software Council's Open Source Software SIG, sponsored by Novell and Ascential Software.  Dan Bricklin was master of ceremonies and one of the organizers.  He's started the blog and wiki for the group, and there are recordings and slides from the sessions.

It was a well attended event.  (I have to admit being very impressed that Jack Messman attended.  Even if he was merely trooping the Novell OSS flag, it's still an impressive trooping.) 

As a first event, it was great.  There was a breadth of presentations:

• Open Source Licensing was introduced by Ira Heffin and Karen Copenhaver.
• Open Source Business Models was a panel that included our own r0ml (Optaros), Douglas Heintzman (Director of Tech Strategy for Software Group, IBM), Nick Carr (Marketing Director, RedHat), Paul Doscher (CEO, JasperSoft), and Eric Newcomer (CTO, IONA)
• A presentation on Open Source Communities from Nat Friedman and Miguel de Icaza, who have an obvious wealth of experience here.
• Marc Fleury (CEO, JBoss) was the lunch keynote.

All the presentations were entertaining and informative, each in their own way.  I especially liked Karen Copenhaver's example for teaching lawyers how to think about open source, by pointing out that no lawyer starts with a blank piece of paper when they start to write a contract, and the reason and rationale behind using what's gone before is that it is a proven way to do things and has been peer reviewed.

Marc Fleury's presentation raised a few hackles in the audience.  He makes perhaps too much of the "professional" versus "amateur" nature of part of the community discussion.  Tone of voice can be everything, and there was an attitude that if you're not being paid for the work, then it is "obviously" not commercially viable, and so some how being paid must make for better software.  I do his logic an injustice here, but it doesn't account for so many examples from the past 40 years of source code sharing and business, and from all the valuable scientific work that has come from "amateur" efforts over history.  Regardless, it was an entertaining lunch presentation. 

I look forward to further SIG meetings.  The organizers should be proud.

Open Source Software and Source Code Inspections

The other day, the vice president of marketing for VA Software (parent of sourceforge.net) said in an article announcing their new Black Duck alliance, "developers had to rely on reading every line of code before marketing their projects, a time-consuming process."  I thought this showed a certain amount of irony.

When Eric Raymond first wrote "The Cathedral and the Bazaar" he coined Linus's Law, roughly stated that "Given enough eyeballs, all bugs are shallow."  This is an axiom in the free and open source world.  Certain threatened vendors love to point out that most users of the software never want to look at the source code, but the project community itself lives for that ability to read, fix, and extend.  Linus's Law, however, applies "after the fact".  The software exists, a bug is noticed, and then the law comes into effect.  (There is likely a certain about of "The Wisdom of Crowds" happening here, as outlined in James Surowiecki's book.)

However, the real strength of good software actually rests on inspection before the fact.  Good software is written by good software developers regardless of licensing concerns.  In 25 years of being around a lot of good developers working on free and open source software projects, closed and proprietary products (at MKS, Softway Systems, and Microsoft), and enterprise applications development (retail and manufacturing), one thing has stood out as a defining measure of software quality.  (Quality to me is fitness for use — it works and solves the customer's problem — and maintainability — it can be easily fixed and extended.)  Every project that succeeded well, had at least one and generally two absolutely disciplined anally retentive developers that read EVERY line that went into the software.  They know of no other way to develop software. 

It didn't matter how elegant they were themselves at the writing act either.  They always ensured someone else checked their work as well before it was checked into the tree.  Disciplined developers already ensured there was adequate tool support for automated test and regression, software build recipes, versioning, configuration management, and debugging and profiling.  But I've seen projects with tool support and knowledgeable developers that still didn't perform to the level of software that was "owned" by the developers that inspected everything regularly. 

These developers would also use certain static checkers as part of the regular build cycle, whether it was lint in the early days, or some of the sophisticated scanning tools used at Microsoft.   But the difference, again, was the projects with a rigorous discipline of review still built better software.  I think this is because the developer has perspective and context that can never be built into a static analysis tool.  Tools can find obvious portability breakage, or security related buffer overflow problems, i.e. things that are likely syntactically based, but a human can understand the semantic content of the code in front of them.

There's even research to back this up:

• Walcélio Melo, Forrest Shull, Guilherme Horta Travassos, "Software Review Guidelines", Technical Report ES 556/01, COPPE/UFRJ, August 2001.  "Code review is more effective than test because in review the faults in the code are found directly, while testing uncovers only the symptoms of problems, requiring debugging to find the direct cause.  The seriousness of the wrong behavior by the system does not have a relation to the type of mistake, since even simple mistakes can cause complex behaviors."
• Reidar Conradi, Amarjit Singh Marjara, Øivind Hantho, Torbjørn Frotveit, and Børge Skåtevik, "A Study of Inspections and Testing at Ericsson, NO", a rewritten edition of the paper presented at PROFES'99 (Oulu, Finland, June 1999). 

  From the Conclusion:

  "Software inspections are indeed cost-effective: They find about 70% of the recorded defects, take 6% to 9% of the development effort, and yield an estimated saving of 21% to 34%. i.e., finding and correcting defects before testing pays off, so indeed “quality is free”."

  and:

  "Individual inspection reading and individual Code Reviews are the most cost-effective techniques to detect defects, while System Tests are the least cost-effective."

So while the existing good developers already live in a culture of inspection and review, let's hope the next generation don't learn to believe they can rely on tools instead of inspections and miss the mark.  Hopefully with the continued growth of free and open source software development, where the culture of inspection and review is almost self wired, the discipline will grow. 

Open Source Risk Management in the Enterprise

Updated [18-Jun-2006]: This work was expanded into a presentation and a pair of white papers.  You can find all the refernces in this blog post. As always, all materials available under a creative commons license. 
 

One of our sales team here at Optaros asked me how I would respond to questions from an enterprise customer about legal risk assessment with open source software in general, as well as in the context of the SCO Group and IBM litigation. 

Here was my answer, in case others find it useful:

SCO/IBM is a about copyright misappropriation between two vendors.  SCO claims IBM used SCO property in its contributions to the Linux operating system.  SCO still needs to establish that its property has been used, as well as establishing that it couldn't be used through the contracts in place at the time between the companies. The Canopy Group, the parent company of SCO Group, has a long history of acquiring the technology assets of a failing company and litigating against a vendor.  (They successfully did this to Microsoft and CA.)

When SCO Group sent letters to a very large number of Linux customers threatening to sue them, then started trying to sue Autozone and Daimler-Chrysler, they were attempting to put pressure on IBM to settle.  It has back fired.  Several large vendors (HP, Novell, and Microsoft) all took the opportunity to loudly offer indemnifications against intellectual property lawsuits in relation to their products. HP and Novell did this for their respective Linux offerings and Microsoft did it for their own product portfolio to try and differentiate their commercial offering from a generic "open source" threat.  IBM loudly did NOT offer an indemnification around Linux, saying it wasn't necessary.

In a more general discussion around open source and legal risk to the enterprise, I would suggest that an enterprise evaluate the risk in relation to its risks around other contract areas with respect to software in general:

• A company stands a larger risk of being sued over license counting issues from a commercial software vendor than they do over intellectual property issues (copyright) and open source.  License management and counting users is a notoriously difficult space.  Both vendors and customers can feel they're being cheated.  There's no good general way to manage user license counting across diverse vendors and products. Vendors regularly litigate in these spaces. Some even wield the BSA as a stick and call for audits.  Open source licensing reduces risk here.
• The only lawsuit to date is the SCO vs. IBM suit.  It isn't seminal — it's an aberration based on a particular company's historical behavior. In a multi-billion dollar market place, Linux has been in active deployment for 10 years, MySQL for 5 years, and Apache for 15 years.  This is the only lawsuit in which a customer was threatened and then only as a legal tactic to pressure another vendor. In a very few other cases, inappropriate use of software licensed under the GNU General Public License (GPL) has been at issue between two vendors or between the vendor and the project.   In every case, the vendor has respected the license, and either published their modifications or modified the project code to no longer use the software covered by the GPL.  There have been no product catastrophes or outrageous damages.  No enterprise end customers to my knowledge have ever had a problem.

From a software development perspective, there is concern of late over whether or not your own software development efforts have been somehow "tainted" with "viral" code and would need to be therefore published at large — and concern over what sort of software development tools and practices need to be put in place to prevent such risk.  Ideas around free and open source license compliance tracking have been raised. Again one needs to consider the risks in context.

• A vendor that has a software product distributed in the marketplace needs to be more sensitive to what their license responsibilities mean to their product than an enterprise simply deploying software in-house. 
• Every developer I've known in 25 years of software development understands that copying code without checking the license can lead to consequences that will affect the company's ability to ship its product.  The developers may not be copyright lawyers, but they know to call in project management for a review. This also applies to code samples found in books and developer network portals (e.g. MSDN with its once confusing array of copyright statements), as well as to third party licensed software (e.g. C++ library and framework source code), not just to open source software.  In other words, this is not specific to open source. 
• The requirement that software source code be either published and licensed (or the software withdrawn) under some open source licenses such as the GPL is triggered on distribution of the software.  This is very relevant to a software development company, but not to an enterprise that is deploying the software internally.  Licenses such as the Apache license or BSD license don't require publication of the software source code at all. Even software development companies, however, have demonstrated they can ship products based on free and open source licenses such as the GPL without "risking their businesses", gaining very different benefits that outweigh the risks of publishing the source code.
• I would sooner invest in better software development tools for testing, debugging, and release and configuration management to improve software quality and reduce software maintenance costs than detecting code taint of specific of open source projects (while ignoring developer portal code and published code in journals, books, and magazines, or worse still software that came into the company from a third party that will never be finger printed and identifiable from a tool.)  The risks don't merit the expense in context.
• There are no tools that can feasibly be created that can detect patent taint. 

Here's another way to think about taint and compliance:  when I was vice president, R&D at a software company, the development team was juggling a Microsoft source code license, our own software assets, and every imaginable free and open source license.  If the Microsoft code base was compromised via source code taint from a GPL project, we would be out of business.  We didn't have deep legal pockets to ride out a storm.  Every one would simply be sent home and our investors would look for new investments.  Because we were shipping product on the Microsoft platform, the open source community was very vigilant about possible license abuse. 

My developers cleared any license for use prior to working with the open source code with product management. We checked with our general counsel once a product cycle (and frankly as I recall we explained it to them the first time around).  We kept a simple manual register of licenses that covered the software we used.  When we shipped the product with our own product EULA, we included a file that detailed all the other licenses and what parts of the product they related to, however, it was our EULA that the customer cared about.  We ran a simple scan for copyright notices as part of our product build process to ensure nothing had been inadvertently missed in the manual process.  This was in a very complex product using upwards of 350 tools from the open source world covered by on the order of 25 different free and open source licenses, many of which are not formally registered with the Open Source Initiative (which didn't exist when we started the company).  [srw — You can still find the file in /docs/CPYRIGHT.TXT in Microsoft Services for UNIX V3.5, available here.]

We did not expose all of our source code.  We contributed source code enhancements and bug fixes back in several different communities (e.g. ed/ex, PD KornShell, the GCC compiler suite), as well as publishing our changes for GPL tools (the GCC compiler suite, the RCS version control suite).   

A little education goes a long way here.  It is not as complex as people fear it is with respect to using open source software, and frankly the commercial software world is riskier for the enterprise with respect to litigation.

Monad, Lucovsky, and Integrated Innovation

"It will take three to five years to fully develop and deliver," said Microsoft Senior Vice President Bob Muglia this week at Tech Ed 2005.  He's talking about the Monad command line shell, quoted in this article, via /.   Here's the Mary Jo Foley reportage. 

Here's what Microsoft's CEO said about tightly integrated innovation a year ago: 
"... we believe in something that we call integrated innovation. We not only need to enable you to do new things, or to do things you didn't think were possible today, we think part of our value is integrating these innovations in ways that make your life simpler, lower cost, and easier to take care of."

Here's what former Microsoftie Marc Lucovsky had to blog about Microsoft shipping software:  "I am not sure I believe anymore, that Microsoft "knows how to ship software". When a Microsoft engineer fixes a minor defect, makes something faster or better, makes an API more functional and complete, how do they "ship" that software to me? I know the answer and so do you... The software sits in a source code control system for a minimum of two years (significantly longer for some of the early Longhorn code)."

Clayton Christensen's research points out that in a market that is underserved, i.e. the product isn't "good" but fundamentally enables the customer, then the company that can control all the variables and deliver a tightly integrated solution best serves the market and "wins".  Once the technology in that market matures and the customer is overserved, the pressure to componentize is great, and indeed the component players can "win" against the tightly integrated encumbant.    Think ... Firefox!  Or OpenOffice.

It's just a command line shell.  You haven't actually reinvented systems management for the heterogeneous masses, despite how innovative I know you've been.  Impress your customers.  Ship it!