Once More unto the Breach

Patents, Red Hat, and Hibernate

Updated [30-June-2006, 14:45]:  Jan Kechel has created a website which is dedicated to collect prior art against this patent, but I didn't want his work lost in the comments. Please see: http://helpredhat.dyndns.org

News is breaking on claims by Firestar Software that JBoss (now Red Hat) infringed a particular patent in the Hibernate 3.0 software release.  Hibernate is an open source project started by JBoss and forms part of their JEMS product suite. 

• There's a good summary here with reference to the tactical timing of Firestar's attacks, and appropriate pointers to the claims and prior art debate that will rage.
• Bruce Perens has commented here, and once more argues that patent reform is the only thing that will save software from software patents. 

I participated in a panel discussion on the future of open source and IP hosted by SDForum in January.  The consensus of our group (mostly lawyers with a lot of experience in IP law and open source) was that the business economics of the situation would actually prevail before legislative change in a system constrained by deep pockets in other industries with different dynamics and entrenched views.  A few messy law suits, and rulings similar to the recent U.S. Supreme Court decision on damage and injunctions, and companies will begin to realize that software patents make bad business investments, and they will spend their dollars on more urgent and important things. 

This current lawsuit will run its course.  The other organizations that have open source patent protection mechanisms (e.g. The Open Invention Network, and the Patent Commons Project hosted by the Open Source Development Labs) will get to test and tune their organizational responses.  Microsoft will make statements about the value of intellectual property, and quietly fret about the lawsuit's outcome. Stories and counter-stories will be told. 

The most important thing to remember is that a patent is just a ticket to a negotiation.  Firestar may have thought it was going to be clever and make some quick cash on this suit in its timing.  One doesn't even need to ascribe malice and malevolence (or Microsoft as was suggested in one report) to the plaintiffs. 

I've been in enough executive meetings to virtually hear the discussion that would of ensued over whether or not this investment (i.e. the patent) was or wasn't viable, and whether or not it should or shouldn't be used and defended, and where the (fiduciary) responsibility lay.  Lawyers would be hired (increasing the investment) and the decision made to "see what happens."  Especially if Firestar's business is in a bit of a slump.

As was pointed out, however, Oracle and BEA and possibly even IBM and Sun (through the effects in the Java Community Process and the Java Persistence API) may have an interest to defend here.  Certainly IBM and Oracle would like to see Red Hat continue to thrive and prosper, even if BEA and Sun interests are merely defensive. 

These other companies need not even name themselves to the suit, but may start quietly helping in the background. Firestar's legal team may discover it is up against a far nastier negotiation than they anticipated, and regardless of their belief, opinion, and advice, the Firestar executive team may not have the stomach to fight if it means throwing considerable more money at the "investment" in the patent, or the negotiation.  SCO Group's "business" might be a law suit, but Firestar's may actually still be software, and their responsibility to invest in customers and products may quickly become more clear. 

Depending how creative Firestar's executive team is and how the product is structured, they could even turn this into a PR win by joining the community and broadening their customer engagement through selective participation in open source projects, rather than spending money on litigation.  This of course makes huge assumptions about how much imagination is left at Firestar. 

In all of this I remain fond of David McGowan's quote from the end of his May 2005 presentation (and paper) at the University of Toronto:

“If the F/OSS community wants to be in the commercial space, community members will have to learn to deal calmly with IP litigation.  The F/OSS production model will work where it makes sense, and it will not work where it doesn’t.  It’s really just that simple. Particular claims in individual suits—even one against a flagship program such as the GNU/Linux OS—will not determine the fate of the community.  Such cases present factual issues that will get resolved one way or another; they do not represent a crisis for F/OSS production as a whole.  Norm entrepreneurial rhetoric that plays off such cases should be treated as entertainment.  Enjoy it if you like it, take inspiration from it if you must, but don’t confuse it with the way things actually get done.”

Other patent related posts on this blog:

• A patent is merely a ticket to a license negotiation (Feb 2005)
• Corwin's Razor and the OASIS IPR Policy (Feb 2005)
• An Interesting Juxtaposition of Microsoft and Patents (Mar 2006)
• Massachusetts, Microsoft, and Open Document Format Standards (Sep 2005)
• IBM, Patents, and Open Source Software (Jan 2006)
• More on Open Source Risk Management and Licensing (May 2006)
• Open Source Software Business Models at USENIX (Jun 2006): This is the audio from the panel discussion with comments on software patents by myself, Mike Olsen (Oracle/Sleepcat), Miguel de Icaza (Novell/Ximian), and Brian Aker (MySQL) in the middle MP3.

Post to del.icio.us

Time to Market with Open Source, and License Considerations

Updated [12-Jun-2006]: Added some subheadings and a little editorial clean-up based on suggestions from friends. 

At the end of the USENIX panel, someone from the audience asked if one could use open source to provide a time to market advantage.  In the past couple of days, I have also had a discussion with a friend in regards to open source licensing and product architecture, and risks with respect to the GPL.  I wanted to outline how "flexible" the GPL can be, and demonstrate a product architecture that certainly worked for us and never exposed the Windows source base.   Far too many people have too many opinions about what "might" happen when a company uses free and open source software.  I thought a little experience might open up the view. A little developer education, and some sound technical architecture goes a long way. 

Interix and the Windows Architecture

Softway Systems was founded in September, 1995.  Interix was our product and allowed you to migrate UNIX applications to Windows NT.  The NT architecture was essentially a kernel, a collection of functional subsystems (I/O, Security, etc.), and a set of environment subsystems (Win32, POSIX, and originally an OS/2 one). 

Every application process runs as a client of an environment subsystem that provides the system services interface, and the services are then provided by the underlying functional subsystems via a fast local messaging system provided by the kernel.  Each environment subsystem is its own executable and runs as a separate process.  (If you look at the process list on Windows XP, you will see CSRSS — the Win32 subsystem — as well as PSXSS if you're running Interix.) 

Interix was a replacement POSIX environment subsystem, providing much more of the system service interface of a UNIX system than was captured in the original Microsoft subsystem. The original Microsoft subsystem was an exact implementation of the original POSIX standard. Interix included a complete collection of the UNIX commands and utilities, and a complete development environment that included the GNU C Compiler suite, and libraries from a number of sources (including Sun).  Interix also included a complete port of the X11R6 suite (with the exception of the X11 server itself).

The Collection of Licenses

Here's how the licenses worked.  We had:

• A [unique] Microsoft source license for the original POSIX subsystem code, and an ability to distribute derivatives (i.e. the Interix subsystem).
• Our own assets, both in the modifications to the subsystem, a number of utilities we created, plus changes to the open source utilities and libraries to internationalize them and otherwise bring them into alignment with the relevant standards.
• All manner of free and open source software.  We had 300+ utilities and libraries in the product covered by 20+ different licenses, ranging from the GPL, MIT Athena, BSD, and Sun licenses (ONC RPC code) to some more esoteric home grown licenses, to some true public domain code.  (We shipped a derivative of the Public Domain Korn Shell, pdksh.)   

We shipped our separate product as a collected work under our Interix End User License Agreement.  [Copyright law supports the idea of collected works such as magazines and encyclopedias having their own copyrights above and beyond the individual copyright associated with each individual component.]  We charged a license fee for the product (collection), but some of the tools (covered by licenses like the GPL) were provided as part of the distribution for customer convenience. 

Interix is the core of Microsoft's Services for UNIX (SFU) product now.  For the past year or so, the SFU product has been a free download from Microsoft.  Parts of it are to be integrated into Windows Server 2003 R2 if I recall correctly.  (I've a blog entry that describes that plan as of last September). If you download and unpack SFU 3.5, you should be able to see all the copyright notices (attached here for convenience) from all the open source software components in a single file in the /docs directory of the install root.

We respected the terms and conditions of each component or library's license.  Copyrights were maintained.  Code to our derivatives that we needed to make available was published, e.g. our changes to the RCS tools and gcc suite were all published because they're licensed with the GPL.  That's what the licenses required, but we also wanted the community engagement of collaboration and enhancements.

The Business Economics of Community (with a gcc example)

Using all this source code from the open source world (and we didn't call it "open source" back when we started the company in 1995) gave us huge advantage in time to market.   The code was also robust as it had stood the test of time and the real world across a host of architectures.

We didn't just take from the community, but also engaged with the community.  We worked a lot of changes back to the pdksh team, had a developer that still maintained the ex/ed editors, and worked with the gcc community to contribute our changes and bug fixes back. This is obviously not the same level of business engagement as MySQL running its community around its own project, or Red Hat and its engagement with the Linux community.  At the time we had different goals and requirements. 

Our use of gcc is interesting on a couple of levels: one from a community perspective and the other from a technical architecture and license obligation perspective.  First we'll tackle the community economics.

When you download gcc from the web, it's a bundle of the compilers, linker, binary formats library, assembler, and debugger all in one tidy package.  Our compiler developer was an 18 veteran in compilers and operating systems (formerly of HP) and made a coherent set of changes to the tools to get them to behave properly developing debuggable executables for the Interix subsystem.   When we began to contribute changes back, we discovered FIVE different communities hiding behind that single download with varying degrees of interest in accepting our changes.  It was quite the negotiation. 

In the end, we tried to hire Cygnus (as the gcc experts) to make/facilitate the changes, but in the late '90s this would have cost US$100K+ and they couldn't start for at least 14 months as they were so backed up with work.  (This was prior to their acquisition by Red Hat.)  We finally hired Ada Core Technologies, as they too employed a primary committer on parts of gcc that could best facilitate a set of changes across the tool set back into the core.  It was considerably less expensive and they could begin immediately.

The gcc compiler also provided technical architect challenges that might be subtle for some.  We used gcc to build our own world (with the exception of the subsystem), because the gcc compiler in those days was better than the Microsoft compiler, and we needed to use gcc if we were to use gdb.  (An artifact of the environment subsystem world in the late '90s meant we could not use Visual Studio and its debugger.)  Because using the gcc headers and libraries would have effectively attached the GPL to our own programs, we used the gcc compiler in conjunction with our own libraries (derived from the Microsoft C library) and our own headers.  (We had a lot of experience building standards-based portability headers.)

It wasn't that we were unwilling to work with the GPL or use the GPL in conjunction with any of our code that made us cautious.  We first had the Microsoft asset we were required to protect, and we also wanted any code we licensed out into community to be by choice and not through the license obligations.

Recognize that these contributions back to the community weren't out of altruism, especially considering the cost in engineering time for both us and hiring ACT developers.  This was a deliberate business decision.  We wanted the engineering expediency of working in a collaborative world.  Despite the initial time to market advantage of using open source, if we had continued to live on our own code fork our team would need to make months of changes when a new version of gcc came out, rather than a few weeks of changes if the Interix related contributions were accepted.

This is the economic strength of the open source collaborative development world — you are distributing the cost of maintenance and development across a large number of players while improving the software's robustness through a test bed of users stretching the software in new ways. Considering all our investment, it was still cheap when compared to developing and maintaining a compiler from scratch, which wasn't a core competency (despite our talent), nor the primary value proposition to our customers. To us, this was no different than IBM's investment in the Apache web server.

(Frankly, in hind sight, I think we would have shared the subsystem code as well, if the Microsoft license hadn't prevented us from doing so.  Our value proposition to our customer was in the distribution build, test, and packaging, and not necessarily in the source code itself.) 

Developer Education and Oversight

As a set of developers, we were essentially responsible for our own legal due diligence.  Every Interix developer understood the ramifications of the GPL.  While überconservative lawyers at large software companies may well fret about the legal ambiguity of some clauses, spreading FUD before them, the average developer can read and understand the license pretty quickly. 

Every Interix developer knew that before using software in the product covered by a new license, they needed to get approval from management — which meant me and the core senior developers that had lived in this space for a while.  We always took the most conservative interpretation.  We had already seen what AT&T did to our friends at BSDI with a law suit.  (This was also why we began with the 4.4BSD-Lite distribution of the source for much of our early utility base — it was the legally blessed clean version.)

I am not advocating that you skip getting legal advice.  We did check licenses from time to time with out external counsel.  But your developers need passing familiarity with the licenses and copyright anyway, so you can actually do a lot of the work from an architectural point of view prior to involving lawyers to both expedite the conversation and save on legal costs.  Few companies have the legal exposure of a Microsoft and require that level of legal due diligence before developers touch code.   That's their problem. 

Mortice Kern Systems was just up the road from us as well, and while we considered our product to be in a different space from theirs (we were about applications migration and they were the UNIX tools on Windows), we didn't want claims of source code infringement as many of us had worked there.  We were religious in our source code control so as to be able to demonstrate code pedigree and cleanliness should the need arise.

We also had the Free Software Foundation keeping us honest.  They would have loved for us to have mistakenly introduced software covered by the GPL into the Windows code base.  Our survival as a start-up depended upon the Microsoft license. Once inside Microsoft after the acquisition, we were even more careful.  And then you had the Microsoft legal team looking over our shoulders as well.  We had email discussions twice with the FSF when we received notification that we hadn't yet published our changes.  (There was always a latency in our making the code available while trying to get the product out the door.)

The FSF also maintains the inbound contribution assignments for gcc.  I signed these assignments twice for the work of my compiler developer, once as vice-president, R&D at Softway Systems, and once as the product unit manager at Microsoft.  (And you can imagine the legal oversight I had the second time from Microsoft Legal and Corporate Affairs.)

In Closing

So GPL code isn't the end of the world as we know it.  You can get highly creative in how you consider its use in a technology architecture.    The economic benefits of free and open source collaborative development are huge.  Contributing back is essential.

The investment in developer understanding is necessary regardless of your use of free and open source software.  (Consider all the other opportunities your developers have to bring in code from third parties and the "outside", regardless of open source.)  Scanning tools can't replace developer education and an understanding of the goals.  Frankly, it's just not that difficult to educate developers. 

Indeed, collaborating and developing open source software may be a far more efficient, less risky, way of rapidly solving your customers problems as you develop products.

Post to del.icio.us

Open Source Software Business Models at USENIX

Updated [10-Jun-2006]: Added a link at the end to Brian Aker's blog commentary on the panel. 

Brian Aker (MySQL), Miguel de Icaza (Novell/Ximian), and Mike Olsen (Oracle/SleepyCat) joined me on a panel on open source software business models at the USENIX Annual Technical Conference in Boston last Thursday (1 June, 2006). At last year's main technical conference, I gave an invited talk (587.6K) on a similar topic.  This year, I thought it would be more interesting with people instead of slides. 

The entire 90 minute panel was recorded.  I've broken it into three ~27MB MP3 files for posting, and provided a key with (very) rough times (hour:mins). 

• Download HeadUSENIX.mp3 (27.3MB) covering the first 30 minutes.
  • (0:00) The First Act begins with introductory remarks.
  • (0:02) Introductions by Brian, Miguel, Mike.  Miguel remarks he might not do an open source company "next time".  Brian suggests, "Selling software at this point is like putting out a tip jar."   
  • (0:08) The panel discusses the challenges of growing out the community and growing the company.  Mike discusses the early ownership of the asset and dual licensing. 
  • (0:11) Miguel talks about early days at Ximian, and the discovery that one of the more valuable assets was Red Carpet, which wasn't the focus of the company at all. 
  • (0:14) Miguel discusses the value of Skype versus JBoss on acquisition and whether that had to do with the fact that JBoss was open source software, and Brian's responds that it may have had more to do with the number of people that could be monetized between a VoIP company versus an app server.
  • (0:18) A discussion about community contributions, from the ratios to the value of the contributions, and the nature of the community contributing. How the maturity of a community may effect where and how contributions happen is discussed.  Contribution value from Windows users is also discussed (with some amazing references). 
  • The value of contributions from an intellectual viewpoint is discussed. Mike (referring to MySQL) observes, "They can hire the smartest guys for their product and project.  You can shop for brains wherever they happen to be." Brian confirms that MySQL has people in 26 Countries, and, "There is no point in the day where someone isn't awake working on MySQL."
• Download MiddleUSENIX.mp3 (26.3MB) covering the middle 30 minutes.
  • The Second Act starts with a discussion about Licensing and how each company thought about licensing, the community it would enable, and the  leverage it would provide as a business.  Interesting commentary on the inability to dual license Linux at this point, and how valuable the Samba project really is.  Some discussion on subscription value in the business.
  • (0:40) The patent discussions begin (and we get our first audience question asking about opinions on provisional patents)!  What we've collectively seen from a business perspective and an investor perspective is discussed, with little straying into software patents being "right" or "wrong".  This IS a business panel after all, and we point out why they're not necessarily a good investment, and how the different companies have used them. 
  • (0:49) Audience questions finally start in earnest:  What were the unexpected things that happened with respect to your business in the community or with open source?  Mike discusses surprises about the cost of community engagement, and not pursuing traditional marketing tactics. Miguel discusses early learnings at Ximian.  Brian talks about early things he did at MySQL, and comes back to the importance of committed communications with the community.
• Download TailUSENIX.mp3 (28.8MB) covering the final 30 minutes.
  • (0:56) The Third Act continues with excellent questions from the audience.  Audience question:  Are there good ways of working with international clients/customers as a small companies?  (The question comes from someone that has not had good experiences.)   Mike and Stephe offer up different perspectives. 
  • (1:03) Audience question:  How do you deal with QA to ensure your customers come back, especially when the support and maintenance from a commercial system can be inferior to the community itself?  (The question comes from someone that has had bad experience with the Red Hat Network.)  Brian answers in terms of the competition that is enabled in an open source environment. 
  • (1:07)  Audience question:  The panel is asked to speak to the differences between collaborative organizations around “open source standards” (e.g. The Gnome Foundation) versus more traditional standards organizations.  Some discussion ensues about standards and open source in general. (I must apologize.  This is a topic near and dear to my heart and I got a little long-winded here.)  This discussion was kicked off by Andy Oram (O'Reilly Media) who wrote up an audience view of the panel. 
  • (1:16) Audience question: How would you maximize your buyout opportunity using open source?  This question carried us almost to the end.  Each panelist and the moderator came back to passion and conviction and the need to build the company you want, regardless of an acquisition end game.  Miguel discussed some of the accounting involved for a large company investing in acquisitions over R&D, and how he thinks about it.  Mike discussed the emotional investment in building a company and the ability to execute compared to trying to “build something flip.”   Brian talked about different motivators and Adam Bosworth's “6 F's”.  Stephe filled in his own experience (with odd references to Canada Day).
  • (1:25) A last quick audience question (with minutes to spare):  Can you use open source for time to market advantage in building a company?  Stephe takes this one with a discussion of the use of open source in the development of Interix at Softway. 

I want to publicly thank my panelists.  They were an awesome experienced panel.  They have all been in the game long enough, and each with remarkable success so as to be able to properly discuss free and open source software in the context of a business and its growth and execution.  There are few panelists I would trust to deliver the experience shared here without needing to wrap it in positions and agendas.  Each of these gentlemen shared their experiences frankly and brilliantly.  Thank you!

Here are Brian Aker's comments on the panel as well.

L-to-R: Brian Aker, Stephen Walli (standing), Miguel de Icaza, and Mike Olsen.

Picture courtesy of Tom Christianson.

Post to del.icio.us