Open Source Software and Source Code Inspections

The other day, the vice president of marketing for VA Software (parent of sourceforge.net) said in an article announcing their new Black Duck alliance, "developers had to rely on reading every line of code before marketing their projects, a time-consuming process."  I thought this showed a certain amount of irony.

When Eric Raymond first wrote "The Cathedral and the Bazaar" he coined Linus's Law, roughly stated that "Given enough eyeballs, all bugs are shallow."  This is an axiom in the free and open source world.  Certain threatened vendors love to point out that most users of the software never want to look at the source code, but the project community itself lives for that ability to read, fix, and extend.  Linus's Law, however, applies "after the fact".  The software exists, a bug is noticed, and then the law comes into effect.  (There is likely a certain about of "The Wisdom of Crowds" happening here, as outlined in James Surowiecki's book.)

However, the real strength of good software actually rests on inspection before the fact.  Good software is written by good software developers regardless of licensing concerns.  In 25 years of being around a lot of good developers working on free and open source software projects, closed and proprietary products (at MKS, Softway Systems, and Microsoft), and enterprise applications development (retail and manufacturing), one thing has stood out as a defining measure of software quality.  (Quality to me is fitness for use — it works and solves the customer's problem — and maintainability — it can be easily fixed and extended.)  Every project that succeeded well, had at least one and generally two absolutely disciplined anally retentive developers that read EVERY line that went into the software.  They know of no other way to develop software. 

It didn't matter how elegant they were themselves at the writing act either.  They always ensured someone else checked their work as well before it was checked into the tree.  Disciplined developers already ensured there was adequate tool support for automated test and regression, software build recipes, versioning, configuration management, and debugging and profiling.  But I've seen projects with tool support and knowledgeable developers that still didn't perform to the level of software that was "owned" by the developers that inspected everything regularly. 

These developers would also use certain static checkers as part of the regular build cycle, whether it was lint in the early days, or some of the sophisticated scanning tools used at Microsoft.   But the difference, again, was the projects with a rigorous discipline of review still built better software.  I think this is because the developer has perspective and context that can never be built into a static analysis tool.  Tools can find obvious portability breakage, or security related buffer overflow problems, i.e. things that are likely syntactically based, but a human can understand the semantic content of the code in front of them.

There's even research to back this up:

• Walcélio Melo, Forrest Shull, Guilherme Horta Travassos, "Software Review Guidelines", Technical Report ES 556/01, COPPE/UFRJ, August 2001.  "Code review is more effective than test because in review the faults in the code are found directly, while testing uncovers only the symptoms of problems, requiring debugging to find the direct cause.  The seriousness of the wrong behavior by the system does not have a relation to the type of mistake, since even simple mistakes can cause complex behaviors."
• Reidar Conradi, Amarjit Singh Marjara, Øivind Hantho, Torbjørn Frotveit, and Børge Skåtevik, "A Study of Inspections and Testing at Ericsson, NO", a rewritten edition of the paper presented at PROFES'99 (Oulu, Finland, June 1999). 

  From the Conclusion:

  "Software inspections are indeed cost-effective: They find about 70% of the recorded defects, take 6% to 9% of the development effort, and yield an estimated saving of 21% to 34%. i.e., finding and correcting defects before testing pays off, so indeed “quality is free”."

  and:

  "Individual inspection reading and individual Code Reviews are the most cost-effective techniques to detect defects, while System Tests are the least cost-effective."

So while the existing good developers already live in a culture of inspection and review, let's hope the next generation don't learn to believe they can rely on tools instead of inspections and miss the mark.  Hopefully with the continued growth of free and open source software development, where the culture of inspection and review is almost self wired, the discipline will grow. 

Comments

[Jing Jing]

Besides the benefit of reducing cost and effort, inspection is also a very efficient and effective process in training the beginners or newcomers to increase their technical and engineering skills. It puts the quality concept into everyone's mind from day one.