The statistics are dramatic. Most code starts life with around five to 10 bugs per hundred lines of code. Without inspections figure on finding about 95% of those preshipping. For a 100KLOC program that means shipping with hundreds of defects! (Note that some sources, such as Stan Rifkin, "The Business Case for Software Process Improvement," Fifth SEPG National Meeting, claim far worse numbers, on the order of six shipped bugs per hundred lines of code--six thousand in a 100KLOC project.³ My unscientific observations, coupled with data from private conversations with Capers Jones, suggest that firmware generally doesn't sink to that abysmal level.)

It's important to understand that testing, as implemented by most companies, just does not work. Many studies confirm that tests exercise only about half of the code. Exception handlers are notoriously problematic. Also consider nested conditionals: IFs nested five or six levels deep can yield hundreds of possible conditions. How many test regimes check all of those possibilities?

In the March issue, I wrote about cyclomatic complexity which, among other things, tells us the minimum number of tests needed to completely exercise a function. In researching that article I found that one function in the Apache web server scored a complexity index of 725. Who constructs nearly a thousand tests for a single function?

Another analysis of the testing problem is scarier. In an article printed in Crosstalk, Watts Humphrey shows that a program with 100 decisions can have nearly 200,000 possible paths.⁴ With 400 decisions that explodes to 10¹¹ paths, each requiring a test! Although he describes a pathological worst-case situation, the implications are important.

Some approaches to test are much more effective. Code coverage, used in the avionics and some other industries, can approach 100% testing, albeit with huge costs.

Test Driven Development advocates also claim good results, although reliable supporting studies are hard to find. However, anecdotal evidence does suggest that most of the agile approaches, which have a disciplined focus on testing, greatly improve on the 50% code test ratio. The agile approaches demand, correctly, automated tests that cost little to run. But in the embedded space that's hard. Someone has to watch the display and press the buttons. I've seen some very cool ways to automate those activities, and some agilists successfully build mock objects to simulate the user interaction. So-called virtual prototypes, too, are gaining market share (see the products from Virtutech or VaST, for instance).

The bottom line is that test is a huge problem. However, a well-run inspection program will eliminate 70 to 80% of the bugs before any testing starts.

When I was a young engineer, my boss demanded that all schematics go through a formal design review. It was expensive to respin boards, and even costlier to troubleshoot and repair design mistakes. At first I found this onerous: a colleague was going to critique my work? How mortifying when someone found errors in my circuits! But a little time spent reviewing saved many hours downstream. Even better was that I was able to tap into the brains of some really smart engineers. One was a whiz at manufacturability issues, and he brought this insight to each review. What did college teach about that subject? Nothing, but Ken's probing dissections of my designs was a crash course in the art of creating products that were buildable and maintainable. Another engineer always eerily spotted those ghostly race conditions that are so tough to find. The old saw that many brains are better than one was proven many times in these reviews, and the same idea applies to inspecting code.

Plenty of evidence suggest well-run inspections are some 20 times cheaper than debugging. That's a staggering number. Since the average project burns about half the schedule in the debugging phase, any technique with a 20X improvement nearly halves the schedule. I'm generally skeptical of any dramatic claim, from the Ginzu knives to politicians' promises to clean up Washington, but even if you cut that number by an order of magnitude, inspections still shave a third from the schedule.

Higher quality code, for less money. What a deal!

Inspections are a quality gate, a filter for bugs. They must take place after one gets a clean compile (in other words, let the tools find the easiest bugs) and before any testing starts. Yes, most developers prefer to inspect code they've beaten into submission with the debugger for a while, but given the 20X efficiency factor, doing any testing before inspecting is like burning stacks of $100 bills. It's simply bad business. Even worse, early testing leads to the demise of the inspections. Few of us are strong enough to resist the siren call of "But it works!" We're all busy, claims of "works," which may mean little depending on the efficacy of the tests, generally result in the inspection meeting being dismissed.

One complaint about inspections is that they may find few bugs. That, of course, is exactly our goal. If I write software no one will see, and then write the same function given the knowledge that my peers will be digging deep into its bowels, be sure the code will be very different. No one wants his errors to be pointed out. The inspections are therefore sort of an audit that immediately raises the program's quality.

My goal in this month's allotted 1,800 words is to paint a picture about how business learned to design quality into a product rather than shoehorn it in late in the process, and then to draw an analogy to software development. Next month I'll dive into how one goes about doing an inspection.

But remember this: after examining 4,000 software projects, Capers Jones found the number one cause of schedule problems was quality.⁵ Bugs. Inspections don't slow the project. Bugs do.

Jack Ganssle (jack@ganssle.com) is a lecturer and consultant specializing in embedded systems' development issues. For more information about Jack click here .

Endnotes

1. Womack, James and Daniel Jones. Lean Thinking.New York: N.Y.:Simon & Schuster, 1996.

2. Ganssle, Jack. "Faster, Better Code," Breakpoints, Embedded Systems Programming, (http://embedded.com/98/9808br.htm).

3. Rifkin, Stan. "The Business Case for Software Process Improvement," Fifth SEPG National Meeting, 26–29 April 1993.

4. Humphrey, Watts. "The Software Quality Challenge," Crosstalk, June 2008, available at: www.stsc.hill.af.mil/crosstalk/2008/06/0806Humphrey.html

5. Jones, Capers. Assessment and Control of Software Risks. Englewood Cliffs, N.J.: Yourdon Press, 1994.