Twists on Testing -

Twists on Testing

In the 1950s W. Edwards Deming proposed his now famous Deming Cycle, a model that underlies most of the new software methodologies. Plan, Do, Check, Act. Repeat till done. Especially for big, complex systems and activities, this cycle insures that the deliverable works and meets customer expectations.

Check is as important as Plan, Do, or Act. Check means run your system against a comprehensive and representative series of tests to make sure the thing does what it's supposed to do.

Kent Beck, the originator of eXtreme Programming, pushes a test-first approach. Write the test code before you build the system. Without tests how can you judge your success? How do you really know you're building something that works? Some gurus feel that this is too stringent, and in a sense harks back to Big Up-Front Design, since a test suite implies a complete specification, something the agile folks feel isn't often provided.

No one argues against testing as a critical part of building any system, especially one that uses lots of software.

Except Donald Rumsfeld. The LA Times and NPR are reporting that the administration wishes to exempt the Missile Defense System (MDS) from a law that mandates extensive tests prior to deployment.

The MDS is viewed by some as an essential shield, by others as an impossible dream, and not a few as a giant handout to the defense industry. Others claim that the current testing regimen is terribly flawed. This isn't the place to debate the merits of such a system. Let's assume it's both necessary and possible.

“Necessary,” though, suggests the MDS simply must work reliably. The only tool we engineers have to prove any system functions properly is to put it through its paces. Run comprehensive and realistic experiments designed to uncover design flaws, not tests meant to convince the customer it works.

The administration views testing as an impediment to their aggressive schedule. It probably is. We know, though, that schedules built on false hopes have alienated generations of developers and led to mountains of abandoned systems. A schedule that ignores development realities is always foolish.

I've not seen an estimate of the system's software size, but experts pegged its previous incarnation during the Reagan years at 100 million lines of code. There's a lot of room for error in a code base of that size.

Mr. Rumsfeld said he doesn't think we should wait till “every 'i' [is] dotted and every 't' crossed” before deploying the system. That's a pretty reasonable statement when building a ship or perhaps even a missile. But in software a single wrong bit can cause complete system collapse. The code will never be perfect, but it's got to be awfully darn good before it will be at all useable.

My take is that if this were a $1 billion system, well, that's chump change (in federal terms). Skipping tests might make sense since the cost of being wrong would be low. But at $70b we taxpayers should demand a system that works.

Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at . His website is .

Reader Feedback

I just read “Twists on Testing”… again, you've told it like it is.

Because I'm a natural cynic, I'm absolutely *certain* that somebody,somewhere is going to disagree with your essay and write in to tell you thanyour an unpatriotic slime-mold, blah blah blather. If any of those peoplecan come up with a justification of *why* a system as complex as MDS should*not* be tested… could you forward that on to me? (Either that, or postit on the website.) I'm curious about what train of logic leads a person tobelieve that *not* testing 100,000,000 lines of code is a prudent managementdecision!

Keep up the good work!

Doug Wolf

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.