The basics of embedded software testing: Part 1
Test is the last step in traditional software development. We gather requirements, do high level design, detailed design, create code, do some unit testing, then integrate and start—finally— final test.
Since most projects run late, what do you think gets cut? Test, of course. The implication is that we deliver bug-ridden products that infuriate our customers and drive them to competitive products.
Best practice development includes code inspections. Yet inspections typically find only 70% of a system’s bugs, so a fabulous test regime is absolutely essential. Test is like a double-entry bookkeeping system that insures mistakes don’t leak into the deployed product.
In every other kind of engineering testing is considered fundamental. In the USA, every Federally funded bridge must undergo extensive wind tunnel tests, for instance.
Mechanical engineers subject spacecraft to an almost bizarre series of evaluations. It’s quite a sight to see a 15-foothigh prototype being nearly torn to pieces on a shaker, which vibrates at a rate that puts a thousand-Hertz tone into the air. The bridge prototype, as well as that of the shaken spacecraft, are discarded at great expense, but in both cases that cost is recognized as a key ingredient of proper engineering practices.
Yet in the software world test is the ugly stepchild. No one likes to do it. Time spent writing tests feels wasted, despite the fact that test is a critical part of all engineering disciplines. Many segments of the embedded systems design community have thankfully embraced test as a core part of their processes, and advocate creating tests synchronously with writing the code, realizing that leaving such a critical step till the end of the project is folly.
Application versus embedded testing. Embedded systems software testing shares much in common with application software testing. Thus, much of this two part article is a summary of basic testing concepts and terminology. However, some important differences exist between application testing and embedded systems testing. Embedded developers often have access to hardware-based test tools that are generally not used in application development.
Also, embedded systems often have unique characteristics that should be reflected in the test plan. These differences tend to give embedded systems testing its own distinctive flavor. This article covers the basics of testing and test case development and points out details unique to embedded systems work along the way.
Before you begin designing tests, it’s important to have a clear understanding of why you are testing. This understanding influences which tests you stress and (more importantly) how early you begin testing. In general, you test for four reasons:
• To find bugs in software (testing is the only way to do this)
• To reduce risk to both users and the company
• To reduce development and maintenance costs
• To improve performance
To Find the Bugs. One of the earliest important results from theoretical computer science is a proof (known as the Halting Theorem) that it’s impossible to prove that an arbitrary program is correct.
To Reduce Costs.The classic argument for testing comes from Quality Wars by Jeremy Main. In 1990, HP sampled the cost of errors in software development during the year. The answer, $400 million, shocked HP into a completely new effort to eliminate mistakes in writing software.
The $400M waste, half of it spent in the labs on rework and half in the field to fix the mistakes that escaped from the labs, amounted to one-third of the company’s total R&D budget and could have increased earnings by almost 67%.
The earlier a bug is found, the less expensive it is to fix. The cost of finding errors and bugs in a released product is significantly higher than during unit testing, for example (Figure 2-1 below).
Figure 2-1: The Cost to Fix a Problem. Simplified graph showing the cost to fix a problem as a function of the time in the product life cycle when the defect is found. The costs associated with finding and fixing the Y2K problem in embedded systems is a close approximation to an infinite cost model.
To Improve Performance. Testing maximizes the performance of the system. Finding and eliminating dead code and inefficient code can help ensure that the software uses the full potential of the hardware and thus avoids the dreaded “hardware re-spin.”