Doing C code unit testing on a shoestring: Part 1- The basics and the tools
Safety standards such as IEC 61508 make entry threshold to safety-related software (firmware) products rather high, including, not in the least, the effort required for unit testing. Automation tools do exist (e.g. LDRA Testbed or IPL Cantata++) but they cost arm and leg and have problems of their own, such as compiler adaptation and steep learning curve.
Much of the requirements can be met by abusing the C preprocessor of your very own standard-compliant C compiler. The technique grew out of the one I employed for my customer where it was sufficient to satisfy the requirements of SIL level 2.
This paper outlines some of the possible techniques in the hope that you may find them useful in evaluating your approach to unit testing and considering whether to go with a commercial test automation tool or to do it yourself.
The techniques focus on C code but are equally applicable to C++, except that proof of code coverage may be problematic in the presence of function overloading or overriding.
Unit testing requirements
There is more to unit testing of safety-related code than testing of
input-state-output relationships: the test is supposed to look under
the hood and demonstrate that the code execution path is as expected
(and, presumably, as designed), that the results of important interim
computations are correct and that all execution paths have been
exercised.
The three horses that pull the cart of unit testing are:
* Test harness " a contrived code that executes the test cases which are invented to test the unit under test
* Test stubs " optional functions (or macros) created to replace functions (or macros) called from the unit under test (UUT) to abstract from the actual behaviors of the real functions.
* Test instrumentation " code plugged into the UUT to expose its behaviors normally not visible from the outside and to output "documentable" traces of execution.
Test Harness
A decent test harness might consist of a common execution framework and
a series of test cases pluggable into this framework but otherwise
specific to the particulars of the UUT.
A test automation tool creates an execution framework for you. This is of course useful but the value of this service is not terribly high: You, all by yourself, can design and implement the framework once and be done with it.
Test cases are to be devised and coded according to the nature of the unit under test. This obvious truism allows, however, to put the claims of test automation tools in perspective.
When a vendor says their tool will generate test cases for you, this may be so, but the cases generated in many (if not most) cases are not what you want. The reason is simple: the automation derives the test cases from analysis of your code and has no knowledge of the semantics implemented in it.
Take a simple example: you need to measure, say, hmmm gullibility and raise an alarm if it exceeds a user-configurable threshold, entered in the units of either gullibs or credules.
Considering that the gullibility sensor and the A/D circuit have noise, you may decide, in the design phase, that converting raw A/D read to gullibs or credules may easily tolerate a fixed-point computation with an error of, say, five counts, as long as the computation itself is very fast and/or simple.
On the other hand, you probably want the round-trip conversion of the threshold from gullibs to credules and back to gullibs to be error-free. (Otherwise, the user will unwittingly change the threshold by simply changing the units back and forth.)
The tests to cover this design are:
* Measurement conversion test. Verify that for all raw A/D values and other inputs (such as sensor calibrations) the result differs from a naïve double-precision calculation by at most five counts.
* Units conversion test. Verify that for all covered levels of gullibility the round-trip conversion of the units yields the original value.
Chances are, your production code will have no traces of the design requirements (other than in comments, if you are particular enough). So it is unreasonable to expect test cases generated automatically for the tests identified above: you have to code these tests yourself.
Note that it varies among the test automation tools how easy it is to integrate your own tests with your own acceptance criteria into the vendor's framework.
This is not of course to say that the no useful test cases can be generated automatically. For instance, generating tests for a state machine is quite possible - simply because all there is to a state machine ends up being in the code and can be analyzed.
Test Stubs
If a function you are testing calls a function in a different unit, you
need to make a decision on the testing approach: You can create a stub
to replace the called function with your own, or you can use the real
function.
The decision depends on whether the UUT execution depends on what the called function does (think of, e.g., strcpy) or whether you merely delegate creating some side effects to that function. In the former case you obviously need a real (or an equivalent) implementation. In the latter case, a stub will do, but it must announce that it has been called.
Loading comments... Write a comment