Safety standards such as
Much of the requirements can be met by
This paper outlines some of the possible techniques in the hope thatyou may find them useful in evaluating your approach to unit testingand considering whether to go with a commercial
The techniques focus on
Unit testing requirements
There is more to unit testing of safety-related code than testing ofinput-state-output relationships: the test is supposed to look underthe hood and demonstrate that the code execution path is as expected(and, presumably, as designed), that the results of important interimcomputations are correct and that all execution paths have beenexercised.
The three horses that pull the cart of unit testing are:
* Test harness ” acontrived code that executes the test cases whichare invented to test the unit under test
* Test stubs ” optionalfunctions (or macros) created to replacefunctions (or macros) called from the unit under test (UUT) to abstractfrom the actualbehaviors of the real functions.
* Test instrumentation “code plugged into the UUT to expose itsbehaviors normally not visible from the outside and to output”documentable” traces of execution.
A decent test harness might consist of a common execution framework anda series of test cases pluggable into this framework but otherwisespecific to the particulars of the UUT.
A test automation tool creates an execution framework for you. Thisis of course useful but the value of this service is not terribly high:You, all by yourself, can design and implement the framework once andbe done with it.
Test cases are to be devised and coded according to the nature ofthe unit under test. This obvious truism allows, however, to put theclaims of test automation tools in perspective.
When a vendor says their tool will generate test cases for you, thismay be so, but the cases generated in many (if not most) cases are notwhat you want. The reason is simple: the automation derives the testcases from analysis of your code and has no knowledge of the semanticsimplemented in it.
Take a simple example: you need to measure, say, hmmm gullibilityand raise an alarm if it exceeds a user-configurable threshold, enteredin the units of either gullibs or credules.
Considering that the gullibility sensor and the A/D circuit havenoise, you may decide, in the design phase, that converting raw A/Dread to gullibs or credules may easily tolerate a fixed-pointcomputation with an error of, say, five counts, as long as thecomputation itself is very fast and/or simple.
On the other hand, you probably want the round-trip conversion ofthe threshold from gullibs to credules and back to gullibs to beerror-free. (Otherwise, the user willunwittingly change the thresholdby simply changing the units back and forth .)
The tests to cover this design are:
* Measurementconversion test. Verify that for all raw A/D valuesand other inputs (such as sensor calibrations) the result differs froma naïve double-precision calculation by at most five counts.
* Unitsconversion test. Verify that for all covered levels ofgullibility the round-trip conversion of the units yields the originalvalue.
Chances are, your production code will have no traces of the designrequirements (other than in comments,if you are particular enough ). Soit is unreasonable to expect test cases generated automatically for thetests identified above: you have to code these tests yourself.
Note that it varies among the test automation tools how easy it isto integrate your own tests with your own acceptance criteria into thevendor's framework.
This is not of course to say that the no useful test cases can begenerated automatically. For instance, generating tests for a statemachine is quite possible – simply because all there is to a statemachine ends up being in the code and can be analyzed.
If a function you are testing calls a function in a different unit, youneed to make a decision on the testing approach: You can create a stubto replace the called function with your own, or you can use the realfunction.
The decision depends on whether the UUT execution depends on whatthe called function does (think of,e.g., strcpy )or whether you merelydelegate creating some side effects to that function. In the formercase you obviously need a real (or an equivalent) implementation. Inthe latter case, a stub will do, but it must announce that it has beencalled.Test Instrumentation
Creating instrumented code out of your production code comprises anintimidating amount of menial labor. It is here where the testautomation tools ought to shine. They do, provided they parse your codeand (important!) its dependencies correctly.
However, a lot of automation can be accomplished by using clever Cmacros. To get a taste of it, consider a definition in file scope(i.e., outside of any block)
The problem is to inspect its value, say, before and after executinga test case, without modifying your source file. While this seemsimpossible, we can do this if we have a macro like this:
Note that it is a valid C macro which makes foo a variable withexternal linkage. Your harness code can now say:
This trick won't work though for static objects in a block scope:the macro will make foo an automatic. This can be repaired by avariation on the theme:
The new macro turns a definition into a declaration, and it turnsout that it is legal in C to put declarations any place you could put adefinition. The definition itself,
will then go to your test harness file.
Note: This technique won't work as expected if there are several fooin several blocks, whether nested or not, as in
static short foo;
Few corporate coding standards, especially in safety-relatedenvironment, would tolerate this style better fit for the Obfuscated CContest. So we can safely disregard this drawback.
Other instrumentation techniques will be demonstrated in the nextsection. Our concern now will be, where to put the macro responsiblefor the trick
We do not want this definition visible anywhere except in the unitunder test. To achieve this goal, let's do the following.
Assume that there is a header file in your project that is includedin every source file. It is more than likely that you have one already;it might hold global project configuration parameters and/or includecommon goodies like stddef.h and limits.h. Let's say it is calledproject.h, so all sources have a statement
Let's now modify this ubiquitous project.h by adding the following:
This can be treated by project developers as a magic incantation;the header “instrum.h” shall have no effect on the normal buildprocess. However, it is responsible for creating instrumentation when asource becomes the unit under test.
To achieve this variable behavior, we construct “
The idea is that in normal compilation of a source file,INSTRUM_HEADER is not defined and the source compiles as it always did.
When, however, a source file, say foo.c ,is theUUT, we create aninstrumentation header file for it, say,
on the command line of the compiler. (Usually, it's a “D compilerswitch, or an equivalent configuration in the integrated developmentenvironment. Note that on Windows platforms passing a quoted definitionin the command line is a tricky dealing with CMD.EXE; try
As we will see later, there is a good chance to use the sameinstrumentation header for all sources to be unit-tested, say,instrum_common.h. This approach will be our target, and the definitionto pass to the compiler is
The instrumentation header, instrum_common.h ,will have
(Or it may have
The “magic” header,
That is, if instrumentation for static is not defined, the keywordwill keep its normal meaning. Otherwise, its meaning becomes whateverthe instrumentation header assigns to it; this remains completelytransparent to the magic header.
C code instrumentation
Now we are in a position to devise instrumentation of the code by
Instrumenting the if statements. Following the pattern outlined inthe previous section, let's add the following to instrum.h:
The instrumentation headerinstrum_common.h would have somethinglike
Implementation of instrum_if can be anything you want it to be,except that in order not to alter the behavior of you code, it mustreturn its second argument (condition). For instance, the followingimplementation just prints what condition where and if it is true orfalse:
printf(“Condition%s in funtion %s”
” (file %s line %d) is %sn”,
This implementation goes to some instrumentation support sourcefile. Of course this file itself should not be instrumented.
Some (pre-C99) compilers might not define
with a corresponding implementation.
We will see later that instrumentation of other keywords willrequire a use of
To read Part 2, go to “
Ark Khasin, PhD, is with