Doing C code unit testing on a shoestring: Part 1- The basics and the tools

Safety standards such as IEC 61508 make entry thresholdtosafety-related software (firmware) products rather high, including, notin the least, the effort required for unit testing.Automation tools doexist (e.g. LDRA Testbed or IPL Cantata++) but they cost armand legand have problems of their own, such as compiler adaptation and steeplearning curve.

Much of the requirements can be met by abusing the C preprocessor ofyour very own standard-compliant C compiler. The technique grew out ofthe one I employed for my customer where it was sufficient to satisfythe requirements of SIL level 2.

This paper outlines some of the possible techniques in the hope thatyou may find them useful in evaluating your approach to unit testingand considering whether to go with a commercial test automationtool orto do it yourself.

The techniques focus on C code butare equally applicable to C++,except that proof of code coverage may be problematic in the presenceof function overloading or overriding.

Unit testing requirements
There is more to unit testing of safety-related code than testing ofinput-state-output relationships: the test is supposed to look underthe hood and demonstrate that the code execution path is as expected(and, presumably, as designed), that the results of important interimcomputations are correct and that all execution paths have beenexercised.

The three horses that pull the cart of unit testing are:

* Test harness ” acontrived code that executes the test cases whichare invented to test the unit under test

* Test stubs ” optionalfunctions (or macros) created to replacefunctions (or macros) called from the unit under test (UUT) to abstractfrom the actualbehaviors of the real functions.

* Test instrumentation “code plugged into the UUT to expose itsbehaviors normally not visible from the outside and to output”documentable” traces of execution.

Test Harness
A decent test harness might consist of a common execution framework anda series of test cases pluggable into this framework but otherwisespecific to the particulars of the UUT.

A test automation tool creates an execution framework for you. Thisis of course useful but the value of this service is not terribly high:You, all by yourself, can design and implement the framework once andbe done with it.

Test cases are to be devised and coded according to the nature ofthe unit under test. This obvious truism allows, however, to put theclaims of test automation tools in perspective.

When a vendor says their tool will generate test cases for you, thismay be so, but the cases generated in many (if not most) cases are notwhat you want. The reason is simple: the automation derives the testcases from analysis of your code and has no knowledge of the semanticsimplemented in it.

Take a simple example: you need to measure, say, hmmm gullibilityand raise an alarm if it exceeds a user-configurable threshold, enteredin the units of either gullibs or credules.

Considering that the gullibility sensor and the A/D circuit havenoise, you may decide, in the design phase, that converting raw A/Dread to gullibs or credules may easily tolerate a fixed-pointcomputation with an error of, say, five counts, as long as thecomputation itself is very fast and/or simple.

On the other hand, you probably want the round-trip conversion ofthe threshold from gullibs to credules and back to gullibs to beerror-free. (Otherwise, the user willunwittingly change the thresholdby simply changing the units back and forth .)

The tests to cover this design are:

* Measurementconversion test. Verify that for all raw A/D valuesand other inputs (such as sensor calibrations) the result differs froma naïve double-precision calculation by at most five counts.

* Unitsconversion test. Verify that for all covered levels ofgullibility the round-trip conversion of the units yields the originalvalue.

Chances are, your production code will have no traces of the designrequirements (other than in comments,if you are particular enough ). Soit is unreasonable to expect test cases generated automatically for thetests identified above: you have to code these tests yourself.

Note that it varies among the test automation tools how easy it isto integrate your own tests with your own acceptance criteria into thevendor's framework.

This is not of course to say that the no useful test cases can begenerated automatically. For instance, generating tests for a statemachine is quite possible – simply because all there is to a statemachine ends up being in the code and can be analyzed.

Test Stubs
If a function you are testing calls a function in a different unit, youneed to make a decision on the testing approach: You can create a stubto replace the called function with your own, or you can use the realfunction.

The decision depends on whether the UUT execution depends on whatthe called function does (think of,e.g., strcpy )or whether you merelydelegate creating some side effects to that function. In the formercase you obviously need a real (or an equivalent) implementation. Inthe latter case, a stub will do, but it must announce that it has beencalled.Test Instrumentation
Creating instrumented code out of your production code comprises anintimidating amount of menial labor. It is here where the testautomation tools ought to shine. They do, provided they parse your codeand (important!) its dependencies correctly.

However, a lot of automation can be accomplished by using clever Cmacros. To get a taste of it, consider a definition in file scope(i.e., outside of any block)

staticshort foo;

The problem is to inspect its value, say, before and after executinga test case, without modifying your source file. While this seemsimpossible, we can do this if we have a macro like this:

#definestatic /*nothing*/

Note that it is a valid C macro which makes foo a variable withexternal linkage. Your harness code can now say:

externshort foo;

This trick won't work though for static objects in a block scope:the macro will make foo an automatic. This can be repaired by avariation on the theme:

#definestatic extern

The new macro turns a definition into a declaration, and it turnsout that it is legal in C to put declarations any place you could put adefinition. The definition itself,

short foo;

will then go to your test harness file.

Note: This technique won't work as expected if there are several fooin several blocks, whether nested or not, as in

staticshort foo;
if (x)
{
       static short foo;        ……………
       if (y)        {
static short foo;
       ……………
    } }
else{
static short foo;
       ……………
}

Few corporate coding standards, especially in safety-relatedenvironment, would tolerate this style better fit for the Obfuscated CContest. So we can safely disregard this drawback.

Other instrumentation techniques will be demonstrated in the nextsection. Our concern now will be, where to put the macro responsiblefor the trick

#definestatic extern

We do not want this definition visible anywhere except in the unitunder test. To achieve this goal, let's do the following.

Assume that there is a header file in your project that is includedin every source file. It is more than likely that you have one already;it might hold global project configuration parameters and/or includecommon goodies like stddef.h and limits.h. Let's say it is calledproject.h, so all sources have a statement

#include “project.h”

Let's now modify this ubiquitous project.h by adding the following:

#include “instrum.h”

This can be treated by project developers as a magic incantation;the header “instrum.h” shall have no effect on the normal buildprocess. However, it is responsible for creating instrumentation when asource becomes the unit under test.

To achieve this variable behavior, we construct “instrum.h” tobeginwith the following passage:

#ifdef INSTRUM_HEADER
#    include INSTRUM_HEADER
#endif /*INSTRUM_HEADER*/

The idea is that in normal compilation of a source file,INSTRUM_HEADER is not defined and the source compiles as it always did.

When, however, a source file, say foo.c ,is theUUT, we create aninstrumentation header file for it, say, instrum_foo.hand pass thedefinition

INSTRUM_HEADER=”instrum_foo.h”

on the command line of the compiler. (Usually, it's a “D compilerswitch, or an equivalent configuration in the integrated developmentenvironment. Note that on Windows platforms passing a quoted definitionin the command line is a tricky dealing with CMD.EXE; try

INSTRUM_HEADER=””instrum_foo.h”

As we will see later, there is a good chance to use the sameinstrumentation header for all sources to be unit-tested, say,instrum_common.h. This approach will be our target, and the definitionto pass to the compiler is

INSTRUM_HEADER=”instrum_common.h”

The instrumentation header, instrum_common.h ,will have

#defineINSTRUM_STATICextern

(Or it may have #define INSTRUM_STATICor no definition for INSTRUM_STATIC at all, depending on the needs.)

The “magic” header, instrum.h, willhave the following:

#ifdef INSTRUM_STATIC
#    define static INSTRUM_STATIC
#endif /*INSTRUM_STATIC*/

That is, if instrumentation for static is not defined, the keywordwill keep its normal meaning. Otherwise, its meaning becomes whateverthe instrumentation header assigns to it; this remains completelytransparent to the magic header.

C code instrumentation
Now we are in a position to devise instrumentation of the code byabusing other keywords.

Instrumenting the if statements. Following the pattern outlined inthe previous section, let's add the following to instrum.h:

#ifdef INSTRUM_IF
#    define INSTRUM_IF(condition)
#endif

The instrumentation headerinstrum_common.h would have somethinglike

#defineINSTRUM_IF(condition)
       if (instrum_if(#condition,(condition)!=0,
        __FILE__, __LINE__,__FUNCTION__))

externintinstrum_if(constchar *condition_name,
                                 intcondition,
                                 constchar *filename,
                                int line,
                                 const char *function_name);

Implementation of instrum_if can be anything you want it to be,except that in order not to alter the behavior of you code, it mustreturn its second argument (condition). For instance, the followingimplementation just prints what condition where and if it is true orfalse:

int instrum_if(constchar *condition_name,
                      int condition,
                      const char *filename,
                    int line,                       const char *function_name)
{
    printf(“Condition%s in funtion %s”
           ” (file %s line %d) is %sn”,            condition_name, function_name,            filename, line,(condition)?”true”:”false”);
returncondition;
}

This implementation goes to some instrumentation support sourcefile. Of course this file itself should not be instrumented.

Some (pre-C99) compilers might not define __FUNCTION__,or you mightnot care to output the file name. Your implementation might be like so:

#defineINSTRUM_IF(condition)
        if(instrum_if1(#condition,(condition)!=0,
        __LINE__))

externint instrum_if1(const char*condition_name,
                                 int condition,
                                int line);

with a corresponding implementation.

We will see later that instrumentation of other keywords willrequire a use of if .Theif willcome there instrumented but we don't want to see the effects of thatinstrumentation. So a real instrumentation function for ifmust detectwhether theifis a resultof instrumentation ofsomething else and if so suppress all instrumentation actions.

To read Part 2, go to “Towardcodecoverage analysis.”
To read Part 3, go to “Buildinga unit test framework.”

Ark Khasin, PhD, is with MacroExpressions. He can be contacted atakhasin@macroexpressions.com.