Doing C code unit testing on a shoestring: Part 3 – Building a unit test framework

A reasonable framework for effective unit testing can be based on thenotion of a test set – a collection of tests covering one unit. A testin the set consists of:

* Description
* Acceptance criteria
* Test setup code (optional )
* A number of test cases
* Test cleanup code (optional )

A test case consists of:

* Description (optional )
* Parameters (optional )
* The number of repetitions
* Test case execution code (whichactually exercises a function you'retesting )

I shall not, of course, insult your intelligence by elaborating onhow to model this framework in C and how to write the generic codeexecuting a test set. A few pointers are due here though.

Instrumenting the unit undertest
We have put together a magic whereby compiling the UUT with thedefinition

INSTRUM_HEADER=”instrum_common.h”

on the command automatically instruments the UUT. There can belegitimate cases, however, where the common instrumentation is not whatyou want; e.g., as we discussed in the beginning, instrum_common.hhas

#define INSTRUM_STATIC /*nothing* /and you want
#define INSTRUM_STATIC extern

The solution is to invent your own instrumentation headerinstrum_myown.h and passit as INSTRUM_HEADER. A preferred way is notto re-do all the work but to include instrum_common.h ininstrum_myown.h ,undefine the inadequate implementation, and define itin an appropriate fashion, e.g.

#include “instrum_common.h”

#undefINSTRUM_STATIC
#define INSTRUM_STATICextern

Producing the test set output
The purpose of the execution of a test set is to generate an outputfile. All output item should indicate whether it is produced byharness, instrumentation or a stub, for easier comprehension. Dependingon the setup, the output should always produce either HTML output orplain-text output.

The (nicely formatted) HTMLoutput can then be used for manualinspection of the execution results and for deciding whether the testset passed or failed, which is necessary if some acceptance criteriaare manual. The HTML output can be easily equipped with additionalinformation (date/time, user, unit under test, version etc.) and bepresented to the auditor as part of test documentation.

The plain-text output can be used for regression testing (Ioptimized the code; does it still work as before?). It can alsobe usedfor post-processing of your choosing so that additional information canbe extracted.

Acceptance criteria
Acceptance criteria should be stated for a test in advance; printingthem (see next section) serves as documentation. They state when youconsider a test case passed or failed, and they can be manual orautomatic.

A manual criterion simply describes what is expected to come out ofthe test case; all such criteria are considered passed if you acceptthe test set output file as a reference. An example of a manualcriterion is a notification that a certain function was called, or thelack of such notification.

An automatic criterion produces the expected output independently(like by a different algorithm ofcomputations or as pre-tabulatedvalues ) and programmatically compares the result of the testcaseexecution with the expected result. The pass/fail info should beprinted with the test case output and propagate up to test and test setsummary result.

Analyzing the output
Plain-text output is of particular interest for code coverage analysis.As discussed earlier, if your code consists only of if/elsestatements,100% code coverage is achieved if controlling expressions in allifstatements have been both true and false.

Similarly, if your code doesn't use the switch statement, 100% codecoverage is achieved if controlling expressions in allif, whileandforstatements have been both true and false, provided that there is nounreachable (dead) code. If there is, the compiler (or at least Lint)should inform you about that.

(Note however that if only 99.9%of controlling expressions havebeen both true and false, we cannot conclude that the code coverage is99.9%: it can be less because of a variety of nested execution pathsnot covered at all. )

So long as each controlling statement is instrumented and isuniquely identified in the output, it is a matter of simplepost-processing of the output file to prove (or disprove) that it wastrue and false at least once during test set execution.

Now let's add the switchstatements to the mix. Assuming that allcontrolling expressions in all if,while and forstatements had beentrue and false at least once, you achieved 100% code coverage if andonly if each of the caseand defaultstatements had been hit at leastonce. (If you have switch statementswithout a default ,it isconsidered not a good practice yet it can be dealt with. Still, thiscase is more complicated and is omitted here.)

In the output file, instrumented caseand defaultstatements thatwere executed would announce themselves. To verify that all of themwere executed, you can scan the source of the UUT to extract the caseand default statements and match them against their announcements inthe test output; if each of them was announced, you've got 100% codecoverage, otherwise, you haven't. This can be done with anot-so-sophisticated script whose complexity may depend on whether ornot you want to account for nested switch statements.Limitations of the approach
The instrumentation techniques for code coverage analysis are notbullet-proof: they require that an announcement of a statement uniquelyidentifies it. A simple example of where it is not the case is aconstruct like

if (++x)a; else if (++x)…

where it is not easy to come up with instrumentation ofif whichwould ensure a unique identification of each ifstatement.

Normally, these cases can be addressed by the coding policy. (E.g.,MISRA wants a block to follow an if and a coding style usually wants anewline to precede or to follow an opening curly brace.)

There are rare cases though where nested blocks with ifsand loopand switchconstructs comprise a macro definition, and occasionallysuch a macro has merits. When such a macro is expanded, allinstrumentation functions will get the same __FILE__, __LINE__,and __FUNCTION__ values, so unique identification may be tricky.

Secondly, there is no transparent way to instrument the ternaryoperator, which we conveniently ignored previously. For instance,

if (x){y=u;} else {y=v;}

has the same meaning and result as

y=(x)?u:v;

The former case can be instrumented and analyzed whereas the lattercannot. A workaround lies in the coding policy: One can require to usethe ternary operator in all non-constant expressions with the ISTRUEmacro (discussed with the for instrumentation), like so:

y=ISTRUE(x)?u:v;

Remarks on C++ and abnormalexecution paths
So far, we've been discussing the normal control flow. The C languageallows only one exceptional control flow mechanism, namely,setjmp/longjmp. There is nothing special to be done to account for itsince it appears as normal control flow based on the return value ofsetjmp.

The try/throw/catchexception mechanism of C++ is a little harder todeal with. We need to analyze whether each catchstatement had been hitduring the test set execution. This can be done with the same trick weused (effectively) for thefor statement instrumentation:

#definecatch(a) catch(a) if(instrum_catch()){} else

The implementation of instrum_catch() isexpected to always return false and to never throw.

In the introduction to this series ( Part 1), I conceded upfront thatproof of coverage in the presence of overloaded or overridden functionscan be problematic.

The reason is that we don't necessarily know what functions havebeen executed. However, in the object oriented (OO) paradigm, that'snot a problem at all: it's precisely the purpose of the design that theunit doesn't know what it is calling as long as the interface isprovided.

Thus, a test of the unit should only establish that any suitableimplementation has been called. In this view, a stub (e.g., from aderived class) that announces its own execution is sufficient (and theconcession in the abstract withdrawn).

Evaluating the commercial testautomation tools
Now that you have an idea of what test automation you can get for freestraight out of your compiler, the first question is, how much morefunctionality you get from the tool XYZ and whether it's worth themoney ” and the learning.

The next question is about usability of the tool XYZ, of courseprovided that it supports your compiler and your CPU. For instance,does it work smoothly with your version control system? (I know of at least one tool that doesn'tlike read-only files.) Is test report independent of the machineon which the test was executed? (Sometools annoyingly insist on absolute paths.) How easy is it tobring inyour own test case in the framework generated by the tool? (Recall thegullibility example.)

Since the test execution usually takes some time, dependenciesmanagement in the tool are important. Does it know to rebuild and rerunthe test if a source file of the unit changed? a header file on whichthe source depends? a stub?

If you are satisfied with the answers your prospective vendor has tooffer, go for it. Otherwise, you may find the techniques outlined inthis paper useful. To get started with the do-it-yourself approach, youcan download Maestra ” a free reference implementation – in adownloadable ZIP file fromMacroexpressions.

A note on free unit testingframeworks
There are at least two applicable testing frameworks that areopen-source and free of charge (CUnitand CppUnit; the latter can be adapted to testing C code ).Theydeservedly gained a fair amount of acceptance; however they share somelimitations, including:

* Dependence on dynamic memory management
* Need in target platform and compiler adaptation
* Tight integration of runtime test management and test result output
* Most importantly, lack of means of code coverage analysis

The first three attributes may pose problems in resource-constrainedembedded environments. The last one is a problem in safety-relatedproduct development. A conceptually simpler and more powerful way oforganizing unit testing is to:

* Configure test sets statically (atbuild time ) to scale to theavailable computing resources
* Run a unit test is to produce an output file (essentially, anexecution log) that lends itself to post-processing on a hostplatform.

Such decoupling is what (some of )those expensive tools aim to do.It may be important if, for example, test output is sent via serial interfaceand captured on a host machine. For instance, the Maestra referenceimplementation usesprintf to output readable text, but it doesn't haveto be so: you can output tokens of any sort (even binary) to reduce theraw output size, and then post-process it into a readable form.

Ark Khasin, PhD, is with MacroExpressions. He can be contacted atakhasin@macroexpressions.com .

To read Part 1 go to “Unit testingrequirements and what you will need.”
To read Part 2, go to “Toward codecoverage analysis.”