Doing C code unit testing on a shoestring: Part 3 – Building a unit test framework
A reasonable framework for effective unit testing can be based on thenotion of a test set – a collection of tests covering one unit. A testin the set consists of:
* Description
* Acceptance criteria
* Test setup code (optional )
* A number of test cases
* Test cleanup code (optional )
A test case consists of:
* Description (optional )
* Parameters (optional )
* The number of repetitions
* Test case execution code (whichactually exercises a function you'retesting )
I shall not, of course, insult your intelligence by elaborating onhow to model this framework in C and how to write the generic codeexecuting a test set. A few pointers are due here though.
Instrumenting the unit undertest
We have put together a magic whereby compiling the UUT with thedefinition
INSTRUM_HEADER=”instrum_common.h”
on the command automatically instruments the UUT. There can belegitimate cases, however, where the common instrumentation is not whatyou want; e.g., as we discussed in the beginning,
The solution is to invent your own instrumentation header
#define INSTRUM_STATICextern
Producing the test set output
The purpose of the execution of a test set is to generate an outputfile. All output item should indicate whether it is produced byharness, instrumentation or a stub, for easier comprehension. Dependingon the setup, the output should always produce either HTML output orplain-text output.
The (nicely formatted) HTMLoutput can then be used for manualinspection of the execution results and for deciding whether the testset passed or failed, which is necessary if some acceptance criteriaare manual. The HTML output can be easily equipped with additionalinformation (date/time, user, unit under test, version etc.) and bepresented to the auditor as part of test documentation.
The plain-text output can be used for regression testing (
Acceptance criteria
Acceptance criteria should be stated for a test in advance; printingthem (see next section) serves as documentation. They state when youconsider a test case passed or failed, and they can be manual orautomatic.
A manual criterion simply describes what is expected to come out ofthe test case; all such criteria are considered passed if you acceptthe test set output file as a reference. An example of a manualcriterion is a notification that a certain function was called, or thelack of such notification.
An automatic criterion produces the expected output independently(like by a different algorithm ofcomputations or as pre-tabulatedvalues ) and programmatically compares the result of the testcaseexecution with the expected result. The pass/fail info should beprinted with the test case output and propagate up to test and test setsummary result.
Analyzing the output
Plain-text output is of particular interest for code coverage analysis.As discussed earlier, if your code consists only of
Similarly, if your code doesn't use the switch statement, 100% codecoverage is achieved if controlling expressions in all
(Note however that if only 99.9%of controlling expressions havebeen both true and false, we cannot conclude that the code coverage is99.9%: it can be less because of a variety of nested execution pathsnot covered at all. )
So long as each controlling statement is instrumented and isuniquely identified in the output, it is a matter of simplepost-processing of the output file to prove (or disprove) that it wastrue and false at least once during test set execution.
Now let's add the
In the output file, instrumented
The instrumentation techniques for code coverage analysis are notbullet-proof: they require that an announcement of a statement uniquelyidentifies it. A simple example of where it is not the case is aconstruct like
where it is not easy to come up with instrumentation of
Normally, these cases can be addressed by the coding policy. (
There are rare cases though where nested blocks with
Secondly, there is no transparent way to instrument the ternaryoperator, which we conveniently ignored previously. For instance,
has the same meaning and result as
y=(x)?u:v;
The former case can be instrumented and analyzed whereas the lattercannot. A workaround lies in the coding policy: One can require to usethe ternary operator in all non-constant expressions with the ISTRUEmacro (discussed with the for instrumentation), like so:
y=ISTRUE(x)?u:v;
Remarks on C++ and abnormalexecution paths
So far, we've been discussing the normal control flow. The C languageallows only one exceptional control flow mechanism, namely,setjmp/longjmp. There is nothing special to be done to account for itsince it appears as normal control flow based on the return value ofsetjmp.
The
The implementation of instrum_catch()
In the introduction to this series (
The reason is that we don't necessarily know what functions havebeen executed. However, in the object oriented (OO) paradigm, that'snot a problem at all: it's precisely the purpose of the design that theunit doesn't know what it is calling as long as the interface isprovided.
Thus, a test of the unit should only establish that any suitableimplementation has been called. In this view, a stub (
Evaluating the commercial testautomation tools
Now that you have an idea of what test automation you can get for freestraight out of your compiler, the first question is, how much morefunctionality you get from the tool XYZ and whether it's worth themoney ” and the learning.
The next question is about usability of the tool XYZ, of courseprovided that it supports your compiler and your CPU. For instance,does it work smoothly with your version control system? (
Since the test execution usually takes some time, dependenciesmanagement in the tool are important. Does it know to rebuild and rerunthe test if a source file of the unit changed? a header file on whichthe source depends? a stub?
If you are satisfied with the answers your prospective vendor has tooffer, go for it. Otherwise, you may find the techniques outlined inthis paper useful. To get started with the do-it-yourself approach, youcan download Maestra ” a free reference implementation – in adownloadable ZIP file fromMacroexpressions.
A note on free unit testingframeworks
There are at least two applicable testing frameworks that areopen-source and free of charge (
* Dependence on dynamic memory management
* Need in target platform and compiler adaptation
* Tight integration of runtime test management and test result output
* Most importantly, lack of means of code coverage analysis
The first three attributes may pose problems in resource-constrainedembedded environments. The last one is a problem in safety-relatedproduct development. A conceptually simpler and more powerful way oforganizing unit testing is to:
* Configure test sets statically (atbuild time ) to scale to theavailable computing resources
* Run a unit test is to produce an output file (
Such decoupling is what (some of )those expensive tools aim to do.It may be important if, for example, test output is sent via serial interfaceand captured on a host machine. For instance, the Maestra referenceimplementation uses
Ark Khasin, PhD, is with
To read Part 1 go to “
To read Part 2, go to “