Black-Box Testing
Black-box tests are generated without knowledge of the code being
tested. When used alone, black-box tests have a low probability of
finding all the bugs in a program. But when used in conjunction with
clear-box tests they help provide a well-rounded test set, since
black-box tests are likely to uncover errors that are unlikely to be
found by tests extracted from the code structure.
Blackbox tests can really work. For instance, when asked to test an
instrument whose front panel was run by a microcontroller, one
acquaintance of the author used his hand to depress all the buttons
simultaneously. The front panel immediately locked up. This situation
could occur in practice if the instrument were placed face-down on a
table, but discovery of this bug would be very unlikely via clear-box
tests.
One important technique is to take tests directly from the
specification for the code under design. The specification should state
which outputs are expected for certain inputs. Tests should be created
that provide specified outputs and evaluate whether the results also
satisfy the inputs.
We can't test every possible input combination, but some rules of
thumb help us select reasonable sets of inputs. When an input can range
across a set of values, it is a very good idea to test at the ends of
the range.
For example, if an input must be between 1 and 10, 0, 1, 10, and 11
are all important values to test. We should be sure to consider tests
both within and outside the range, such as, testing values within the
range and outside the range. We may want to consider tests well outside
the valid range as well as boundary-condition tests.
Random tests form one category of black-box test.
Random values are generated with a given distribution. The expected
values are computed independently of the system, and then the test
inputs are applied. A large number of tests must be applied for the
results to be statistically significant, but the tests are easy to
generate.
Another scenario is to test certain types of data values. For
example, integer-valued inputs can be generated at interesting values
such as 0, 1, and values near the maximum end of the data range.
Illegal values can be tested as well.
Regression tests form an extremely important
category of tests. When tests are created during earlier stages in the
system design or for previous versions of the system, those tests
should be saved to apply to the later versions of the system. Clearly,
unless the system specification changed, the new system should be able
to pass old tests.
In some cases old bugs can creep back into systems, such as when an
old version of a software module is inadvertently installed. In other
cases regression tests simply exercise the code in different ways than
would be done for the current version of the code and therefore
possibly exercise different bugs.
Some embedded systems, particularly digital signal processing
systems, lend themselves to numerical analysis. Signal processing
algorithms are frequently implemented with limited-range arithmetic to
save hardware costs. Aggressive data sets can be generated to stress
the numerical accuracy of the system. These tests can often be
generated from the original formulas without reference to the source