Building more secure embedded software with code coverage analysis

David and Mike Kleidermacher, Green Hills Software

September 18, 2013

David and Mike Kleidermacher, Green Hills SoftwareSeptember 18, 2013

Editor’s Note: Excerpted from their book Embedded Systems Security,  David and Mike Kleidermacher discuss how the use of code coverage techniques can improve the reliability and security of embedded software without necessarily increasing cost or development time.

A comprehensive test regimen, including functional, regression, performance, and coverage testing, is one of the best mechanisms to assure that software is reliable and secure. Indeed, testing is an important component of many high-assurance development standards and guidance documents, such as that promulgated by the U.S. Food and Drug Administration.

In addition, two approaches to testing are almost always required to ensure security. First, all software within security-critical components must be covered by some form of functional test: white-box, black box, fault-based, error-based and stress.. Then coverage is verified using code coverage tools. Further, all security-critical software must be traceable to the software’s component requirements. Software that fails to trace back to a test and to a requirement is more likely to introduce latent security vulnerabilities.

Modified Condition/Decision Coverage
Because code coverage analysis is so important for security, it is worth examining the various levels of coverage testing that can be applied to embedded software.To aid this discussion, we consider the code coverage requirements across the five assurance levels specified in the standard that the U.S. Federal Aviation Administration (FAA) uses to perform safety certification of commercial aircraft.

This standard, published by RTCA, is titled Software Considerations in Airborne Systems and Equipment Certification, commonly referred to as DO-178B. In fact, DO-178B is the most commonly used software safety standard in the worldwide avionics industry. The five assurance levels, in increasing level of criticality, of DO-178B are as follows:

  • Level E: Failure has no impact on flight safety.
  • Level D: Failure impact is minor, noticeable but not critical lto flight safety (e.g.,passenger
  • inconvenience).
  • Level C: Failure impact is major, safety-related but not severe (e.g., passenger discomfort
  • but not injury).
  • Level B: Failure impact is severe (e.g., passenger injury).
  • Level A: Failure impact is catastrophic (e.g., aircraft crash).

The structural code coverage requirements corresponding to each assurance level are shown in the following table:

Table 1: Assurance Level coverage table

DO-178B Level C requires statement coverage: demonstrating that every program statement has been executed at least once (covered) by the verification test regimen. Statement coverage is what most developers equate with the more general term code coverage.

Level B augments statement coverage with decision coverage, a requirement that every decision point in the program has been executed with all possible outcomes. For example, a conditional branch’s comparison both succeeds( branch taken) and fails (branch not taken) at least once each.

Finally, modified condition/decision coverage (MC/DC) augments decision coverage with a specialized form of condition coverage in which each condition within a decision must be shown to have an independent effect on the outcome of that decision. We use a few simple code examples to illustrate the increasing rigor and security-enforcing quality of each coverage approach:

  if (a || b || c) {
    <code executed on true decision>

Statement coverage requires that the if statement is executed and that the code within the if Block (executed on a true decision ) is fully executed.As there are no statements corresponding to a false decision, statement coverage would not require any test cases that force the if block not to execute.

In contrast, decision coverage would require at least one test to execute the false decision path, even though there is no explicit code associated with that path. This extra coverage is desirable from a security perspective because it indicates that the developer has considered the impact of a false decision, which may have some other side effects. Let’s consider this slightly more detailed example:

  uint32_t divisor = 0;
  if (a || b || c) {
    divisor = a | b | c;
  result /= divisor;

The final division statement will fail (divide by zero) on a false decision, but statement coverage testing may never activate this pathway. If an attacker were somehow able to control the decision (e.g., by controlling the values of a, b, and c), then the attacker could cause a denial of service (program crash). Decision coverage testing would have pointed out this problem before it could be fielded.

Condition coverage requires that each condition within the decision be tested with true and false values. The following two test cases will force each of the three conditions to take on both a true and a false value at least once: (a=1, b=1, c=1) and (a=0, b=0, c=0). While testing a decision’s constituent conditions may seem like an improvement over decision coverage, condition coverage is not a superset of decision coverage,as shown in this example below:

  if (a || !b) {
    <code executed on true decision>
  } else {
    <code executed on false decision>

The two test cases, (a=0, b=0) and (a=1, b=1), satisfy condition coverage (both conditions executed with true and false inputs) but neglect to cover the false decision path. Clearly, decision and condition coverage techniques used in concert is desirable.

Multiple condition coverage requires all combinations of conditions. In other words, every row of a decision’s truth table must have a corresponding test case. In the earlier test case with conditions a, b, and c, the truth table is as follows:

Table 2: Truth Table

Thus, multiple condition coverage requires 2n tests, where n is the number of independent conditions. This approach is viewed as impractical; exhaustive condition testing would simply take too many test cases and too long to execute. Languages with short-circuiting Boolean operators (e.g.,C,C++,Java) reduce the number of required test cases:

Table 3: Required test cases table

Nevertheless, compound Boolean expressions may yield an impractical explosion in test cases across realistic programs.

< Previous
Page 1 of 2
Next >

Loading comments...