In the mid-1990s, a formal investigation was conducted into a series of fatal accidents with the Therac-25 radiotherapy machine. Led by Nancy Leveson of the University of Washington, the investigation resulted in a set of recommendations on how to create safety-critical software solutions in an objective manner. Since then, industries as disparate as aerospace, automotive and industrial control have encapsulated the practices and processes for creating safety- and/or security-critical systems in an objective manner into industry standards.
Although subtly different in wording and emphasis, the standards across industries follow a similar approach to ensuring the development of safe and/or secure systems. This common approach includes ten phases:
- Perform a system safety or security assessment
- Determine a target system failure rate
- Use the system target failure rate to determine the appropriate level of development rigor
- Use a formal requirements capture process
- Create software that adheres to an appropriate coding standard
- Trace all code back to their source requirements
- Develop all software and system test cases based on requirements
- Trace test cases to requirements
- Use coverage analysis to assess test completeness against both requirements and code
- For certification, collect and collate the process artifacts required to demonstrate that an appropriate level of rigor has been maintained.
Phases 6 through 8 are discussed in the main body of this article. System safety and security objectives can only be guaranteed if the original requirements are adhered to, and used as the basis for the software and system level testing. Although presented sequentially in the table above, the phases described here are best started at the time that the system requirements are agreed upon. The principles described here adhere to the mantra “test early, test often”, helping to identify defects and issues as early in the development process as possible, resulting in higher system quality and the lowest possible costs for defect correction.
Standards compliance is not just about checking the code for programming errors. Standards compliance also has a foundation in a solid development process. This article looks at three specific development phases addressed by safety- and mission-critical software development standards:
- Mapping the code under development to requirements
- Generating test cases from requirements
- Mapping test cases back to requirements
From this list, it appears that the code development phases should start only when the code creation process actually starts. But, nothing could be further from the truth. These phases are all about establishing a verification framework that focusses on identifying defects early in the development process in order to reduce the cost of finding and fixing them. These phases create a feedback loop between the requirements, the software being produced, and the test cases used to verify them. The resulting process helps identify defects, ambiguities, and incompleteness early in the product development lifecycle so that they can be addressed in a manner that is both cost-effective and conducive to building quality into safety- and mission-critical systems.
Tracing Code to Requirements
Requirements provide an initial definition of the system under development, providing the reference for how the system is supposed to behave. Once this definition of the system has been translated into code, it is necessary to ensure that the “right system” is built by mapping the code under development back to the original requirements. Not only does this traceability enable project managers to be confident that the features outlined in the requirements are being implemented, but it is especially important for safety- and mission-critical systems as it ensures that the functionality required to meet the system target failure rates has been implemented.
In practice, the traceability process is normally started by mapping requirements to code, but it is the start of the feedback process between the code under development and the requirements. It is, of course, critical that all of the code under development be traceable to the original requirements to ensure that the system be feature complete, but this traceability also helps to ensure that no components are added to the software that are not part of the original system definition.
Extraneous components identified through this traceability process must be evaluated against the original requirements to determine whether they are superfluous to the system design. If they are, then they should be removed or disabled. However, extraneous components identified through this process are not always superfluous. Often they are critical for system functionality, and they therefore become what are referred to as “derived requirements”. As critical components, these components must also be tested to completion, so the requirements documentation must be updated to reflect these derived requirements as they are identified so that test cases can be created for them.
For project managers, this process is also valuable for monitoring development progress. Being able to trace code to requirements helps answer the question, “How much of the system has been completed?” For systems developed under contract, this traceability becomes a critical way to ensure that contractual schedule obligations are met.
An increasing number of software tools work with requirements capture tools to make this process even easier. Figure 1 shows a tool that integrates with requirements capture tools and creates a mechanism not only for mapping requirements to the implementing code but also for assessing the impact of a change in either requirements or code.
click for larger image
Figure 1: A screenshot of TBmanager showing the mapping between requirements and code. (Source: LDRA Technology)
Developing Software and System Test Cases Based on Requirements
When it comes to developing software and system test cases, there are two primary objectives for using requirements as the basis for creating them:
- to validate the requirements themselves, and
- to provide a basis for creating a necessary and sufficient set of test cases for testing the system under development.
By being a source of what the “right” system is, requirements provide the basis for creating the black box test cases required by safety- and mission-critical development standards, such as DO-178C. As a result, requirements must be stated in testable terms, and in turn, the final software product must be proven to satisfy the stated requirements. This means that all of the system test cases stem from one or more of the original requirements.
Good, testable requirements provide the best reference for creating black box test cases. They provide key criteria such as the starting state of the system, including the status of any data structures and/or databases, the required test case inputs, and the expected final state of the system. In practice, however, requirements can seldom be taken as read. The process of creating test cases from requirements helps to identify incomplete and ambiguous requirements and helps to raise the additional questions necessary for clarifying their meaning. Being aware of this allows good testers to stay alert to unintentional gaps in the original requirements and to work to resolve them.
Since the publication in 1981 of Barry Boehm’s groundbreaking book Software Engineering Economics , it has been recognized that the earlier in the development process that a defect is found, the cheaper it is to correct. Black box test case development can start as soon as the requirements are received and be used in parallel with software development as a means of validating the requirements themselves. By adopting this “test early, test often” approach, the incompletions, ambiguities, and open questions identified in the requirements can be highlighted early in the development process, saving significant development cost.
To meet the objectives for testing safety- and mission-critical systems as defined by the industry standards, it is necessary to prove that the complete system was built by verifying that all of the system requirements had been implemented. This is one of the primary purposes of mapping code to requirements. And, subsequently, when it comes to testing, code coverage analysis must be used to prove that all of the software in the system has been exercised to the level required for the system safety level.
Both of these phases are brought together by generating test cases from requirements, as the formal testing required for measuring the system code coverage level (referred to by practitioners as “testing for credit”) can only be achieved by the use of the black box test cases created from the original requirements. Creating test cases from requirements, starting early in the process, therefore provides an extremely powerful defect detection loop helping to verify not only the software under creation, but the actual requirements.
Once again, this is where software tools become invaluable as they help tie requirements to their associated test cases. The test framework shows the percentage of requirements addressed by test cases while also offering insight into the impact of adding or changing the requirements or their test cases. Figure 3 shows a screenshot of a tool that helps to create and maintain the traceability between requirements and the underlying test cases.
click for larger image
Figure 2: A screenshot of TBmanager showing the mapping between requirements and test cases. (Source: LDRA Technology)
Trace test cases/results to requirements
Closing the loopon test cases and requirements is what tracing test cases — and theirresults — back to the original requirements is all about. From astandards perspective, the DO-178C standard requires bidirectionaltraceability — from requirements to code, and back again. Tracing testcases and their results back to requirements is a significant steptoward achieving this objective.
A key part of the overall process is gap analysis — finding holes inthe required traceability — where additional test cases or requirementsare required. This can occur for many reasons, including implicit lowlevel requirements that come from the domain expertise of theimplementers or derived requirements identified during development.Whatever the reason, each test case must be reconciled against theoriginal requirements to ensure that the code under test is actuallyneeded to meet the system objectives. This step also helps identify codesegments that are surplus to requirements, as may occur whenincorporating third-party software components.
To satisfy the development standards for safety- and mission-criticalsystems, developers must account for all code in the system, includingunreachable and “dead” code. Many third-party components include codenot germane to the system and that will not be used. However, due to theway that these components are provided and/or licensed, it is notalways possible to strip out those portions that are not required, sothey must be proven to be unreachable or disabled (or “dead”) code.Mapping the test cases for third-party components to requirements helpsto identify components that are used, and those that are not and toidentify which components must be proven to be unreachable or disabled.
Maintaining they type of bidirectional traceability required byDO-178C in a practical way requires the use of software developmenttools. As requirements, code, test cases, and test results evolvethrough the development lifecycle, automation is the only way of keepingtrack of all of the changes and/or assessing the impact of them. Figure3 shows a screenshot of a traceability matrix that ties all of theseartifacts together into an easy to access presentation.
click for larger image
Figure3: TBmanager showing a traceability matrix that ties requirements toboth code and the test cases used to verify them. (Source: LDRATechnology)