Software Standards Compliance 101: Critical role of documentation -

Software Standards Compliance 101: Critical role of documentation


In the mid-1990s, a formal investigation was conducted into a series of fatal accidents with the Therac-25 radiotherapy machine. Led by Nancy Leveson of the University of Washington, the investigation resulted in a set of recommendations on how to create safety-critical software solutions in an objective manner. Since then, industries as disparate as aerospace, automotive and industrial control have encapsulated the practices and processes for creating safety- and/or security-critical systems in an objective manner into industry standards.

Although subtly different in wording and emphasis, the standards across industries follow a similar approach to ensuring the development of safe and/or secure systems. This common approach includes ten phases:

  1. Perform a system safety or security assessment
  2. Determine a target system failure rate
  3. Use the system target failure rate to determine the appropriate level of development rigor
  4. Use a formal requirements capture process
  5. Create software that adheres to an appropriate coding standard
  6. Trace all code back to their source requirements
  7. Develop all software and system test cases based on requirements
  8. Trace test cases to requirements
  9. Use coverage analysis to assess test completeness against both requirements and code
  10. For certification, collect and collate the process artifacts required to demonstrate that an appropriate level of rigor has been maintained.

Phase 10 is discussed in the main body of this article, the last in this standards series. 

There is a famous cartoon amongst aerospace engineers titled “Dream Airplanes” by C. W. Miller that is featured in the L. M. Nicolai book Fundamentals of Aircraft Design (METS Publishing, 1975). In this illustration, an aircraft design is presented from the perspective of the many engineering disciplines that are required to make such an aerospace project possible. From the perspective of the Wing group, the aircraft design is dominated by oversized wings, while for the Power plant group everything is built around an unfeasibly large engine. The Aerodynamics group’s perspective is a sleek, drag-free design that would be impossible to build, while the Stress group would rather the design consist of a series of steel girders.

The point of this diagram is to enforce the fact that it takes multiple disciplines to make an aerospace project successful, and no one discipline can dominate; there has to be a balance. Like an aircraft project, safety-critical embedded systems built to comply with industry standards are really an amalgamation of many different disciplines, from system safety concepts to process and software engineering, with a particular emphasis on building quality into the final work product. Nowhere is this amalgam more evident than in the project lifecycle documentation that encapsulates the work products that are produced at each phase in the development lifecycle. In addition to the final software product, the certification authorities depend on project documentation to ensure that the software was produced in an acceptable manner to meet or exceed the minimum safety criteria objectives for the project.

This article looks at some examples of the documentation produced during the development lifecycle for safety-critical software, and how the documentation work products from each phase affect and are affected by the work products from other phases. We’ll also look at the role that documentation plays in the certification process for different standards. In particular, we’ll focus on how the use of software tools throughout the development process can streamline not only the document generation process, but also the validation and verification processes that occur as the project evolves.

Documentation starts long before development
Contemporary civil engineering is responsible for an ever-increasing number of engineering marvels, often in places where it was not thought possible. Good examples include the highest bridge in the world, the Millau Viaduct in Southern France, or Burj Khalifa—the tallest building in the world that was erected in Dubai in the United Arab Emirates. Long before construction started on these projects, a significant amount of documentation was generated, and continues to be archived and maintained even after construction is complete. Focusing specifically on the technical side, examples of the documentation required for this type of construction project include:

  • Geological surveys
  • Geological mitigation designs
  • Architectural design
  • Structural design
  • Mechanical design (utilities)
  • HVAC design
  • Fire-detection and -suppression system design

It would be unimaginable that even simple construction projects would get off the ground without much of this documentation already being in place. Without them it would be impossible to define what to build, how much it would cost, and how long it would take. The same goes for software.

Next Page >>

Whether a new safety-critical embedded system project is an implementation of a completely new concept, or the evolution of an existing system type, documentation starts at the concept phase. This is where the system application, functionality, target user base, and dependence on other systems starts to get fleshed out. At this stage, it is possible to start identifying all of the standards that will apply to the final system. With the steady evolution of existing standards and the introduction of new standards into established industry sectors, this last point is not as obvious as it may seem.

Consider a new medical device software project. On the surface, it must comply with the IEC 62304—the international software lifecycle processes standard for medical device software—but are there any other standards to which the system must comply? For example, will the system have any knowledge of and/or access to patient records? If so, then the system will probably be subject to the requirements of HIPAA. Is the system intended to be connected to a wireless network? If so, then it will need to meet the accreditations of the Wi-Fi Alliance. In meeting both of these requirements, will there be any impact to the overall system safety assessment and/or design?

From this simple example it is clear that a thorough and accurate assessment of all of the standards to which the system must comply will be necessary before the project can be accurately scoped for both effort and cost. This is also the time that the first communications with the pertinent industry compliance regulators occurs in the form of a Concept of Operations document.

V lifecycle model illustrates documentation dependencies
Throughout the product development lifecycle, the required documentation at each phase and for every pertinent standard follows an evolutionary path. Each artifact is affected by those that came before it and impacts those that follow. No matter what lifecycle model is being used, a good reference for understanding the evolution and dependencies of the documentation in a project is the classic V Lifecycle Model, as described in Figure 1. (The ten phases of developing standards-compliant, safe and/or secure embedded systems discussed in this series of articles can easily be mapped to the V lifecycle model.)

The model traces the evolution of work products throughout the project lifecycle, in addition to identifying the inter-dependencies of the artifacts produced at each phase. This clearly illustrates how the Concept of Operations becomes a source document for the system Requirements and Architecture, while at the same time serving as a reference for the Validation and Verification of the final system. These dependencies identify the traceability paths that certification authorities will use to map the operation concepts through the development process, to the implementing of code, and on to the unit- and system-level tests that are used to validate and verify that the system has been implemented correctly. As a result, the V lifecycle model provides a good framework for identifying the essential artifacts required to document the work on the system under development.

click for larger image

Figure 1: The Classic V Lifecycle Model encapsulates the evolution of documentation over time, including the dependency that each piece of documentation has on those that precede it and the influence that they have on those that follow. Image extracted from Clarus Concept of Operations. Publication No. FHWA-JPO-05-072, Federal Highway Administration (FHWA), 2005

Mapping the ten software development phases for safe and/or secure embedded systems starts in the top left-hand corner of Figure 1. This is where the initial system concepts and design are documented. All subsequent documents and project work products will trace back to these documents. More detail is added as the project goes through the Requirements Capture, Architectural Design, and Detailed Design processes. The work product for each of these steps is documentation that details how the product is to be built. Each document traces back to the documents that came before, and influences those that follow. In this way, the Concepts of Operation document will inform the Requirements Capture and Architectural design documents, and will also serve as a reference for ensuring that they have covered every element of the Concepts of Operation.

This forward influence and backwards traceability continues throughout the development lifecycle until the final system software is produced and the final system acceptance testing is conducted to ensure that the implemented system matches the original Concepts of Operation.

When it comes to the certification process, it is not possible for authorities to examine every piece of project documentation. Most will be reviewed at a cursory level during the certification process, with deep-dives taken into the content based on the curiosity and experience of the reviewing team. A traceability matrix that maps the evolution of each of the Concepts of Operation through the development process to the final code and the requisite unit, module, and system tests greatly simplifies this process. This is why traceability documentation is such a strong theme throughout the standards processes.

Iterative development processes are supported by software tools
Although presented as a linear process from left to right, most (if not all) projects actually follow an iterative development process in which traceability ensures that the evolution of the project work product is on track. A complementary feedback mechanism lies with the coverage analyses that ensure that final source code is tested to an appropriate level of completion. This ensures that as much of the code as possible has been executed by tests that map back to the original system definition.

Through this iterative development and documentation process, the development team and certification authorities can answer questions such as whether all of the Concepts of Operation have been implemented, or whether every requirement has been tested. Ultimately, the objective is to ensure not only that the implementation matches the Concept of Operation but that the minimum safety and/or security requirements have been met in order to achieve standards compliance.

Managing the number, nature, and dependencies of project documentation is an area where software tools shine. At the beginning of the process, lifecycle management tools such as the example in Figure 2 can be used to establish a baseline of the documents that are required to meet standards compliance. Requirements management and traceability tools can then be used to capture the original system requirements and to document and trace the evolution of the requirements through the product development process and down to source code implementation. The same tools can also be used to tie the software test cases back to the original requirements and to capture the coverage analysis results that the test cases produce.

click for larger image

Figure 2: A screenshot of the LDRA Compliance Management System (LCMS) that provides companies with the proper infrastructure for enabling standards compliance (Source: LDRA)

During product development, the productivity gains that these software tools yield are significant, but the fact that they help to manage the documentation process as well makes them invaluable.

This article brings this series on developing standards-compliant safety and/or security critical embedded systems to a close, but with this close comes new beginnings. Certainly, documenting the work products that are generated throughout the development process is critical for ensuring that the system under development meets the minimum standards for achieving certification. But this is also true for the process of continued improvement in a manner advocated by the Capability Maturity Model Integration (CMMI). Documentation provides the reference material for the post-release analyses that are essential to identifying areas for improvement, whether they be feature enhancements, future products, and/or process improvements. The end result is a baseline for the next generation of the safety- and/or security-critical system.

1 thought on “Software Standards Compliance 101: Critical role of documentation

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.