The “Big Thaw” - An Agile Process for Software Certification - Embedded.com

The “Big Thaw” – An Agile Process for Software Certification

To achieve certification, safety-critical systems must demonstrate compliance with domain-specific standards such as DO-178 for commercial avionics. Developing a certified system consists of various interrelated activities that produce outputs (collections of artifacts) as evidence of successful completion. For example, one of the DO-178 verification activities is a traceability analysis; its output is a report showing that each software requirement is implemented in the source code. Conducting the certification-required activities and producing the artifacts demand a major effort, much more than for conventional Quality Assurance on non safety-critical systems.

Real-world systems rarely stay unchanged. New requirements arise, bugs may be detected, better hardware may become available. Software providers have needed to deal with these issues since the earliest days, but the challenge is especially acute for safety-certified systems. The basic issue is how to assess the impact of a change. Which activities need to be carried out, which artifacts are affected? In short, what is needed to re-achieve system certification, and what will it cost?

Traditionally these questions have been difficult to answer, and as a result the general practice for certified systems has been to make few if any changes until a major upgrade is required. This situation is sometimes referred to as the “Big Freeze”. All components and tools are baselined: the application software, the platform (hardware, operating system, peripheral devices), the tools (compilers, linkers, static analysis tools, etc.). Baselining can raise other issues, of course, since support and maintenance costs for third-party provided elements will typically be much higher than for the current releases.

A new solution to the “Big Freeze” problem is proposed here, drawing on principles from Agile Development. This approach, which we call the “Big Thaw”, involves continuous (or, more precisely, frequently iterated) performance of the activities that are affected by a change, to keep all artifacts up to date and to preserve the complete system’s certifiability. The approach is based on an analysis of the artifacts’ structure and interdependencies, since the artifacts are the concrete realization of the effect of the activities.

Process- versus product-based certification
Software certification standards fall into two main categories. A process-based standard relies on evidence that the various activities associated with development have been performed successfully. DO-178B [1] is an example of such a standard. Figure 1 shows the three general process categories (Planning, Development, and Liaison) and some of the associated outputs. The standard defines a set of ten specific processes in these three categories. Each process is captured by a table that lists its associated objectives; in total there are 66 objectives. For each objective there are specific actions to be performed, and a corresponding output consisting of one or more artifacts. A system safety assessment establishes the level for each software component, ranging from E (no effect on safety) to A (anomalous behavior can cause a catastrophic failure and prevent continued safe flight and landing). The level in turn determines which objectives need to be met.

The underlying logic is that a safe system can only be achieved through the performance of sound software engineering processes, which can in turn be assessed through an evaluation of the various artifacts that they produce. Although this means that the inference of system safety is indirect, DO-178B has proved to be successful in practice. In the nearly twenty years since this standard’s inception, no commercial aircraft fatality has been attributed to DO-178B-certified software.

Standards evolve based on application experience and new technologies, and DO-178B is no exception. A new version, DO-178C [2], is expected to be finalized in the near future. It will address several modern software methodologies – in particular Object-Oriented Technology, Model-Based Design, and Formal Methods – but the overall approach remains strongly process-based.

In contrast, a product-based (or goal-based) certification standard provides a more direct assessment of software safety. Examples are the UK Ministry of Defence Standard OO-56 [3] and the FDA’s Draft Guidance on infusion pump premarket notification submissions [4], which call for an assurance case approach [5]. The developer provides assurance cases consisting of claims concerning the relevant system attributes, arguments justifying those claims, and evidence backing up the arguments. For safety-critical systems the assurance cases are known as safety cases, and the relevant attributes are properties related to the system’s safety. Safety cases are typically hierarchical, with higher-level attributes broken down into lower layers. The safety cases are the certification artifacts; the activities are implicit.

Both process- and product-based approaches result in artifacts that are affected as a system evolves. The Big Thaw approach applies to each.
Dependencies
Artifacts may be related in various ways, and animportant relationship in DO-178B is traceability links. For example,the developer must show that each low-level requirement is traceable tosource code that implements that requirement and to one or more testcases (and resulting test executions) that show that the requirement hasbeen implemented correctly. The source code and the test cases dependon the requirement; changing the requirement may invalidate itsdependents (i.e., require their adaptation if the system is to remaincertifiable).

Other kinds of dependences are manifest in thespecific DO-178B objectives. For example, the source code depends on thecoding standard. Adding a restriction to the coding standard mayrequire modifying one or more source code modules.

Regardless ofthe nature of the dependence relation, if artifact B depends onartifact A then there is some compliance condition that B must meet withrespect to A. The compliance condition defines the nature of thedependence.

Figure 2 shows a dependence graph for asimplified portion of a DO-178B-based workflow. The nodes are theartifacts, and the directed arcs are the activities that derive the“target” artifact from the “source”. The dependence relation defines aDirected Acyclic Graph.

Whethera change in an artifact invalidates a dependent artifact with respectto system certifiability depends upon the nature of thechange. Correcting a typographical error, for example, does not affectthe dependents and thus does not require repeating the activities thatcreate the dependent artifacts. Likewise, removing a rule (restriction)from the coding standard is a purely localized effect (although therationale for the removal would need to be supplied). If the source codecomplied with the previous version of the coding standard, then it alsocomplies with the new version.

Granularity
Minimizingthe recertification effort requires a fine-grained view of theartifacts. For example, an individual requirement is an artifact; asource code compilation unit is an artifact. The outputs of the variousactivities aggregate their associated artifacts. Thus the designdescription produced by the software development processes contains,among other things, all of the low-level requirements. The source codeproduced by the software development processes is not simply a singlemonolithic artifact but rather a collection of artifacts that in turnmay have semantic interdependences; for the source code, the granularityof the artifacts is at the level of individual compilation units.

Certification Readiness
Inthe context of the overall system an artifact is certification ready ifits traceability relationships are satisfied, each artifact it dependson is certification ready, and it is up-to-date with (i.e., meets thecompliance condition for) each artifact it depends on.

Thisgeneral definition covers artifacts such as test executions, which havean associated “result” (pass or fail). In this case the test executiondepends on the executable, and the compliance condition is that the testis passed.

We are using the term “certification ready” ratherthan “certifiable” to emphasize that individual artifacts are nevercertified; only the complete system is certified.

Activity- versus artifact-centric certification
Traditionalapproaches for managing certification or recertification have beenbased on activity-centric models, focusing on the temporal and causaldependences among the various activities. However, this requires aseparate formalism for capturing the nature of the artifacts and theirtraceability and other dependence relations. The approach proposed heredirectly models the certification process’s artifacts and theirinterrelationships. It can be represented in a directed acyclic graphwhere the nodes are artifact types, and the edges are dependencies,traceability or otherwise. A traceability arc is annotated with the nameof the activity that produces the dependent artifact. This approacheases the task of checking for artifact certification readiness in thepresence of changes, and can help automate some of the activities thatare needed.

The focus on artifact status does not imply thatsomehow the activities have reduced importance in the certificationprocess. On the contrary, the activities are essential, but whether andwhen they need to be performed can be deduced from the state of theartifacts.

Delta and continuous certification
Theartifact-centric model can be used to implement a two-part process forkeeping a system certification ready in the presence of changes. Firstis the “delta” calculation: the determination of the effect of a changein terms of artifacts that are affected and activities that need to beperformed. The implementation can deduce the minimal set of artifactsthat need to be brought to a certification-ready state, and thecorresponding activities that need to be performed. Carrying out theseactivities, either in an automated fashion or manually, will bring thesystem into a certification ready state. At this point the process knownas “continuous certification” can be used: the automatable activitiesrequired by the artifact model are performed as soon as possible,perhaps during the next build/test cycle. This is similar to the conceptof continuous integration espoused by the Agile Development community[6].

Conclusion
From one point of view the Agilecertification process may be regarded as a smart “make”facility. Although this may be true from an implementation perspective,the artifact-centric approach also marks a major change in developmentphilosophy for most organizations that need to demonstrate compliancewith certification standards. The focus on artifacts and theirinterrelationships makes it easier to assess which activities need to beperformed as the system evolves, and how much effort will berequired. The Ice Age for safety certification is coming to an end; theBig Freeze is about to give way to the Big Thaw.

References

  1. RTCA SC-167 / EUROCAE WG-12. DO-178B – Software Considerations in Airborne Systems and Equipment Certification; December 1992.
  2. RTCA SC-205. DO-178C – Software Considerations in Airborne Systems and Equipment Certification. Draft IP 50, Rev X; 2011.
  3. UK Ministry of Defence. Defence Standard 00-56, Issue 4, Safety Management Requirements for Defence Systems; 1 June 2007.
  4. US FDA. Total Product Life Cycle: Infusion Pump – Premarket Notification [510(k)] Submissions, Draft Guidance; April 23, 2010.
  5. Graydon, P., Knight, J., and Strunk, E., “Assurance Based Development of CriticalSystems,” Proc. of 37th Annual International Conference on Dependable Systems and Networks, Edinburgh, U.K., 2007.
  6. Beck, K. et. al. Manifesto for Agile Software Development. agilemanifesto.org/

Matteo Bordin isa software engineer at AdaCore's Paris office, where his work focuseson the certification and qualification process in a DO-178 context. Hisarea of expertise is model-based development – in particular multi-levelmetamodeling, formal model-based analysis and automated code generation- and he is a member of the subgroup that is producing the Model-BasedDesign and Verification supplement to DO-178C. Mr. Bordin received hisMasters degree in Computer Science from the University of Padova(Italy).

Jerome Lambourg is the manager of theQualifying Machine project at AdaCore, an effort aimed at applying Agilemethodologies to safety certification. Before joining AdaCore's Parisoffice in 2005, he worked at Canal+ on interactive televisionback-office servers and also served as a systems architect for atechnology consulting company and as a software engineer at Thales NavalFrance. Mr. Lambourg holds a degree from Ecole Nationale Superieure desTelecommunications (Paris, France).

Dr. Benjamin Brosgol is a senior member of the technical staff of AdaCore in the US. He hasbeen involved with programming language design and implementation formore than 30 years, concentrating on languages and technologies forhigh-integrity systems. Dr. Brosgol was a member of the design team forAda 95, and he has also served in the Expert Groups for several JavaSpecification Requests. He has presented papers and tutorials on safetyand security certification on numerous occasions. Dr. Brosgol holds a BAin Mathematics from Amherst College, and MS and PhD degrees in AppliedMathematics from Harvard University.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.