How to verify your compiler for use in IEC 61508 safety-critical applications -

How to verify your compiler for use in IEC 61508 safety-critical applications

IEC 61508 is a standard for electrical, electronic and programmableelectronic, safety-related systems. It is interesting because it spanshardware and software components. Therefore, with care, software can beincorporated directly into a safety function, reducing the requirementfor physical safety mechanisms. Broadly speaking, IEC61508 covers three areas:

1. Businessprocesses and structure. This should be covered by systems thatare compliant with a standard such as ISO9001:2000.

2. Hardwaredevelopment. This focuses on techniques to reduce systematic andrandom hardware failures.

3. Softwaredevelopment. This focuses on techniques used to reducesystematic errors in safety-related software – the probability of whichis not generally quantifiable.

Safety considerations extend throughout the design cycle (Figure 1 below ). This articlefocuses on one aspect of IEC61508 software development – the vexingquestion of compiler verification.

Figure1: Software safety integrity and the development lifecycle (theV-model) [Source: IEC]

IEC61508 describes two routes to compiler verification, but crediblecompliance with either can be difficult. This article sets out analternative that has recently been used successfully during theIEC61508 process certification of SAFERTOS – a mini real-timescheduler. IEC61508 has no legally-sanctioned entity authorised tocertify compliance.

In this case, I suggest that 'certification' means having complianceindependently verified by a recognized authority. Furthermore, thecertification means that the rigor with which the SAFERTOS componentwas developed makes it suitable for applications that have beenassigned a SafetyIntegrity Level (SIL) of 3. It does not mean that the useof the software in an application makes the application SIL 3compliant.

It does mean that systems to which a SIL of 3 has been assigned canuse the compliance evidence gathered during the SAFERTOS development asevidence in the 'certification' of the entire system into whichSAFERTOS is being built.

It therefore does not preclude the requirement for analysis and duediligence in the development of the whole system, but provides a veryrapid low-risk route and framework to the formulation of a complianceargument for the software components.Development tool usage
Within IEC61508, the term “highly recommended” is applied to items thatmust be complied with, or a detailed rationale for non-compliance ” anda suitable alternative ” must be accepted by the assessor. For supporttools and programming languages, IEC61508 “highly recommends”:

1) Use of a suitableprogramming language

2) Use of a strongly-typedprogramming language

3) Use of a definedlanguage subset

4) Certified tools, orproven confidence from extended tool usage

5) A certified translator,or proven confidence from extended translator usage

6) Use of a library oftrusted and verified software modules and components.

Consideration should also be given to the availability of thecompiler for the lifetime of the product. There are two “highlyrecommended” paths to justify why a compiler is suitable for the SIL ofthe application under development. The compiler can be certified to thesame or greater level, or the concept of 'proof-in-use' can be applied.Both have their problems.

Also, IEC61508 imposes some requirements on language choice.SAFERTOS had to be written in C as it was to be compiled into a Cprogram. IEC61508 attempts to impose rigorous restrictions on languageselection, but then, perhaps in recognition that this is not alwayspracticable, provides alternative clauses. For example, it is firststated that the language should:

a) Have atranslator/compiler which has either a certificate of validation to arecognized national or international standard, or should be assessed toestablish its fitness for purpose;

b) Be completely andunambiguously defined or restricted to unambiguously defined features;

c) Match thecharacteristics of the application;

d) Contain features thatfacilitate the detection of programming mistakes;

e) Support features thatmatch the design method.

The use of C as the language makes compliance with points a, b, andd difficult. In our case, where an alternative compliance route wasdesirable, IEC61508 permits deviation provided:

1) The deviation isjustified within the project documentation.

2) The fitness for purposeof the language is assessed.

3) Risks identified duringthe fitness- for-purpose assessment are mitigated through the use ofextra measures.

4) The assessor accepts thearguments for the above.

Due to our usage of the C language, compliance with these conditionswas essential to the compliance of the SafeRTOS development withIEC61508.

Figure2: Overall framework of the IEC61508 standard [Source: IEC]

Conformance through compilercertification
It is not clear from the standard what criteria need to be met for acompiler to be accepted as 'certified', so we had to formulate our ownlogical argument. I suggest that a C compiler can legitimately be botha trusted and a long term supportable component for use in asafety-related system if:

1) It is proven using avalidation process equal in rigor to that necessary for the SIL of theapplication under development.

2) The validation evidenceis either available for inspection or has passed an audit withdocumented scope by a credible and independent organization.

3) It implements a clearlyspecified set of the language features in a clearly specified manner,with any deviations clearly documented.

4) It is maintained understrict configuration control with the chosen version remaining as aused and supported product by the vendor for the foreseeable life ofthe product.

Conformance through compilerproof-in-use
Does such a C tool chain exist for the processor you are using? If not,there is the option of conformance through “proof-in-use”.

Looking at the issue logically again, I would suggest thatproof-in-use evidence would be credible if:

1) There are a large andknown number of users all of whom have been using at a known frequencyand for an extended period, a stable version of the tool that isunchanging other than through a formal and published change-controlprocess. But, how much use does a compiler get, and how can this bequantified? If a user performs only a 10 second compilation every fourhours, how much evidence does this provide?

2) The language constructsbeing used by this pool of existing users is known. Are existing users,on whom the proof-in-use evidence is being based, using the samelanguage constructs as you will use? If not, is any proof-in-useevidence being provided?

3) There is a mandated andenforceable error reporting and logging mechanism. If this is not inplace, how do you know your evidence is complete?

4) All deviations from theintended behavior are formally documented and available for inspection.This implies that the intended behavior is itself formally documented.

Again, does such a tool chain exist for the processor you are using?

It is unlikely that tools credibly meeting either certification orproof-in-use criteria will be available. It is no surprise then thatdeep in an appendix to IEC61508 there is a statement that, when appliedto compilation tools, verification usually only certifi es how the toolcomplies with the relevant language standard, and that translationtools are usually not certifi ed in any way with respect to safety.

At first this seems to be a bit of a get-out clause, but we aredeveloping a safety-related system and this statement says that wedon't know if the tools are safe! As system developers, it is ourresponsibility to ensure safety. So, rather than being a get-outclause, it is just a restatement of the problem:

1. How to gain confidencethat the compiler output is correct for its input and safe so far asthe output causes the behavior intended with respect to its input, andno other behavior.

2. How to determine the'standard' implemented by the tools. Embedded C compilers are,generally speaking, not fully compliant with any recognized C standard.

Furthermore, it has already been noted that the use of C does notfulfill the IEC61508 recommendation that a strongly typed language beused, resulting in a possible source of error that requires reasonable,documented and assessor-agreed mitigation. Given the above, one mightconclude that there is an impasse to the inclusion and reliance onsoftware in the safety path. However, this article suggests analternative.If we have a detailed specification and code as input, and wethoroughly test the output, we have provided a way of eliminating thetranslation from input to output (the compiler) from the consideration.

The SAFERTOS test harness contains tests that are designed toprovide complete requirements coverage, along with the usual boundaryand equivalence class analysis. With respect to the described toolverification impasse, the solution we proposed was to analyze theSAFERTOS test harness results to ensure Modified Decision/ConditionCoverage (MCDC) was obtained.

MCDC is a measure of structural coverage used widely in the aviationsoftware industry. It requires that test cases exercise each conditionof each decision within the program, and that when doing so eachcondition within a decision can independently affect the outcome of thedecision.

As an example, consider the following decision:

if( (a>b) && (c>d)

The two conditions are (a>b), which I will refer to as CondA, and c>d) which I will refer to as CondB. There are four possible input states as described in the truth table below:

To achieve MCDC it must be shown that while holding CondA constant,modifying CondB can cause the result of the decision to be both trueand false. Likewise, while holding CondB constant, modifying CondA canresult in both a true and false result from the decision. Therefore inthis case, MCDC can be achieved by providing test cases with true true,true false and false true inputs.

The false false case is not required, as the intention is todemonstrate that each condition can independently affect the outcome ofthe decision. Using the same logic it can be seen that, should thedecision take the form:

if((a>b) || (c

then only the false false, true false, and false true test cases arerequired. MCDC must be demonstrated for all Boolean expressions, notjust those contained directly within a decision point.Rationale
MCDC is used as a structural coverage measure for a number of reasons:

1. Enoughtesting? It is difficult to assess the adequacy of testing,especially in systems where dependability is paramount. Ideally,testing would be exhaustive, but this is rarely possible. MCDC canprovide a sense of adequacy.

2. Adequacy ofrequirements-based testing. There may be any number of ways toimplement a requirement, so test cases generated purely fromrequirements are not guaranteed to exercise all the code. Structuralcoverage analysis confi rms that the requirements-based tests actuallyexercised the code structure.

3. Identifyunintended functions. Test cases that provide MCDC assist in thedemonstration of the absence of unintended behavior.

4.Identify unreachable code

5. High objectcode coverage ” In our case this is the main point. The objectcode coverage obtained was not measured, but in the absence of compileroptimization it is assumed to be very high.

Achieving MCDC demonstrates that each component of each decision istested in isolation, and that the object code generated for both thepositive and negative outcome of each such component has been executedand the resultant behavior checked against that expected.

Using this rationale, by employing a thorough structural coveragemeasurement analysis to our complete requirements coverage testing, weare exercising a high percentage of the code output by the compiler (ifnot all reachable code) and comparing this to the intended function orthe compiler input ” the requirement.

We are therefore effectively testing the compiler's interpretationof its input, rather than trusting the compiler to behave as expected.

We have seen the restrictions on language and tool usage containedwithin the IEC 61508 standard and my argument as to why it is not easyto comply with these restrictions. Also, for reasons of due diligence,it is necessary to assess the 'safety' of the tools being used.

The civil aviation standard DO-178B is formulated on the premise that a compiler cannot be certified and therefore contains mitigations for the untrusted nature of thecompilation tools.

Borrowing one such mitigation from DO-178B level A – the requirementfor MCDC – permitted the SAFERTOS development procedures to be acceptedby the independent assessor as adequate for a SIL 3 development.

The policy is to effectively say that the compilation tools are nottrusted, with the risk mitigated through a test structure designed toensure that, for every case, the output of the compiler produces thebehavior that was intended.

Two interesting observations were made during the test process:

1. Extradevelopment cost. Good development processes will requirerigorous management, design, coding and testing procedures. When theseare already in place, the additional burden of demonstrating that theoutcome of these procedures achieves MCDC is minimal.

2. Toolindependence. The resultant test suite was compiler independentto all intents and purposes. A change in the development tools wouldrequire a change impact analysis followed by the re-execution of thetest suites in order to have confidence in the compiler output.

Acknowledgements: The author wishes to thank Mike Dewalt ofCertification Services Inc. and Ian Cleare of The 61508 Association for theirhelpful comments during the drafting of this article.

Richard Barry (r.barry@HighIntegritySystems. com), is head ofinnovation at WittensteinHigh Integrity Systems.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.