Which achieves better security: Ad hoc testing or certification? - Embedded.com

Which achieves better security: Ad hoc testing or certification?

When it comes to security within embedded systems there are several approaches one can take. Which one is right for your project is a matter of what you’d like to achieve with it.

There are various approaches to determining the security of IT systems with the aim of increasing them through suitable measures. So-called ad-hoc security testing is widely used for assessment. This means one studies the source code of a program and its documentation in order to then search specifically for weak points on the basis of this knowledge. Hackathons are a curious peculiarity. These are competitions in which an attempt is made to find weak points in a device or software. A prominent example was the hackathon, which was dedicated to the handheld Palm V from HP.

Ad hoc security testing delivers quick results and usually finds weaknesses in software. However, the more carefully the software has been developed in terms of security, the more difficult it becomes for the testers to find vulnerabilities. Ad-hoc security testing is a suitable method for quickly finding vulnerabilities that attackers can easily identify and then remedying them.

The procedure is comparatively cheap, but from the IT security point of view it has its own weak points, because the lack of a systematic security review approach as with Common Criteria (CC) can overlook some possible attacks. Vectors that can be overlooked include microarchitectural side-channel attacks such as Spectre and Meltdown.

The quality of ad-hoc security testing depends to a large extent on the skills of the test laboratory’s testers. The better they are trained and the more knowledgeable and experienced they are, the more likely they will recognize weaknesses in software and work out successful attack vectors.

Security certification (e.g., according to Common Criteria) on the other hand, is a procedure that includes not only ad-hoc security testing, but also pursues a systematic security review approach. Independent experts are involved in the certification process, which controls the results and accepts or rejects them using well-documented methods. If the results are insufficient, improvements can be made and, in the event of escalation, certification can also be refused.

At the beginning of the certification process, the client, the test laboratory that supports the client with its certification efforts and the independent certification authority agree on what to certify, what to achieve and how it is to be implemented. A special document is created for this, the so-called security target (ST). The ST is the basis for this procedure and ensures that the evaluated product and associated security problem definition is clearly defined, and the derived functional and assurance security requirements are expressed in the Common Criteria language that all parties from sponsor and developer to evaluator and certification authority unambiguously understand.

This procedure is far more complex and expensive, but also offers more actual protection. This way, security experts pursue the goal of finding every weak point in software. This kind of vulnerability search has a strong advantage: Problematic structures within the source code can be found early in the product life cycle and corrected permanently.

So, while ad-hoc testing is good for getting a quick assessment of the obvious potential for attack, certification pursues a long-term, holistic commitment to increasing the architectural security of software. Weak points that are found during ad-hoc testing are also found during security certification, but above all those that can be overlooked during ad-hoc testing. This methodical approach to making software secure dramatically reduces the chance of an irreparable software architecture. 

In the last phases of the CC certification process, a thorough vulnerability analysis through penetration testing is conducted by the evaluators, very much like in an ad-hoc security approach, but based on all the knowledge about the product gathered within the earlier phases of the evaluation (production specifications and documentation review with traceability up-to the code, functional testing of security requirements, etc.).


Simon Jakob is a Specialized Computer Scientist (IHK) and Technical Journalist (BSc UAS Bonn-Rhein-Sieg) and works in the marketing department for the leading European embedded software solutions manufacturer SYSGO GmbH, subsidiary of the Thales Group.

Related Contents:

For more Embedded, subscribe to Embedded’s weekly email newsletter.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.