Why every embedded software developer should care about the Toyota verdict
Toyota’s Misra C Violations
The second expert, who examined Toyota’s code, told the jury that he found more than 80,000 MISRA C violations in Toyota’s one million lines of code. Both experts told the jury that this was an indication that Toyota’s software was of poor quality.
On the second expert’s website, I found two bodies of C code that the expert has written and made publicly available for use by the embedded systems community, which would include developers of safety-critical embedded systems (the focus of MISRA C). One body of code computes CRCs, and the other performs memory testing. I downloaded both bodies of code and checked them for MISRA C violations.
These two bodies of code together, which comprise a total of 572 lines, exhibited 136 MISRA C violations. At a violation rate of 136 violations per 572 lines of code, these two bodies of code together would exhibit over 237,000 violations if they were scaled up to one million lines (compared to roughly 80,000 in Toyota’s one million lines of code).
Thus, the second expert’s own C code, which he advertises on the web for use by the embedded systems community, has a violation rate of 2.9 times the violation rate he claims to have found in Toyota’s code. Therefore, if the number of MISRA C violations in Toyota’s code (80,000 out of one million lines) is an indication of quality problems, as conveyed to the jury by both experts, then the number of MISRA C violations in the second expert’s own code (237,000 out of one million lines) is an indication of much more severe quality problems. The second expert apparently has two sets of standards when it comes to his use of MISRA C as an indicator of software quality, one that he applies to other people’s code, including Toyota’s, and another that he applies to his own code.
Both experts also testified that the number of MISRA C violations in a body of code is a predictor of the number of bugs in the code. They told the jury that for every 30 MISRA C violations, one can expect an average of three minor bugs and one major bug.
According to this metric, the second expert’s CRC and memory test code together should contain roughly 14 minor bugs and 5 major bugs. One would reasonably expect that the expert would not have made his code available to the public if he knew or suspected that it contained roughly 19 bugs. But he did make it available to the public even though it exhibits MISRA C violations that, according to his testimony to the jury, predict that it contains these 19 bugs. Thus, he apparently has two sets of standards when it comes to the use of MISRA C as a predictor of bugs, one that he applies to other people’s code, including Toyota’s, and another that he applies to his own code.
The first expert told the jury that based on the number of MISRA C violations found in Toyota’s code, that code would be expected to have roughly 2,717 major bugs (81,514/30). He also told the jury that, based on this metric (but without actually seeing Toyota’s source code), Toyota “has far, far too many bugs.”
Because the predictive relationship between MISRA C violations and number of bugs that the two experts presented to the jury is not limited to safety-critical code, I ran a MISRA C check on one of the first expert’s Ballista source files that was written in C, to see how many bugs it contains according to what he told the jury. I chose the largest of his C source files, which, together with 2 Ballista .h files that it includes, comprises 591 lines. This file exhibited 121 MISRA C violations in 591 lines of source code. According to the metric conveyed to the jury by both experts, this would mean that this C file has roughly 4 major bugs and 12 minor bugs in just 591 lines of C code. Scaled up to one million lines, it would have roughly 6,820 major bugs, which is roughly 2.5 times as many bugs as he predicted for Toyota’s one million lines of code (2,717 major bugs). One wonders how he would characterize his own 6,820 major bugs, given that he characterized Toyota’s supposed 2,717 major bugs as “far, far too many.”
The first expert’s Ballista code is academic code, created by computer science researchers who advise others on how to build high quality software. If, as the expert told the jury, MISRA C violations are a predictor of bugs, one would reasonably expect that he would ensure that his own academic code had zero MISRA C violations, to minimize the risk of it containing bugs. But he did not do that, resulting in this one file alone containing roughly 4 major bugs and 12 minor bugs, according to what he told the jury. Thus, he apparently has two sets of standards when it comes to the use of MISRA C as a predictor of bugs, one that he applies to other people’s code, including Toyota’s, and another that he applies to his own code.
For additional details, and to verify all of my numbers for yourself, please go here.