Why every embedded software developer should care about the Toyota verdict
If you develop embedded software for a living, and especially if you work for a large company with deep pockets, you could wake up one day to see the quality of your software being publicly disparaged based on highly questionable accusations. This happened to the Toyota engineers, who saw a flurry of articles published about their software with titles such as “Toyota Unintended Acceleration and the Big Bowl of ‘Spaghetti’ Code” and “Toyota's killer firmware: Bad design and its consequences.” This could happen to any one of us, as I will now explain.
Several months ago I wrote an article for Embedded.com describing my technical analysis of the 2013 Oklahoma jury verdict that Toyota’s embedded software was to blame for unintended acceleration that resulted in a fatal accident. The full analysis appeared in a peer-reviewed IEEE article I had previously published, which I summarized for Embedded.com. The analysis in the IEEE article reveals that the accident theory presented to the jury by the plaintiffs’ software expert was technically flawed and thus not credible, and therefore the jury reached the wrong verdict.
I was delighted to receive a number of thoughtful comments from readers of the Embedded.com article, as well as from readers of a reposting of that article on EETimes.com. A common issue raised by a number of those readers was the low quality of Toyota’s embedded software, as alleged by the plaintiffs at the trial and discussed in a number of articles and presentations after the trial. The feeling expressed by several of these readers was that even if the plaintiffs did not adequately establish that the software caused the accident, surely the low quality of Toyota’s software can’t be ignored. Some readers expressed the opinion that, since the software was so bad, it must have caused the accident, even if the specific accident theory offered by the plaintiffs falls apart under technical scrutiny (which it does).
Due to space limitations for the IEEE article, I only had room to address the plaintiffs’ flawed causation theory. I was not able to address the plaintiffs’ criticisms of Toyota’s software quality, which are also highly questionable. I subsequently published another article about the trial in IEEE Consumer Electronics Magazine in which I address one of my many concerns about the plaintiffs’ criticisms of Toyota’s software quality, namely, their criticisms of Toyota’s use of global variables. Toward the end of that article, I provide a link to a more complete analysis in which I detail a number of additional reasons why the plaintiffs’ quality criticisms are highly subjective and thus highly questionable. I will summarize those reasons in this column, covering the following three areas:
Toyota’s use of global variables
The number of MISRA C violations found in Toyota’s code
Plaintiffs’ contention that unintended acceleration events would be expected to occur in the fleet of Toyota vehicles once every week or two, based on research by Obermaisser
In this summary I will refer to the publicly-available trial testimony of the two software experts who testified on behalf of the plaintiffs. The first expert to testify did not see any of Toyota’s source code, yet much of his testimony was directed toward criticizing the quality of that code. The second expert to testify, who did examine Toyota’s source code, criticized the quality of the code and also claimed to have found a likely cause of the accident in the code (which is the subject of my original IEEE article). As you will see, the experts’ questionable criticisms of the quality of Toyota’s code can be used to disparage anyone’s code irrespective of its quality, including the code of the experts themselves.
Toyota’s Use of Global Variables
The first expert, who did not see any of Toyota’s source code, told the jury that others who had examined the code had determined that there were approximately 10,000 global variables in Toyota’s one million lines of code. He told the jury that this was an indicator that “Toyota’s source code is of poor quality.” He further told the jury that “global variables are evil” and the “academic standard is there should be zero” global variables. He then said to the jury: “Well, I know that the right answer academically is zero. And in practice, five, ten, okay, fine. 10,000, no, we're done. It is not safe, and I don't need to see all 10,000 global variables to know that that is a problem.”
However, it turns out that this same expert’s own academic code violates the standards for the use of global variables that he presented to the jury. His academic code exhibits the same use of global variables that he told the jury was an indicator that “Toyota's source code is of poor quality.”
On his university website, the expert advertises a software project that he led called Ballista. He provides a download link for the Ballista source code. I downloaded and examined the code.
That source code includes 6 C files, 19 .h files, and 21 C++ files, among others. These files comprise a total of 8,552 lines of code. In those 8,552 lines, there are a total of at least 68 global variables. That would scale up to 7,951 global variables in one million lines of code. This is close to the number of global variables (10,000 – 11,000) allegedly found in Toyota’s one million lines of code, which was severely criticized by both experts. Thus, the first expert, who told the jury that “the right answer academically is zero” global variables, clearly did not apply that standard to his own academic code. He apparently has two sets of standards regarding global variables, one that he applies to other people’s code, including Toyota’s, and another that he applies to the code developed by his own academic research group.
In fact, there are comments in the Ballista code describing some of the reasons why the expert and his team decided to use global variables. These comments reflect just a few of the many possible considerations that could lead to a decision to make variables global. Such considerations were not acknowledged by either of the two plaintiffs’ software experts during the trial testimony. Rather, the impression conveyed to the non-technical jury was that the numbers presented by these two experts (“five, ten” global variables in one million lines of code) are hard and fast, and if those numbers are exceeded then the software is of low quality, period. This is not an accurate reflection of real-world software development, as the first expert’s own academic code shows, but nonetheless this is what the jury was told. Furthermore, the first expert also told the jury that “there is no reason” for Toyota to have this many global variables, even though he did not see any of Toyota’s code, and even though his own academic code contains explicit comments describing reasons to use global variables.
For additional details, including access to the full trial testimony and the Ballista links, and to verify my analysis for yourself, please go here.