Unintended acceleration and other embedded software bugs

Michael Barr

March 30, 2011

Michael Barr

Are there bugs in Toyota's firmware?
In the NASA Report's executive summary it is made clear that "because proof that the ETCS-i caused the reported UAs was not found does not mean it could not occur." (NASA Report, p. 17) The report also states that NASA's analysis was time-limited and top-down, remarking "The Toyota Electronic Throttle Control (ETC) was far more complex than expected involving hundreds of thousands of lines of software code" and that this affected the quality of a planned peer review.¹⁰

It's stated that "Reported [Unintended Accelerations (UAs)] are rare events. Typically, the reporting of UAs is about 1/100,000 vehicles/year." But there are millions of cars on the road, and so NHTSA has collected some "831 UA reports for Camry" alone. "Over one-half of the reported events described large (greater than 25 degrees) high-throttle opening UAs of unknown cause" (NASA Report, p. 14), the causes of which are never fully explained in these reports.

The NASA team apparently identified some lesser firmware bugs themselves, saying "[our] logic model verifications identified a number of potential issues. All of these issues involved unrealistic timing delays in the multiprocessing, asynchronous software control flow." (Appendix A, p. 11) NASA also spent time simulating possible race conditions¹¹ due to worrisome "recursively nested interrupt masking" (pp, 44 to 46); note, though, that simulation success is not a sufficient proof of lack of races. As well, the NASA team seems to recommend "reducing the amount of global data" (p. 38) and eliminating "dead code" (p. 40).

Additionally, the redacted text in other parts of Appendix A seems to be obscuring that:

• "The standard gcc compiler version 4" generated a redacted number of warnings (probably larger than 100) about the code, in 11 different warning categories. (p. 25)12
• "Coverity version 4.2" generated a redacted number of warnings (probably larger than 154) about the code, in 10 different warning categories. (p. 27)¹³
• "Codesonar version 3.6p1" generated a redacted number of warnings (probably larger than 136) about the code, in 10 different warning categories.¹⁴
• "Uno version 2.12" generated a redacted number of warnings (probably larger than 72) about the code, in nine different warning categories.¹⁵
• The code contained at least 347 deviations from a subset of 14 of the MISRA-C rules.⁹
• The code contained at least 243 violations of a subset of nine of the 10 "Power of 10—Rules for Developing Safety Critical Code," published in IEEE Computer in 2006 by NASA team member Gerard Holzmann.¹⁶

It looks to me like Figure 6.2.3-1 of the NASA Report (p. 30) shows that UA complaints filed with NHTSA increased in the year of introduction of electronic throttle control for the vast majority of Toyota, Scion, and Lexus models—and that complaint counts have remained higher but generally declined over time since those transitions years. Such a complaint data pattern is perhaps consistent with firmware bugs. (Note to NHTSA: It would be helpful to see this same chart normalized by number of vehicles sold by model year and with the rows sorted by the year of ETC introduction. It would also be nice to see a chart of ETCS-i firmware versions and updates, which vehicles they apply to, and the dates on which each was put into new production vehicles or distributed through dealers.)

Figure 6.2.3-1:


Click on image to enlarge.

Final thoughts
I am not privy to all of the facts considered by the NHTSA or NASA review teams and thus cannot say if I agree or disagree with their overall conclusion that embedded software bugs are not to blame for reports of unintended acceleration in Toyota vehicles. How about you? If you've spotted something I missed in the reports from NHTSA or NASA, please send me an e-mail or leave a comment below. Let's keep the conversation going.

Michael Barr is the author of three books and over fifty articles about embedded systems design, as well as a former editor-in-chief of this magazine. Michael is also a popular speaker at the Embedded Systems Conference, a former adjunct professor at the University of Maryland, and the president of Netrino. He has assisted in the design and implementation of products ranging from safety-critical medical devices to satellite TV receivers. You can reach him via e-mail at mbarr@netrino.com or read more of what he has to say at his blog (www.embeddedgurus.net/barr-code).

Endnotes:

1. Appendix A— NASA Engineering and Safety Center (NESC). "NASA Engineering and Safety Center Technical Assessment Report National Highway Traffic Safety Administration Toyota Unintended Acceleration Investigation—Appendix A. Software," January 18, 2011. Appendix A describes NASA's review of Toyota's embedded software in detail and has the most heavy redactions of NASA's report. www.nhtsa.gov/staticfiles/nvs/pdf/NASA_FR_Appendix_A_Software.pdf.
2. Toyota Corporation. "Toyota Statement in Response to NHTSA/NASA Study." Press release on Toyota's web site (viewed on March 16, 2011) available at www.toyota.com/about/news/corporate/2011/02/09-1-Statement.html
3. National Highway Traffic Safety Administration. "NHTSA Toyota Pre-Crash EDR Field Inspections During March – August 2010," February 2011. PDF is available at www.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-Toyota_EDR_field_inspection.pdf.
4. For more info about tin whiskers see NASA—Tin Whisker (and Other Metal Whisker) Homepage at http://nepp.nasa.gov/whisker/.
5. NASAReport—"NASA Engineering and Safety Center Technical Assessment Report. National Highway Traffic Safety Administration Toyota Unintended Acceleration Investigation" Version 1. Technical Support to the National Highway Traffic Administration (NHTSA) on the Reported Toyota Corporation (TMC) Unintended Acceleration (UA) Investigation (NESC Assessment #: TI-10-00618), January 18, 2011. www.nhtsa.gov/staticfiles/nvs/pdf/NASA-UA_report.pdf
6. NEC Corporation. "User's Manual: V850E1—32-bit Microprocessor Core Architecture," document number U14559EJ2V0UM00 (2nd edition), date published March 2001, available at http://america2.renesas.com/docs/files/U14559EJ2V0UM00.pdf.
7. Denso Global (www.globaldenso.com/en/) developed the firmware (design, coding, and unit testing).
8. QA·C Source Code Analyzer, one of code analyzer tools Toyota used, available at www.programmingresearch.com/qac_main.html.
9. 1998 version of MISRA-C. See www.misra-c.com/Activities/MISRAC/tabid/160/Default.aspx.
10. NHTSA. "Peer Review of NHTSA and NASA Test Plan into Toyota Unintended Acceleration," February 2011, available at www.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-Toyota_peer_review.pdf.
11. For background information on race conditions, see Barr, Michael. "Firmware-Specific Bug #1: Race Condition," EmbeddedGurus web site, http://embeddedgurus.com/barr-code/2010/02/firmware-specific-bug-1-race-condition/.
12. For standard gcc compiler, see http://gcc.gnu.org/.
13. For more about Coverity, see www.coverity.com/.
14. Codesonar, www.grammatech.com/products/codesonar/.
15. Uno Tool Synopsis, current tool version: 2.13, October 26, 2007. Available at http://spinroot.com/uno/.
16. "The Power of Ten: 10 Rules for Writing Safety Critical Code" web site at http://spinroot.com/p10.