Unintended acceleration and other embedded software bugs
Event data recorders
Event data recorder (EDR) is the generic term for the automotive equivalent of an aircraft black-box flight data recorder. EDRs were first installed in cars in the early 1990s and have increased in use as well as sophistication in the years since. Generally speaking, the event data recorder is an embedded system residing within the airbag control module located in the front center of the engine compartment. The event data recorder is connected to other parts of the car's electronics via the controller-area network (CAN bus) and is always monitoring vehicle speed, the position of the brake and accelerator pedals, and other key parameters.
In the event of an impossibly-high (for the vehicle operating normally) acceleration or deceleration sensor reading, Toyota's latest event data recorders save the prior five 1Hz samples of these parameters in a nonvolatile memory area. Once saved, an event record can be read over the car's on-board diagnostics (OBD) port (or, in the event of a more severe accident, directly from the airbag control module) via a special cable and PC software. If the airbag actually deploys, the event record will be permanently locked. The last two or three (depending on version) lesser "bump" records are also stored, but may be overwritten in a FIFO (first in, first out) manner.
This investigation of Toyota's unintended acceleration marked the first time that anyone from NHTSA had ever read data from a Toyota event data recorder. (Toyota representatives apparently testified in Congress that there had previously just been one copy of the necessary PC software in the U.S.) As part of this study, NHTSA validated and used tools provided by Toyota to extract historical data from 52 vehicles involved in incidents of unintended acceleration, with acknowledged bias toward geographically reachable recent events. After reviewing driver and other witness statements and examining said black-box data, NHTSA concluded that 39 of these 52 events were explainable as "pedal misapplications." That's a very nice way of saying that whenever the driver reported "stepping on the brake," he or she had pressed the accelerator pedal by mistake. Figure 5 of a supplemental report describing these facts portrays an increasing likelihood of such incidents with driver age vs. the bell curve of Camry ownership by age.3
Click on image to enlarge.
Note that no record is apparently ever made, in the event data recorder or elsewhere, of any events or state changes within the ETCS-i firmware. So-called "Diagnostic Trouble Codes" concerning sensor and other hardware failures are recorded in nonvolatile memory and the presence of one or more such codes enables the "Check Engine" light on the dashboard. But no logging is done of significant software faults, including but not limited to watchdog-initiated resets.
Engine control module
ETCS-i is a collection of components and features that was changed in the basic engine design when Toyota switched from mechanical to electronic throttle control. (Electronic throttle control is also known as "throttle-by-wire.") Toyota has used two different types of pedal sensors in the ETCS-i system, always in a redundant fashion. The earlier design, pre-2007, using potentiometers was susceptible to current leakage via growth of tin whiskers.4 Although this type of failure was not known to cause sudden high-speed behaviors, it did seem to be associated with a higher number of warranty claims. The newer pedal sensor design uses Hall effect sensors.
Importantly, the brakes are not a part of the ETCS-i system. In the 2005 Camry, Toyota's brake pedal was mechanically controlled. (It may still be.) It appears this is one of the reasons the NASA team felt comfortable with their conclusion that driver reports of wide open throttle behavior that could not be stopped with the brakes were not caused by software failures (alone). "The NESC team did not find an electrical path from the ETCS-i that could disable braking." (NASA Report, p. 15)5 It is clear, though, that power assisted brakes lose the enabling vacuum pressure when the throttle is wide open and the driver subsequently pumps the brakes; thus any system failure that opened the throttle could indirectly make bringing the vehicle to a stop considerably harder.
The engine control module at the heart of the ETCS-i consists of a Main-CPU and a Sub-CPU located within a pair of ASICs. The Sub-CPU contains a set of A/D converters that translates raw sensor inputs, such as voltages VPA and VPA1 from the accelerator pedal, into digital position values and sends them to the Main-CPU via a serial interface. In addition, the Sub-CPU monitors the outputs of the Main-CPU and is able to reset (in the manner of a watchdog timer) the Main-CPU.
The Main-CPU is reported to be a V850E1 microcontroller, which is "a 32-bit RISC CPU core for ASIC" designed by Renesas (nee NEC).6 The V850E1 processor has a 64-MB program address space, which is part of an overall 4-GB linear address space. The Main-CPU also keeps tabs on the Sub-CPU and can reset it if anything is found wrong.
NASA reports that the embedded software in the Main-CPU is written (mostly) in ANSI C and compiled using a GreenHills C compiler (Appendix A, p. 14). Furthermore, an OSEK-compliant real-time operating system with fixed-priority preemptive scheduling is used to manage a redacted (but apparently larger than 10, based on the size of the redaction) number of real-time tasks. The actual firmware development (design, coding and unit testing) was outsourced to Denso (p. 19).7 Toyota apparently performed integration testing and ran several commercial and in-house static analysis tools, including QA-C (p. 20).8 The code was written in English, with Japanese comments and design documents, and follows a proprietary Toyota naming convention/coding standard that predates but half overlaps with the 1998 version of MISRA-C.9
Loading comments... Write a comment