Toyota’s Expensive Software

Toyota has agreed to a $1.2 billion fine to settle a U.S. government criminal case over unexpected acceleration in Toyota and Lexus vehicles that resulted in injuries and deaths. A jury in Oklahoma found that, in one case at least, the culprit was the firmware. (The plaintiff’s lead expert, Mike Barr, is giving a talk about the case at EELive!).

This payout is on top of another $1B settlement for the same problem . Bottom line: poor firmware has cost the company staggering amounts of cash.

Testimony shows that the firmware wasn’t just suffering from a bug; instead it was reeking with problems . One has to wonder what the engineers were thinking.

Everyone in this industry faces twin pressures: fast and cheap. Deliver now. Cut engineering costs. I have no insight into how many person-hours went into the Toyota code, nor do I know their delivery schedule. But let’s look at that most recent $1.2B payout. How does that compare to the engineering effort?

The NASA report talks about a code base of “more than 280,000 lines” of code. Mike Barr tells me there were “over a million lines of C source code”. For argument’s sake, let’s figure on a million.
The most expensive code ever written is that of the Space Shuttle, which ran about $1000/LOC (201 Principles of Software Development , Alan M. Davis, 1995). With just the most recent settlement, Toyota’s code cost them over $1200 per line – without accounting for any engineering effort. The difference is that the Shuttle’s code is the best ever written, averaging about one bug per 400KLOC, and Toyota’s has been intensely litigated.

I am not suggesting that Shuttle development practices should be anyone’s goal. Perhaps a better benchmark is avionics. It’s largely believed that no one has been killed by defective firmware in commercial aircraft, yet that code controls pretty much everything. Sure, the pilots can take over, but modern planes are fly-by-wire. The pilot flies a computer. What does it cost to develop the fabulous software that mediates billions of passenger-miles per day in the air?

Commercial avionics is done to a standard called DO-178B (supplanted recently by DO-178C). Level E applies to software that won’t impact operations in any significant way. Level A is for code that can lead to the loss of the aircraft. How much does it cost to write code to level A?

Who knows? Data is sparse and proprietary. However, most pundits figure it’s about twice the cost of typical commercial firmware. Others (“DO-178B Costs Versus Benefits” by Vance Hilderman), in this case based on data from some 150 avionics programs, claim code written to level A is 65% more expensive than that to level E. That figure includes both the engineering effort and the certification process

Note that level C is roughly equivalent to Capability Maturity Model level 3, which is generally considered the entry point to disciplined software engineering. Level A (in the B version of the standard) has 66 objectives that must be met, while there are none at E.

If one skips the certification effort – which is huge – and just writes level A code, some studies (e.g., “Safety Critical Software and Development Productivity” by O. Benediktsson – though this paper looks at IEC 61508 rather than DO-178B) show that with the use of highly-disciplined processes there is no extra cost to producing the best possible software. Use an ad hoc approach and there’s a 70% hit.

Let’s be pessimistic and assume the very best avionics costs twice that of typical commercial firmware. My data pegs the latter at $20 to $40 per line of code, from initial specification to shipping. Doubling the high end puts the cost at $80/LOC, or 15 times cheaper than Toyota’s most recent payout. Add in their other settlements, legal costs, lost sales, bad PR, and, oh, yeah, the actual firmware engineering, and that difference grows dramatically.

Take your pick: $1200+++/LOC for crappy code, or $80– for world-class.

One wonders why we as an industry continue to flail around with poor practices that are ultimately expensive and even deadly, when there is a body of knowledge that is provably effective.

23 thoughts on “Toyota’s Expensive Software

  1. Jack,

    You mentioned aspects such as lost sales, bad PR, legal costs, etc. for Toyota. No doubt these will all hurt.

    My first thought was the company's valuation: while several of the major indices are up 5-10% since the October verdict against Toyota, T

    Log in to Reply
  2. Jack

    This is a very interesting take on software costs. Unfortunately the cost/benefit is normally done at the start of the project rather than after something like this happens.

    While Toyota undoubtedly made some mistakes, I am not aware of any “sm

    Log in to Reply
  3. A comment from the aerospace industry –

    If you think that the cost of doing aircraft maintenance and certification is high, just look at the costs of dealing with the accident that happens if you don't.

    As you have been saying for years, it will take a f

    Log in to Reply
  4. You are all being too much engineer and not enough marketing/sales.

    IF Toyota spent the time & money on doing it “right”, there wouldn't be any left for their “great” commercials.

    It is all about “the show”.
    And whenever

    Log in to Reply
  5. You are all being too much engineer and not enough marketing/sales.

    IF Toyota spent the time & money on doing it “right”, there wouldn't be any left for their “great” commercials.

    It is all about “the show”.
    And whenever

    Log in to Reply
  6. The Space Shuttle computer designers made another set of assumptions about Hardware, that most Software engineers do not assume — they assumed the hardware could, and would fail at some point in the project — As a result the software ran on three compute

    Log in to Reply
  7. Over and over we see examples of how our system is fundamentally flawed. Among the multitudes connected with technological developments, the percentage of those doing meaningful technical work is pitifully small. Not everyone else is along for the ride,

    Log in to Reply
  8. “the lecturer asked everyone who loves their mom & dad to raise their hands. He then said, “you all can leave! There is no place for that crap in business”

    I worked for a mid-sized software company for five years. They sold a very high

    Log in to Reply
  9. It is better to light one candle than to curse the darkness. If you don't like the system, change it! Start your own company and hire the kind of people who want to do meaningful technical work. If you're correct about the other companies having a very low

    Log in to Reply
  10. In the world who ever easily can change, put issues on that.so that change is the isseue for the problem,at the same time put issues on invisiable things lik software.everybady thinks that software is easy.but its not like that.every one look about cost wh

    Log in to Reply
  11. Far too many development projects are focused on providing the specified function over the specified operational conditions.

    The important thing is to design for failure. What happens when something goes out of range or breaks. Not in the spec, not my j

    Log in to Reply
  12. Far too many development projects are focused on providing the specified function over the specified operational conditions.

    The important thing is to design for failure. What happens when something goes out of range or breaks. Not in the spec, not my j

    Log in to Reply
  13. Far too many development projects are focused on providing the specified function over the specified operational conditions.

    The important thing is to design for failure. What happens when something goes out of range or breaks. Not in the spec, not my j

    Log in to Reply
  14. Far too many development projects are focused on providing the specified function over the specified operational conditions.

    The important thing is to design for failure. What happens when something goes out of range or breaks. Not in the spec, not my j

    Log in to Reply
  15. Far too many development projects are focused on providing the specified function over the specified operational conditions.

    The important thing is to design for failure. What happens when something goes out of range or breaks. Not in the spec, not my j

    Log in to Reply
  16. Far too many development projects are focused on providing the specified function over the specified operational conditions.

    The important thing is to design for failure. What happens when something goes out of range or breaks. Not in the spec, not my j

    Log in to Reply
  17. Whilst the discussion is interesting, no mention as yet has been made about either hardware or software architecture.
    It has been my experience over a working life time with mission critical systems both host and embedded that if you architect to deal with

    Log in to Reply
  18. Whilst the discussion is interesting, no mention as yet has been made about either hardware or software architecture.
    It has been my experience over a working life time with mission critical systems both host and embedded that if you architect to deal with

    Log in to Reply
  19. “Thank you, Jack. Very sobering from a cost standpoint, but worse in my view is the loss of life and seriousness of the injuries that resulted from the bugs. No amount of money will bring back a loved one or compensate adequately for the permanent damage t

    Log in to Reply
  20. “A really good article on a topic, which are frequently ignored by the high level managers. I believe that this cases root cause is a similar thing. Deadline pressure and “I believe this is OK now” decision from a manager without following safe practices

    Log in to Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.