There was a time when a “database” was a drawer of index cards. Word processing meant creating text on a typewriter, using carbon paper instead of a “print 3 copies” command. The only intelligence in a car resided in the brain of the driver. Factory controllers used banks of relays singing their clicking songs.
Mainframe computers were generally inaccessible to ordinary people. Most folks experienced computing disasters only in the form of an insane credit card bill or the IRS' repeated demands for immediate payment of a zero dollar tax due.
Today we're not far from computerized car brakes sans backup hydraulic system. All modern aircraft can and do fly themselves and many can even land without human intervention. A 100,000-ton tanker heavily laden with a potential environmental catastrophe relies entirely on autopilot, GPS and radar for navigation and collision avoidance. Factories producing the most noxious and toxic of chemicals would grind to a standstill — or perhaps fail spectacularly — without an array of microprocessors that sequence every activity.
The firmware component of these systems has all grown spectacularly in the quarter century since the advent of the microprocessor, from programs consisting of but a few thousand lines of assembly to today's hundreds of thousands of lines of C or C++. We've learned many things over that time about building better code; one is that “perfect” software is a practical impossibility. The best code ever written is probably that in the Space Shuttle, but even there, even after spending $1000 per line, they still run into defects, though at the amazingly low rate of about 1 bug per 400,000 lines.
One ABS brake vendor told me their software is junk. “It's not a problem,” he went on, “because the hydraulic system takes over when the code crashes.” The rate of change of customer demands far exceeds our ability to learn and implement better design processes; one wonders how this organization will cope with demands for safe braking without mechanical backups.
All large systems have lurking hidden problems. Most of us work hard to ensure we've addressed real safety issues. None of us knows how to prove we've built something that is correct and that will never fail.
I was struck by a letter in the RISKS digest about the most common security problem found in Unix and Windows systems. A submission from Henry Baker suggests that programmers who produce software that does not check for buffer overflows are criminally negligent and should perhaps be liable for resulting damages. He makes the very interesting point that it's truly trivial to check incoming data streams for length, that we've known to do this for a generation or more, and yet an endless succession of bug reports out of Redmond and CERT testify that programmers totally neglect this obvious problem.
One or two buffer overflow problems in IE that cause crashes or security vulnerabilities are just bugs, easily correctable, and perhaps do not reflect on the developers' abilities or ethics. Repeated buffer overflow bugs, though, are a different animal indeed. I would think that at some point, after a history of years of identical problems in IE and far too many other similar programs, the team leader might suggest auditing the code. Look at every input and make sure this common problem cannot occur.
That Redmond and most other providers of PC programs do not take such action suggests they either have no clue what they're doing, or simply don't care. That is, they are negligent. If such a simple and easily avoidable bug causes a corporation to lose data, does the corporation have a case against the developers?
Extending this line of reasoning a bit, the software community has learned that a disciplined development process yields better code, fewer bugs. Yet very few of us employ a rigid process. For instance, we know from countless studies that keeping functions to a single page reduces defects. How many of us enforce a short-functions requirement? Code testing is a notoriously ineffective way to uncover bugs. How many of us couple effective tests with code coverage checks or inspections?
If bugs result from cavalier engineering, who is responsible?
In America most change today seems driven by litigation or the fear of it. Today, an engineer appearing in court is usually an expert witness. Will we soon be the defendants?
I have unbounded faith in our ingenuity to produce really fine products that are awfully safe. But since bugs cannot be completely eradicated, sooner or later the law of averages – or ambulance-chasing lawyers – will catch up with us. What then?
What do you think?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at . His website is .
Unchecked buffer overflows are indeed inexcusable. The Ada programming language solved the problem by making the compiler responsible for inserting run-time checks. With a good optimizing compiler, it didn't even cost anything in terms of final code efficiency, because the optimizer used global data flow analysis to find and remove checks that could never fail. It was a very rigorous automatic process. Unfortunately, Ada was unable to overcome the entrenched popularity of C and C++.
Maybe if programmers were held accountable for bugs, we'd see a sudden revival of interest in sound programming languages and tools. I hate abusive litigation as much as anyone, but what is there, short of the spectre of litigation, to finally force good software engineering practices to be taken seriously?
Good article as usual. Very thought provoking.
For me it boils down to if any claims were made that the code or product was tested to some standard or is 'guaranteed' to alarm in 100 mSec or something.
Antilock brakes had better perform to some standard by the automaker, US DOT, or something. Else they are a risk to all of us on the road. Ditto with planes, trains and Conestoga wagons with airbags. The individual team should not be responsible, but the company employing them may well be liable if their product was unleashed on the public in an unsafe condition. This is no different than recalls for microwave ovens with faulty interlocks or lawn chairs that fold up and amputate your fingers. Bad products will exist and software is yet another component in a product that can be the failure point. Critical systems or components should always fail 'safe' when possible. Redundant safety measures are often good engineering when cost allows.
Much of this does boil down to cost (surprise) and the better designed, tested, safe products do tend to cost more. People often pay more for that security, durability or whatever. The auto market is full of examples of different levels of safety and buyers at all levels.
Another excellent article Jack.
The simple answer here is that the software 'profession' will kill easily hundreds, perhaps thousands, of people without fear of retribution. Why am I so confident in this prediction? Well, technological advancement has done it all before – just read Professor Nancy Leveson's article on 'Steam'.
Quick Summary: The steam engine industry killed loads of people, over several years, before governments acted. Problems were caused by commercial pressures (excuse the pun), lack of professional standards and poor application of engineering theory. Sound familiar?
Software Team Leader, UK
A good question again!
For an engineer – like me – those USA style Software License Agreements havebeen ununderstandable since they appeared. If an architect or an electricalengineer draws something, sings it then sells it, it is a clear picture. Heis responsible for it. If the building falls down, he goes to prison.Firmware cases should also go like this. But there are two ways:
1.) The vendors, whose products contain embedded software take fullresponsibility for the complete product, including material and workmanship.In this case the software bug is an internal issue.
2.) They give Limited Responsibility Statement, stating “We did our bests,but heaven knows…” In this case those products will be banned by themarket in that very moment. Except for… If all the manufacturers playthis shameful game, users will swallow this frog – as we hungarians say.