At April's Embedded System Conference in San Francisco I moderated a “shop talk,” a round-table discussion on strategies for building firmware. Scheduled for 7:00 a.m., I expected a half dozen bleary-eyed attendees, but was delightfully surprised to find the room packed, standing room only, with a lot of passionate and thoughtful engineers.
Is reliable software an oxymoron? Many of the attendees seemed to think so. Several made the point that bugs in shipped products, even in mission-critical devices, are inevitable. One avionics designer said the 747's user manual has an entire section listing software anomalies, with appropriate work-arounds. I started thinking about driving cross-country to the next conference, until I realized that the car is equally chock-full of presumably buggy embedded code.
Last year I talked to a manufacturer of ABS brakes who said their code isn't really all that great, but since it's backed up by a manual hydraulic system no one cares. Guess what? In 2002 at least one car will have a totally electronic braking system. Yikes!
I shudder when people accept bugs as a fact of life in completed products. The high moral ground says we should ship stuff that works. Period. But… no one knows how to make a large project perfect. Even the Space Shuttle, which is surely the best code ever written in the history of the galaxy (or at least this solar system), has one bug per 400,000 lines. That's in code that costs $1,000 per line. Since commercial products run closer to $30/line, there must be a corresponding decrease in quality.
Douglass Adams , the ESC's incredibly entertaining keynote speaker and author of the Hitchhiker's Guide series, claimed in his address that we know when something is “technology” because it doesn't work. It's no longer technology when it fades into the background, when, like a pencil, it's something we don't even think about. Even kids know that when anything electronic acts odd, you cycle power.
One wag wrote: the mainframe industry spent 20 years making more fault-tolerant computers, while PC companies created more fault-tolerant users. More than a glimmer of truth, there, I'm afraid.
I was amazed, though, that the shop talk attendees all thought that technology wasn't the answer to the quality dilemma. There was a brief mention of OOP as a partial solution, but the conversation quickly moved on.
Everyone — at least all who spoke up — suggested that process and ethics were our only hope of creating good products. I've always thought proper process outweighs any tech tool any time, but had never really thought about the ethics issue before.
By and large we do know how to write damn good firmware. Maybe not perfect, but a lot better than much of the stuff being shipped today. The boss, though, makes rules that compromise quality. When we come up with a schedule based on a capriciously imposed deadline, we're really lying to ourselves and the boss. Is that ethical?
Our own natural propensity to take shortcuts hurts as well. Aren't we lying when creating a schedule without first doing a detailed design?
Is shipping a known defective product the right thing to do? How much responsibility should individual engineers take?
One attendee made an analogy to the Challenger explosion. Though launching in cold weather was a known hazardous condition, few engineers took a strong stand against the flight. Under pressure from management — that old bugaboo — most of the naysayers backed down.
But ethics is never easy. Ethical behavior means taking strong, difficult positions that are often contrary to our own short-term interests. That's tough, especially when most of us take pains to avoid confrontation.
Nancy Reagan suggested that teenagers “just say no.” Dare we try the same with our boss?
What do you think? Have you ever taken a strong stand against the boss's wishes? What happened?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at His website is .
I once worked at a firm which was to ship a project they knew would not be implemented. Their managerial position was for us to write code so that they could charge for the code lines based on quantity rather than functionality. All my code was then to print out (“printf()”) the story. if anyone bothers to inspect my code they will know exactly what they paid for. The very next bit of typing I did was to polish my resume so that I could get out of there!
C. O. M.
At all employers, I have taken the “Just say No” stand with the personsadvocating the release or shipment of product before I feel it is ready.
Saying “NO” is much more effective before the corporate memories forget (3-4months) the previous problems caused by meeting schedule dates and notestimatedates.
The down side of saying “no” after the corporate memory forgets theproblems ishaving to look for more work. Or is that the upside?
Keep up the good work.
Jack Responds: Also, remember that some day those managers will die,move to sales, or quit. Then you'll be boss. Are we going to be strongenough to take the right action when our engineers come to us in the samefix we're in now?
I completely agree with theethics standpoint. Invariablythe project schedules neverallow the required manhoursfor creating bug free code and theengimeers end up sending half-baked code to the customerjust to meet their schedule. Inadequate manpower addsto the problem.
ACES PVT LTD
Releasing bug-free software is very difficult. It is true thatirrespective of the tools and the processes, it is the commitment and the ethics which adds to bug_free software writing.
I'm relatively new to coding, having graduated only a couple ofyears back. But during the last 2 years I have had the experience of being with two employers. One followed software development procedures to the hilt, was rated at SEI CMM level 4, and the other rarely followed any processes.
The difference was the marked increase in the number of bugs/kloc for the latter (several times over).
In spite of being this obvious I'm not very sure why managers still are hesistant to ensure that software processes are implemented.
Just to relate an incident that happened to me twice while on vacation shows how much processes, in general, are important. I was twice offered already occupied rooms … a very embarassing situation indeed especially if it is late in the night. Facing such a situation the manager profusely apologised and offered sops for me stay on. But this could have easily been avoided by having a much more dependable system of keeping tabs on room occupancy.
Hope common sense prevails.
Sr. Development Eng