Firmware basics for the boss

If your boss understands the beast that is firmware development, your team can be more productive. Here's what the boss needs to know.

I hear from plenty of readers that their bosses just don't “get” software. Efforts to institute even limited methods to produce better code are thwarted by well-meaning but uninformed managers chanting the “can't you just write more code?” mantra.

Yet when I talk to the bosses many admit they simply don't know the rules of the game. Software engineering isn't like building widgets or designing circuit boards. The disciplines are quite different, the techniques and tools vary, and the people themselves all too often quirky and resistant to standard management ploys. Most haven't the time or patience to study dry tomes or keep up with the standard journals. So this month and next I'm doing a short intro to the subject. Here's the first installment; give it to your boss.

Firmware costs
So, dear boss, assuming you're reading this, the first message is one you already know. Firmware is the most expensive thing in the universe. Building embedded code will burn through your engineering budget at a rate matched only by a young gold-digger enjoying her barely sentient ancient billionaire's fortune.

Most commercial firmware costs around $15 to $30 per line, measured from the start of a project till it's shipped. When developers tell you they can “code that puppy over the weekend,” be very afraid. When they estimate $5/line, they're on drugs or not thinking clearly. Defense work with its attendant reams of documentation might run upwards of $100 per line or more; the space shuttle code is closer to $1,000 per line, but is without a doubt the best code ever written.

The rate of $15 to $30 per line translates into a six-figure budget for even a tiny 5k line application. The moral: embarking on any development endeavor without a clear strategy is a sure path to squandering vast sums.

Like the company that asked me to evaluate a project that was five years late and looked more hopeless by the day. I recommended they trash the $40 million effort and start over, which they did. Or the startup that, despite my best efforts to convince them otherwise, believed the consultants' insanely optimistic schedule. They're now out of business—the startup, that is. The consultants are thriving.

Version control
First, before even thinking about building any sort of software, install and have your people use a version control system (VCS). Building even the smallest project without a VCS is a waste of time and an exercise in futility.

The NEAR spacecraft dumped a great deal of its fuel and was nearly lost when an accelerometer transient caused the on-board firmware to execute abort code—incorrect abort code, software that had never really been tested. Two versions of the 1.11 flight software existed; unhappily, the wrong set flew. The code was maintained on uncontrolled servers. Anyone could, and did, change the software. Without adequate version control, it wasn't clear what made up correct shipping software.

A properly deployed VCS ensures these sorts of dumb mistakes just don't happen. The VCS is a sort of database for software, releasing the code to users but tracking who changed what when. Why did the latest set of changes break working code? The VCS will report what changed, who did it, and when, giving the team a chance to efficiently troubleshoot things.

Maybe you're shipping release 2.34, but one user desperately requires the old 2.1 software. Perhaps a bug snuck in sometime in the last 10 versions and you need to know which code is safe. A VCS reconstructs any version at any time.

Have you ever misplaced code? In October 1999 the FAA announced they had lost the source code to all of the software that controlled air traffic between Chicago and the regional airports. The code all lived on one developer's machine, one angry person who quit and deleted it all. He did, however, install it on his home computer, encrypted. The FBI spent six months reverse engineering the encryption key to get their code back. Sound like disciplined software development? Maybe not.

Without a VCS, a failure of any engineer's computer will mean you lose code, since it's all inevitably scattered around amongst the development team. Theft or a fire—unhappily everyday occurrences in the real world—might bankrupt you. The computers have little value, but that source code is worth millions.

The version control database—the central repository of all of your valuable software—lives on a single server. Daily backups of that machine, stored offsite, ensures your business's survival despite almost any calamity.

Some developers complain that the VCS won't protect them from lazy programmers who cheat the system. You or your team lead should audit the VCS's logs occasionally to be sure developers aren't checking out modules and leaving them on their own computers. A report that takes just seconds to produce will tell you who hasn't checked in code, and how long it has been out on their own computers.

Version control systems range in price from free (like the GNU products) to expensive, but even the expensive ones are cheap. For a comprehensive list of products, see www.codeorganizer.com/version_control/tools.htm.

Firmware standards
Although English is spoken throughout America, try talking to randomstrangers on a street corner in Baltimore today. The dialects range fromeducated middle-American to incomprehensible dialects. It's allEnglish, of a sort, but it sounds more like the fallout from the Towerof Babel.

In the firmware world we speak a common language: C, C++, or assembly,usually. Yet there's no common dialect; developers exploit differentaspects of the lingoes or create their programs using legal butconfusing constructs.

The purpose of software is to work, of course, but also to clearlycommunicate the programmer's intentions to maintenance people. Clearcommunication means we must all use similar dialects. Someone—that'syou, boss—must specify the dialect.

The C and C++ languages are so conducive to abuse that there's a yearlyobfuscated C contest whose goal is to produce utterly obscure butworking code. Normally I don't publish the URL as these people are codeterrorists who should be hunted down and shot like the animals they are,but the examples are compellingly illustrative. To see how bad thingscan get, see www0.us.ioccc.org/2001/williams.c. And then vow that your group will produce world-class software that's cheap to maintain.

The code won't be readable unless we use constructs that don't cause oureyes to trip and stumble over unusual indentation, brace placement, andthe like. That means setting rules, a standard, used to guide thecreation of all new code.

The standard defines far more than stylistic issues. Deeply nestedconditionals, for instance, lead to far more testing permutations thanany normal person can manage. So the standard limits nesting. Itspecifies naming conventions for variables, promoting identifiers thathave real meaning. Tired of seeing i , ii , and (my personal favorite) iii for loop variable names? The standard outlaws such lazy practices.Rules define how to construct useful comments. Comments are an integraland essential part of the source code, every bit as important as for andwhile loops. Replace or retrain any team member who claims to write”self commenting code.”

Some developers use the excuse that it's too time consuming to produce astandard. Plenty exist on the net; mine is in Word doc format at www.ganssle.com/misc/fsm.doc. It contains the brace placement rule that infuriates the most people—you'll change it and make it your own.

So write or get a firmware standard. And boss, please work with your folks to make sure all new code follows the standard.

Code inspections
What's the cheapest way to get rid of bugs? The answer's simple: just don't put any in!

Trite, perhaps, yet there's more than a grain of wisdom there. Too manydevelopers crank lots of code fast and then spend ages fixing theirmistakes. The average project eats 50% of the schedule in debugging andtest. Reduce debugging by inserting fewer bugs, and you'll acceleratethe schedule.

Inspect all new code. That is, use a formal process that puts everyfunction in front of a group of developers before they spend any timedebugging. The best inspections use a team of about four people whoexamine every line of C in detail. They'll find most of the bugs beforetesting.

Study after study shows inspections are 20 times cheaper at eliminatingbugs than debugging. Maybe you're suspicious of the numbers—fine, divideby an order of magnitude. Inspections still shine, cutting debugging inhalf.

More compellingly it turns out that most debugging strategies nevercheck half the code. Things like deeply-nested IF statements andexception handlers are tough to test. My collection of embeddeddisasters shows a similar disturbing pattern: most stem from poorlyexecuted, pretty much untested error handlers.

Inspections and firmware standards go hand in hand. Neither workswithout the other. The inspections ensure programmers code to thestandard, and the standard eliminates inspection-time arguments overstylistic issues. If the code meets the standard, then no debates aboutsoftware styles are permitted.

Most developers hate inspections. Tough. You'll hear complaints thatthey take too long. Wrong. Well-paced inspection meetings examine 150lines of code per hour, a rate that's hardly difficult to maintain(that's 2.5 lines of C per minute), yet that costs the company only abuck or so per line. Assuming, of course, that the inspection has novalue at all, which we know is simply not true.

Your role, boss, is to grease the skids so the team efficiently cranksout fabulous software. Inspections are a vital part of that process.They won't replace debugging, but will find most of the bugs verycheaply.

Have your people look into inspections closely. The classic reference is Software Inspection by Gilb and Graham, but Karl Wiegers' newer and much more readable book Peer Reviews in Software: A Practical Guide targets teams of all sizes (including solo programmers).1, 2

Toss out bad code
A little bit of the software is responsible for most of the debuggingheadaches. When your developers are afraid to make the smallest changeto a module, that's a sure sign it's time to rewrite the offending code.

Developers tend to accept their mistakes, to attempt to beat lousy codeinto submission. It's a waste of time and energy. Barry Boehm showed in Software Engineering Economics that the crummy modules consume four times the development effort of any other module.3

Identify bad sections early, before wasting too much time on them, andthen recode. Count bug rates using bug tracking software. Histogram thenumbers occasionally to find those functions whose error rates scream”fix me!” and have the team recode.

Figure on tossing out about 5% of the system. Remember that Boehm showed this is much cheaper than trying to fix it.

Don't beat your folks up for the occasional function that's a bloodymess. They may have screwed up, but have learned a lot about what shouldhave been done. Use the experience as a chance to create a killerimplementation of the function, now that the issues are clearlyunderstood. Healthy teams use mistakes as learning experiences.

Use bug tracking software, such as the free bugzilla (www.bugzilla.org/) or any of dozens of commercial products (nice list at www.aptest.com/resources.html).

Even the most disciplined developers sometimes do horrible things in thelast few weeks to get the device out the door. Though no one condonesthese actions, fact is that quick hacks happen in the mad rush to ship.That's life. It's also death for software.

Quick hacks tend to accumulate. Version 1.0 is pretty clean, but theevil inflicted in the last few weeks of the project add to problemsinduced in 1.1, multiplied by an ever-increasing series of hacks addedto every release. Pretty soon the programming team says things like “wecan't maintain this junk anymore.” Then it's too late to take correctiveaction.

Acknowledge that some horrible things happened in the shipping mania.But before adding features or fixing bugs in the next release, give thedevelopers time to clean up the mess. Pay back the technical debt theyincurred in the previous version's end game. Otherwise these hacks willhaunt the system forever, reduce overall productivity as the teamstruggles with the lousy code in each maintenance cycle, and eventuallycause the code to rot to the point of uselessness.

Stay tuned for more next month.

Jack G. Ganssle is a lecturer and consultant on embeddeddevelopment issues. He conducts seminars on embedded systems and helpscompanies with their embedded challenges. Contact him at .

Further reading

  1. Gilb, Tom and Dorothy Graham. Software Inspection . Addison-Wesley: NY NY, 1993.
  2. Wiegers, Karl. Peer Reviews in Software: A Practical Guide . Addison-Wesley: NY Boehm, Barry. Software Engineering Economics . Prentice Hall PTR: Upper Saddle River, NJ, 1981.

Excellent article. As the only software engineer in the company, even inspecting a set of code youhaven't looked at in a while is a huge help. When writing new code, I've found that writing it and then testing it in one dayand then inspecting it the next morning works. Thanks again.

– Dan Stone


Next month, please include tips for bosses who used to be software engineers. Such managersnaturally profess to understand our jobs, yet the schedule still reigns supreme.

– David M. Tomer

Jack replies:
David, all the rules are the same. I'll just add one more: Be Brave. Stand up for what you know is right.Don't cave to unrealistic pressures.


I like a lot of what you said in this article, but I wanted to add a correlary to your comments on codeinspections. To have code inspections on at most 150 lines of code and to have them every time you want to submit fixes, Ithink its essential to reduce the overhead of the inspections.

On my project, the bureaucratic overhead of inspections are aweful. I can literally spend a whole day pulling together thecode packets making sure all of my SQA boxes are checked. Another several hours of effort are required to make sure all of theSQA forms are completed after the review to make sure all of the forms are completed.

This overhead kills. What happens is people stop following the 150 line maximum rule. I've gone to reviews where people haveliteral pages of software to inspect. Nobody comes prepared to such a review.

Or, nobody wants to make changes because they don't want to have an inspection. Software development and especially innovationslows to a hault. We are stuck with bad and bulky software because nobody wants to refactor.

Finally, who you invite to the review is critical. You want people to attend that know the code. I've gotten little valuableinput from people who were required to attend the review but who have little knowledge on the working software.

Maybe these issues are covered in the books you reference. I'll have to check that out.

– Scott Turley


Scott, it sounds like your problem isn't the code inspection, but thepaperwork you have to fill out to prove you did the inspection. Is thissome sort of government project? I spent a few years in defensecontracting (hardware, not software), and documenting that we'd done thejob right cost about twice as much as doing the job right…

– Mark Moss

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.