Firmware basics for the boss

January 22, 2004

Jack Ganssle-January 22, 2004

If your boss understands the beast that is firmware development, your team can be more productive. Here's what the boss needs to know.

I hear from plenty of readers that their bosses just don't "get" software. Efforts to institute even limited methods to produce better code are thwarted by well-meaning but uninformed managers chanting the "can't you just write more code?" mantra.

Yet when I talk to the bosses many admit they simply don't know the rules of the game. Software engineering isn't like building widgets or designing circuit boards. The disciplines are quite different, the techniques and tools vary, and the people themselves all too often quirky and resistant to standard management ploys. Most haven't the time or patience to study dry tomes or keep up with the standard journals. So this month and next I'm doing a short intro to the subject. Here's the first installment; give it to your boss.

Firmware costs
So, dear boss, assuming you're reading this, the first message is one you already know. Firmware is the most expensive thing in the universe. Building embedded code will burn through your engineering budget at a rate matched only by a young gold-digger enjoying her barely sentient ancient billionaire's fortune.

Most commercial firmware costs around $15 to $30 per line, measured from the start of a project till it's shipped. When developers tell you they can "code that puppy over the weekend," be very afraid. When they estimate $5/line, they're on drugs or not thinking clearly. Defense work with its attendant reams of documentation might run upwards of $100 per line or more; the space shuttle code is closer to $1,000 per line, but is without a doubt the best code ever written.

The rate of $15 to $30 per line translates into a six-figure budget for even a tiny 5k line application. The moral: embarking on any development endeavor without a clear strategy is a sure path to squandering vast sums.

Like the company that asked me to evaluate a project that was five years late and looked more hopeless by the day. I recommended they trash the $40 million effort and start over, which they did. Or the startup that, despite my best efforts to convince them otherwise, believed the consultants' insanely optimistic schedule. They're now out of business—the startup, that is. The consultants are thriving.

Version control
First, before even thinking about building any sort of software, install and have your people use a version control system (VCS). Building even the smallest project without a VCS is a waste of time and an exercise in futility.

The NEAR spacecraft dumped a great deal of its fuel and was nearly lost when an accelerometer transient caused the on-board firmware to execute abort code—incorrect abort code, software that had never really been tested. Two versions of the 1.11 flight software existed; unhappily, the wrong set flew. The code was maintained on uncontrolled servers. Anyone could, and did, change the software. Without adequate version control, it wasn't clear what made up correct shipping software.

A properly deployed VCS ensures these sorts of dumb mistakes just don't happen. The VCS is a sort of database for software, releasing the code to users but tracking who changed what when. Why did the latest set of changes break working code? The VCS will report what changed, who did it, and when, giving the team a chance to efficiently troubleshoot things.

Maybe you're shipping release 2.34, but one user desperately requires the old 2.1 software. Perhaps a bug snuck in sometime in the last 10 versions and you need to know which code is safe. A VCS reconstructs any version at any time.

Have you ever misplaced code? In October 1999 the FAA announced they had lost the source code to all of the software that controlled air traffic between Chicago and the regional airports. The code all lived on one developer's machine, one angry person who quit and deleted it all. He did, however, install it on his home computer, encrypted. The FBI spent six months reverse engineering the encryption key to get their code back. Sound like disciplined software development? Maybe not.

Without a VCS, a failure of any engineer's computer will mean you lose code, since it's all inevitably scattered around amongst the development team. Theft or a fire—unhappily everyday occurrences in the real world—might bankrupt you. The computers have little value, but that source code is worth millions.

The version control database—the central repository of all of your valuable software—lives on a single server. Daily backups of that machine, stored offsite, ensures your business's survival despite almost any calamity.

Some developers complain that the VCS won't protect them from lazy programmers who cheat the system. You or your team lead should audit the VCS's logs occasionally to be sure developers aren't checking out modules and leaving them on their own computers. A report that takes just seconds to produce will tell you who hasn't checked in code, and how long it has been out on their own computers.

Version control systems range in price from free (like the GNU products) to expensive, but even the expensive ones are cheap. For a comprehensive list of products, see

< Previous
Page 1 of 2
Next >

Loading comments...