Getting disciplined about embedded software development
The seduction of the keyboard has been the downfall of all too many embedded projects. Writing code is fun. It's satisfying. We feel we're making progress on the project. Our bosses, all too often unskilled in the nuances of building firmware, look on approvingly, smiling because we're clearly accomplishing something worthwhile.
As a young developer working on assembly language-based systems I learned to expect long debugging sessions. Crank some code, and figure on months making it work. Debugging is hard work so I learned to budget 50% of the project time to chasing down problems.
Years later, while making and selling emulators, I saw this pattern repeated, constantly, in virtually every company I worked with. In fact, this very approach to building firmware is a godsend to the tool companies who all thrive on developers' poor practices and resulting sea of bugs. Without bugs debugger vendors would be peddling pencils.
A quarter century after my own first dysfunctional development projects, in my travels lecturing to embedded designers, I find the pattern remains unbroken. The rush to write code overwhelms all common sense.
The overused word "process" (note only the word is overused; the concept itself is sadly neglected in the firmware world) has garnered enough attention that some developers claim to have institutionalized a reasonable way to create software. Under close questioning, though, the majority of these admits to applying their rules in a haphazard manner.
When the pressure heats up—the very time when sticking to a system that works is most needed—most succumb to the temptation to drop the systems and just crank out code.
Any Idiot Can Write Code
In their studies of programmer productivity, Tom DeMarco and Tim Lister (authors of Peopleware) found that all things being equal, programmers with a mere 6 months of experience typically perform as well as those with a year, a decade, or more.
As we developers age we get more experience—but usually the same experience, repeated time after time. As our careers progress we justify our escalating salaries by our perceived increasing wisdom and effectiveness. Yet the data suggests that the value of experience is a myth .
Unless we're prepared to find new and better ways to create firmware, and until we implement these improved methods, we're no more than a step above the wild-eyed teen-age guru who lives on Jolt and Twinkies while churning out astonishing amounts of code.
Any idiot can create code; professionals find ways to consistently create high quality software on time and on budget.
Firmware Is the Most Expensive Thing in the Universe
Norman Augustine, former CEO of Lockheed Martin, tells a revealing story about a problem encountered by the defense community. A high performance fighter aircraft is a delicate balance of conflicting needs: fuel range versus performance, speed versus weight.
It seems that by the late 1970s fighters were about as heavy as they'd ever be. Contractors, always pursuing larger profits, looked in vain for something they could add that cost a lot, but which weighed nothing.
The answer: firmware. Infinite cost, zero mass. Avionics now accounts for more than 40% of a fighter's cost.
Two decades later nothing has changed except that firmware is even more expensive.
What Does Firmware Cost?
Bell Labs found that to achieve 1-2 defects per 1000 lines of code they produce 150-300 lines per month. Depending on salaries and overhead, this equates to a cost of around $25-50 per line of code.
Despite a lot of unfair bad press, IBM's space shuttle control software is remarkably error free, and may represent the best firmware ever written. The cost? $1000 per statement, for no more than one defect per 10,000 lines.
Little research exists on embedded systems. After asking for a per-line cost of firmware I'm usually met with a blank stare followed by an absurdly low number. "$2 a line, I guess" is common. Yet, a few more questions (How many people? How long from inception to shipping?) reveal numbers an order of magnitude higher.
Anecdotal evidence, crudely adjusted for reality, suggests that if you figure your code costs $5 a line you're lying—or the code is junk. At $100/line you're writing software documented almost to DOD standards. Most embedded projects wind up somewhere in-between, in the $20-40/line range.
There are a few gurus out there who consistently do produce quality code much cheaper than this, but they're on the 1% asymptote of the bell curve.
If you feel you're in that select group—we all do—take data for a year or two. Then, measure time spent on a project from inception to completion (with all bugs fixed) and divide by the program's size.
Apply your loaded salary numbers (usually around twice the number on your paycheck stub).
You'll be surprised.
Currently no items