Getting disciplined about embedded software development -

Getting disciplined about embedded software development

The seduction of the keyboard has been the downfall of all too many embedded projects. Writing code is fun. It's satisfying. We feel we're making progress on the project. Our bosses, all too often unskilled in the nuances of building firmware, look on approvingly, smiling because we're clearly accomplishing something worthwhile.

As a young developer working on assembly language-based systems I learned to expect long debugging sessions. Crank some code, and figure on months making it work. Debugging is hard work so I learned to budget 50% of the project time to chasing down problems.

Years later, while making and selling emulators, I saw this pattern repeated, constantly, in virtually every company I worked with. In fact, this very approach to building firmware is a godsend to the tool companies who all thrive on developers' poor practices and resulting sea of bugs. Without bugs debugger vendors would be peddling pencils.

A quarter century after my own first dysfunctional development projects, in my travels lecturing to embedded designers, I find the pattern remains unbroken. The rush to write code overwhelms all common sense.

The overused word “process” (note only the word is overused; the concept itself is sadly neglected in the firmware world ) has garnered enough attention that some developers claim to have institutionalized a reasonable way to create software. Under close questioning, though, the majority of these admits to applying their rules in a haphazard manner.

When the pressure heats up—the very time when sticking to a system that works is most needed—most succumb to the temptation to drop the systems and just crank out code.

Any Idiot Can Write Code
In their studies of programmer productivity, Tom DeMarco and Tim Lister (authors of Peopleware ) found that all things being equal, programmers with a mere 6 months of experience typically perform as well as those with a year, a decade, or more.

As we developers age we get more experience—but usually the same experience, repeated time after time. As our careers progress we justify our escalating salaries by our perceived increasing wisdom and effectiveness. Yet the data suggests that the value of experience is a myth .

Unless we're prepared to find new and better ways to create firmware, and until we implement these improved methods, we're no more than a step above the wild-eyed teen-age guru who lives on Jolt and Twinkies while churning out astonishing amounts of code.

Any idiot can create code; professionals find ways to consistently create high quality software on time and on budget.

Firmware Is the Most Expensive Thing in the Universe
Norman Augustine, former CEO of Lockheed Martin, tells a revealing story about a problem encountered by the defense community. A high performance fighter aircraft is a delicate balance of conflicting needs: fuel range versus performance, speed versus weight.

It seems that by the late 1970s fighters were about as heavy as they'd ever be. Contractors, always pursuing larger profits, looked in vain for something they could add that cost a lot, but which weighed nothing.

The answer: firmware. Infinite cost, zero mass. Avionics now accounts for more than 40% of a fighter's cost.

Two decades later nothing has changed except that firmware is even more expensive.

What Does Firmware Cost?
Bell Labs found that to achieve 1-2 defects per 1000 lines of code they produce 150-300 lines per month. Depending on salaries and overhead, this equates to a cost of around $25-50 per line of code.

Despite a lot of unfair bad press, IBM's space shuttle control software is remarkably error free, and may represent the best firmware ever written. The cost? $1000 per statement, for no more than one defect per 10,000 lines.

Little research exists on embedded systems. After asking for a per-line cost of firmware I'm usually met with a blank stare followed by an absurdly low number. “$2 a line, I guess” is common. Yet, a few more questions (How many people? How long from inception to shipping? ) reveal numbers an order of magnitude higher.

Anecdotal evidence, crudely adjusted for reality, suggests that if you figure your code costs $5 a line you're lying—or the code is junk. At $100/line you're writing software documented almost to DOD standards. Most embedded projects wind up somewhere in-between, in the $20-40/line range.

There are a few gurus out there who consistently do produce quality code much cheaper than this, but they're on the 1% asymptote of the bell curve.

If you feel you're in that select group—we all do—take data for a year or two. Then, measure time spent on a project from inception to completion (with all bugs fixed) and divide by the program's size.

Apply your loaded salary numbers (usually around twice the number on your paycheck stub ).

You'll be surprised.

Quality Is Nice As Long As It's Free
The cost data just described is correlated to a quality level. Since few embedded folks measure bug rates, it's all but impossible to add the quality measure into the anecdotal costs. But quality does indeed have a cost.

We can't talk about quality without defining it. Our intuitive feel that a bug-free program is a high quality program is simply wrong. Unless you're using the Netscape “give it away for free and make it up in volume” model, we write firmware for one reason only: profits. Without profits the engineering budget gets trimmed. Without profits the business eventually fails and we're out looking for work.

Happy customers make for successful products and businesses. The customer's delight with our product is the ultimate and only important measure of quality.

Thus: the quality of a product is exactly what the customer says it is .

Obvious software bugs surely mean poor quality. A lousy user interface equates to poor quality. If the product doesn't quite serve the buyer's needs, the product is defective.

It matters little whether our code is flaky or marketing overpromised or the product's spec missed the mark. The company is at risk due to a quality problem so we've all got to take action to cure the problem.

No-fault divorce and no-fault insurance acknowledge the harsh realities of transmillennium life. We need a no-fault approach to quality as well, to recognize that no matter where the problem came from, we've all got to take action to cure the defects and delight the customer.

This means when marketing comes in a week before delivery with new requirements, a mature response from engineering is not a stream of obscenities. Maybe just maybe marketing has a point. We make mistakes (and spend heavily on debugging tools to fix them). So do marketing and sales.

Substitute an assessment of the proposed change for curses. Quality is not free. If the product will not satisfy the customer as designed, if it's not till a week before shipment that these truths become evident, then let marketing and other departments know the impact on the cost and the schedule.

Funny as the Dilbert comic strip is, it does a horrible disservice to the engineering community by reinforcing the hostility between engineers and the rest of the company.

The last thing we need is more confrontation, cynicism, and lack of cooperation between departments. We're on a mission: make the customer happy ! That's the only way to consistently drive up our stock options, bonuses, and job security.

Unhappily, Dilbert does portray too many companies all too accurately. If your outfit requires heroics all the time, if there's no (polite) communication between departments, then something is broken. Fix it or leave.

Few would deny that firmware is a disaster area, with poor quality products getting to market late and over budget. Don't become resigned to the status quo. As engineers we're paid to solve problems. No problem is greater, no problem is more important, than finding or inventing faster, better ways to create code.

The Software Engineering Institute's Capability Maturity Model Integration (CMMI) defines five levels of software maturity and outlines a plan to move up the scale to higher, more effective levels:

1. Initial : Ad hoc and Chaotic. Few processes are defined, and success depends more on individual heroic efforts than on following a process and using a synergistic team effort.

2. Repeatable : Intuitive. Basic project management processes are established to track cost, schedule, and functionality. Planning and managing new products are based on experience with similar projects.

3. Defined : Standard and Consistent. Processes for management and engineering are documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing software.

4. Managed : Predictable. Detailed software process and product quality metrics establish the quantitative evaluation foundation. Meaningful variations in process performance can be distinguished from random noise, and trends in process and product qualities can be predicted.

5. Optimizing : Characterized by Continuous Improvement. The organization has quantitative feedback systems in place to identify process weaknesses and strengthen them pro-actively. Project teams analyze defects to determine their causes; software processes are evaluated and updated to prevent known types of defects from recurring.

( Captain Tom Schorsch of the US Air Force realized that the CMMI is just an optimistic subset of the true universe of development models. He discovered the CIMM—Capability Integration Immaturity Model—which adds four levels from 0 to _3:

0. Negligent: Indifference. Failure to allow successful development process to succeed. All problems are perceived to be technical problems. Managerial and quality assurance activities are deemed to be overhead and superfluous to the task of software development process.

1. Obstructive: Counter Productive. Counter productive processes are imposed. Processes are rigidly defined and adherence to the form is stressed. Ritualistic ceremonies abound. Collective management precludes assigning responsibility.

2. Contemptuous: Arrogance. Disregard for good software engineering institutionalized. Complete schism between software development activities and software process improvement activities. Complete lack of a training program.

3. Undermining: Sabotage. Total neglect of own charter, conscious discrediting of organization's software process improvement efforts. Rewarding failure and poor performance.

If you've been in this business for a while, this extension to the CMMI may be a little too accurate to be funny


The idea behind the CMMI is to find a defined way to predictably make good software. The words “predictable” and “consistently” are the keynotes of the CMMI. Even the most dysfunctional teams have occasional successes—generally surprising everyone. The key is to change the way we build embedded systems so we are consistently successful, and so we can reliably predict the code's characteristics (deadlines, bug rates, cost, etc.).

Figure 6.1 below shows the result of using the tenants of the CMMI in achieving schedule and cost goals. In fact, Level 5 organizations don't always deliver on time. The probability of being on time, though, is high and the typical error bands low.

Compare this to the performance of a Level 1 (Initial) team. The odds of success are about the same as at the craps tables in Las Vegas. A 1997 survey in EE Times confirms this data in its report that 80% of embedded systems are delivered late.

Figure 6.1 : Improving the process improves the odds of meeting goals and narrows the error bands

One study of companies progressing along the rungs of the CMMI found per year results of:

37% gain in productivity
18% more defects found pre-test
19% reduction in time to market
45% reduction in customer-found defects

It's pretty hard to argue with results like these. Yet the vast majority of organizations are at Level 1. In my discussions with embedded folks I've found most are only vaguely aware of the CMMI. An obvious moral is to study constantly. Keep up with the state of the art of software development.

At the risk of being proclaimed a heretic and being burned at the stake of political incorrectness, I advise most companies to be wary of the CMMI. Despite its obvious benefits, the pursuit of CMMI is a difficult road all too many companies just cannot navigate. Problems include:

1. Without deep management commitment CMMI is doomed to failure. Since management rarely understands—or even cares about—the issues in creating high quality software, their tepid buy-in all too often collapses when under fire from looming deadlines.

2. The path from level to level is long and torturous. Without a passionate technical visionary guiding the way and rallying the troops, individual engineers may lose hope and fall back on their old, dysfunctional software habits.

CMMI is a tool. Nothing more. Study it. Pull good ideas from it. Proselytize its virtues to your management. But have a backup plan you can realistically implement now to start building better code immediately. Postponing improvement while you “analyze options” or “study the field ” always leads back to the status quo. Act now!

To read Part 2 , go to The Seven-Step Plan

To read Part 3 , go to The value of postmortems.

This series of three articles was printed with permission from Newnes, a division of Elsevier, Copyright 2008, from “The Art of Designing Embedded Systems, Second Edition” by Jack Ganssle. For more information about this title and other similar books, go to

With 30 years in this field Jack was one of the first embedded developers. He writes a monthly column in Embedded Systems Design about integration issues, and is the author of several embedded books, including: The Art of Designing Embedded Systems and The Art of Programming Embedded Systems. Jack conducts one-day training seminars that show developers how to develop better firmware, faster. His most recent column on this topic is The Non-Quality Revolution. .

At the Embedded Systems Conference 2010 in San Jose, Ca., Jack will be conducting a number of classes on firmware programming issues, including: “Managing Firmware Projects [ESC-102]”; “Mars ate my spacecraft [ESC-304]”; and “Really real time systems [ESC-402]”.

Jack will also be a participant in a panel session on “The State of Embedded [SFT-12].”

5 thoughts on “Getting disciplined about embedded software development

  1. The fundamental weakness of CMMI and all such genre industry standards is that they place no constraint on the prime contractor/customer. The prime does not have to follow CMMI themselves. They are therefore at total liberty to change requirements without

    Log in to Reply
  2. There's a certain irony here, in that the links to parts 2 and 3 and broken.

    The easiest way to turn a churn-it-out-coder into a disciplined professional: make them write unit-tests with at least 85% code coverage. And then have other developers verify t

    Log in to Reply
  3. I think that experience is undervalued because there is not a large enough sample of software developers with a lot of experience. I have been programming for over 42 years – 20 years of assembly language on more architectures than most engineers can name,

    Log in to Reply
  4. “A small company building a small product can't afford to be rigorous with its processes…”

    Turn that around. A small company with small resources can't afford to be wasteful or make expensive mistakes.

    Consider the physical world “artist/craftsman” ma

    Log in to Reply
  5. Modern embedded systems are typically supported microcontrollers however standard microprocessors also are still common, particularly in additional complicated systems. Since the embedded system is devoted to specific tasks, style engineers will optimize i

    Log in to Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.