How embedded projects run into trouble: Jack’s Top Ten – Number Two
As we close in on the top reason firmware projects get into trouble, this week I’ll cover the honorable mention.
2 – Quality gets lip service
Two decades ago I sold my in-circuit emulator business and went on a quest for firmware quality. Emulators are (well, were, before they became obsolete) tough. Since they plugged into a huge range of customer CPU sockets and varied board design, customer support was a never-ending activity. We traveled to a lot of sites, and, in working with these usually very-smart engineers, I often found a lack of focus on firmware quality.
In the last 20 years I’ve been to hundreds of companies on six continents, and constantly see the same problem.
Back in the 1960s when my dad bought a new car it went back to the dealer time after time to have manufacturing defects repaired. But that was OK; all cars had the same issues. Everyone expected a stream of fixes.
Then the Japanese started selling cars here, for less money. And they never went back to the dealer. Detroit freaked out and sent armies of management consultants to Japan to figure out how they pulled off this miracle of high quality and low price. They found that management there had learned, ironically from Americans (e.g., Deming, Juran, etc.), that the pursuit of quality always leads to lower prices.
Today you can go to pretty much any manufacturing facility and see the walls covered with graphs and statistics. Quality is the goal, and it is demanded, measured, and practiced.
But go into a firmware lab and you’ll find no metrics. The company will have some airy goal including the word “quality,” but with no concrete goals, at least when it comes to the code. You can’t manage something, like quality, if you don’t measure it. Saying “we build world-class code” is as meaningless as “nice first date; I’ll call you sometime.” Better: “our defect removal efficiency will be 99%, which puts us in the top 0.4% of all software organizations.” (See this for more on DRE).
Absent this, quality will take a back seat to cranking code. Required steps will be cut on the cross of the schedule. For instance, when test is left to the end, and project runs late, guess what isn’t done?
One thing the agile folks get right is their insistence on test. They require developers to test all the time.
But test is not enough. Quality code happens when we view development as a series of filters designed to, first, not inject errors, and second, to find them when they do appear.
But why use a lot of filters when we don’t know the efficacy of these? Well, actually, we do. See Economics in the references for quantitative information on this.
Did you know most firmware projects chew up half the schedule in debugging? I guess that means the other half is bugging. It seems logical, and the data bears this out, that if we can reduce the bugging we’ll greatly accelerate the schedule. And, with fewer bugs to find, deliver a much better product.
In manufacturing, specific practices are followed to ensure a high-quality product. You don’t randomly decide to use a hammer instead of a pick-and-place machine. In software engineering, too, we have practices that, if followed, will generally result in great code. There are many; the agile people have some great ideas, though I do think that they often rely solely on test to the exclusion of other important filters. Others, like the Personal Software Process (see references) have known results and are intrinsically coupled with continuous metrics tracking.
There’s a parallel to the early days of the automobile. Cars were assembled by hand. Craftsmen spent much time filing parts to fit. Henry Ford wanted a very inexpensive car, so he demanded that the parts be of great precision, which could be installed with no hand fitting. Though the demand for precision sounds expensive, getting rid of the handwork let him deliver excellent cars for a cheap price. (See The Perfectionists in the references for a wonderful book about this.)
Next week, the zinger, the most common way firmware projects implode. Stay tuned!
- The Economics of Software Quality, Capers Jones
- A Discipline for Software Engineering, Watts Humphrey
- The Perfectionists, Simon Winchester.