In my last real job I developed and sold in-circuit emulators (ICEs). Talk about a tough product! An ICE designer needs insight into every quirk of the target processor, which never works quite as advertised. The firmware in the ICE is huge and hideously complex, while insane timing requirements make the hardware equally difficult; just the sort of thing that appeals to geeks like me. But over the years, I found an even more fascinating aspect of the business: the customers.
Anyone using a debugging tool is living on the edge of product development. They're in that difficult phase of the project where things just don't work. Typically the schedule is in a state of collapse, the boss's temper flares, and everyone is tired from working too much overtime. In my role as an ICE vendor, I worked with thousands of these developers, looking metaphorically or physically over their shoulders as they struggled to complete their projects. I found that few—astonishingly few—followed any sort of even vaguely-disciplined process. Yet the minority who did use reasonable methods outperformed all other developers by astonishing margins. They delivered good code fast. The difference in performance of the two sorts of teams was staggering.
It's incredibly satisfying to see how a little bit of information and encouragement can change a development team. But a couple of times a month I hear from a group who can neither attend one of my lectures nor come to an Embedded Systems Conference because they're desperately behind on the latest project. Further inquiry generally shows that the group is always in panic mode. There's no time to learn how better development techniques can bring projects to fruition on time.
Like sailors on a sinking ship, these developers are too busy bailing to fix the leak. The water slowly rises so they bail ever more frantically. Sooner or later they're going down, but working faster and harder staves off the inevitable end for just a while longer.
An old cartoon shows a fierce battle, the soldiers wielding swords and spears. The general turns away a machine-gun salesman. He complains: “I don't have time to talk to you—can't you see we're fighting?”
The recent Boston Embedded Systems Conference, for instance, had over 75 classes on all aspects of building embedded systems. No other venue packs so much information into a single week. About 5,000 people attended. That's a healthy increase over last year, but, in my opinion, an appallingly low number.
We're part of a fraternity of about a quarter million embedded developers. Why is it only 2% of us take advantage of this incredible learning opportunity? Is everyone too busy bailing?
Why are so many firmware projects so late and so bug-ridden? A lot of theories abound about software's complexity and other contributing factors, but I believe the proximate cause is that coding is not something suited to Homo sapiens . It requires a level of accuracy that is truly super-human.
Cavemen didn't have to bag every gazelle they hunted—just enough to keep from starving. Farmers never expect the entire bag of seeds to sprout; a certain wastage is implicit and accepted. Any merchant providing a service expects to delight most, but not all, customers.
The kid who brings home straight A's (not mine, darn it) thrills his parents. Yet we get an A for being 90% correct. Perfection isn't required. Most endeavors in life succeed if we miss our mark by 10% or less.
Except in software. 90% correct is an utter disaster, resulting in an unusable product. 99.9% correct means we're shipping junk. A hundred thousand lines of code with 99.9% accuracy suggests some 100 lurking errors. That's not good enough. Software requires near perfection, which defies the nature of intrinsically error-prone people.
Software is also highly entropic. Anyone can write a perfect 100 line-of-code system, but as size soars, perfection or near perfection requires ever-increasing investments of energy. It's as if the bits are wandering cattle trying to bust free from the corral; coding cowboys work harder and harder to avoid strays as the size of the herd grows.
So what's the solution? Is there an answer?
In my opinion, getting software correct will always be a problem, and no silver bullet will ever satisfy all stakeholders. But some well-known, though rarely practiced, strategies offer hope.
The first is to fight featurism. A nifty feature is just a software change, right? We can slap in a bit of extra functionality in no time at all. This attitude has changed cell phones from basically simple devices to million-line-of-code firmware monstrosities.
The easiest feature to implement is the one that's not there. Prune unneeded requirements to get the code done better and faster. Unhappily, customers—or at least marketing departments—usually demand far more than any product really needs. Developers usually have little influence over the requirements.
Cheat complexity growth by breaking the system up into many small pieces, each of which is as independent as possible. Minimize coupling between functions and modules. As the firmware evolves, coupling invariably increases. We add a few more parameters in function calls, sneak in a global or 10, and morph the simple keypad driver into a display/keypad/LED handler. Though I struggle with a lot of what the eXtreme Programming (XP) folks mandate, they laudably require that crummy code gets refactored rather than beaten into submission.
Practice reuse. You won't; no one does. But reuse is, in my opinion, our only hope of salvation. Unfortunately, reuse goes against the nature of early 21st century capitalism, which stresses short-term profits above all. Reuse fails unless one makes a large up-front effort in generalizing functions. Few bosses accept the argument, “If we double development time on this project, we can halve it on the next dozen.”
Most importantly, use a disciplined development process.
In the olden days all products were handmade, the work of artistes rather than manufacturers. Plows, shears, and guns were all turned out one at a time, each was subtly different than the one the blacksmith finished just the day before. Over time industry learned about the value of making standardized, interchangeable parts, and ultimately about mass production. Though the retro movement has long admired this handwork, for most products, mass production yields far better products at much lower costs.
But not in software. We still view ourselves as artistes, cranking out handmade code highly optimized for each specific application. While an engineering department will never resemble a factory, the utter chaos that rules most firmware shops is unacceptable.
I estimate that under 2% of firmware developers have any sort of process they use religiously. It matters little what that approach may be: XP, CMM, inspections, and standards are all valuable ways to improve the code if used religiously. None offer much benefit when casually or intermittently employed.
Software artistes have missed the “process” boat. Most other industries use various sorts of defined processes to work efficiently. One way to identify an amateur organization of any sort, be they accountants, lawyers, craft shops, or software developers, is a lack of process. By contrast, an efficient company like McDonald's defines a rigorous way to do just about everything. Even a teenager, using Ray Kroc's process, can crank out Big Macs that taste exactly the same anywhere in the world.
Firmware developers must look beyond the world of software engineering for insight into better ways to build our products. I highly recommend Michael Gerber's book The E-Myth Revisited: Why Most Small Businesses Don't Work and What to Do About It . Skip the book's irrelevant last half. Gerber says that poor businesspeople work “in” the business—they're technicians who daily make the product or service. The business can't succeed without that individual, who may be a genius at providing a product or service but spends every day firefighting.
Gerber feels that brilliant company owners work “on” the business. They build systems, processes, and techniques so the business runs smoothly. These awesome managers don't just solve problems, they invent solutions that eliminate the problem forever or that automatically deal with the issue when it comes up again.
They stop bailing and plug the leaks.
We must adopt the same philosophy. For instance, I've observed that most teams spend 50% of the project debugging the code. Try something else—maybe pair programming or code inspections.
Developers often blame management for lousy development methods. They get neither support nor direction from the boss, who is preoccupied with just shipping the product no matter what it takes. So the developers employ heroics, working ever harder to produce bigger and more complex products.
We can't abdicate our responsibility to do things right. Another great management book is Quality Is Personal: A Foundation for Total Quality Management by Harry Roberts and Bernard F. Sergesketter. The authors contend that “working harder” is a lousy way to solve problems. The couch potato who promises to watch less TV will probably fail. Better: make a profound change. Throw the TV away, or put it on a timer. Dieting? Empty the fridge, buy only low-cal foods, don't carry money so you don't eat out, but stop making futile promises to do better.
The authors' philosophy applies to software engineering, too. Are variable names always totally meaningless? Skip that meeting where everyone promises to do a better job with variables on the next project; it'll fail. Instead, change something. Maybe adopt naming conventions and software standards. Appoint a naming czar. Build a tool that identifies stupid names.
If the boss is indeed capricious and unsupportive, if he thwarts every attempt to improve engineering, you can still change something. Adopt a stealth process, one that requires no buy-in from management, and one you can implement with neither his consent nor knowledge. A great example is Watts Humphrey's Personal Software Process (PSP). The PSP will improve your work even if everyone else on the team is a cowboy.
Most of the developers I talk to do have at least an inkling about how to get their efforts on track. Some understand the potential benefits from using a reasonable process, and others have adopted some or all of these techniques. The vast majority, though, sheepishly admit that they don't employ them religiously.
If the pimple-faced McDonald's worker doesn't change the deep-frying oil at prescribed intervals, all of the food will be even worse than normal. So the company has a system that ensures oil changes happen on-schedule, all of the time. If we don't adopt the same determination to always use a decent process—whatever it is—our code will be a mess, guaranteed. Cranking code is, to me, less interesting than finding better ways to crank code.
Stop bailing and plug the leaks.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at .
1. Gerber, Michael, The E-Myth Revisited: Why Most Small Businesses Don't Work and What to Do About It , New York, NY: HarperBusiness, 1995.
2. Roberts, Harry V. and Bernard F. Sergesketter. Quality Is Personal: A Foundation for Total Quality Management , New York, NY: Free Press, 1993.
I very much enjoyed your article, and I agree about the lack of discipline in the programmer's ranks. Ihave also worked with IT guys, and its even worse there.
As one of the few who has taken a disciplined approach to product development, I have always been viewed asdoing the work of two others, while working only 40-45 hours a week.
As a result, I have often been promoted to manager. But as a manager, I found that what the companies saythey want, and what they really want, are two different things.
First, companies say they want to develop products quickly and efficiently that will meet their customer'sneeds. In order to make that happen, contributions are required by marketing, sales, manufacturing,service, etc. The management in these functions are often not interested in bringing any change to theirorganizations, and they don't want to make any additional decisions. I have been left hanging many timeswhen VPs play their political games to protect their turf, but do nothing to improve the company overall.
Second, increasing the speed of product development means that critical decisions have to be make earlierthan people are used to, and a lot of management just doesn't like it. I remember one meeting where we hadto make a commitment to tooling six months in advance. The VP of Marketing had to make some hard decisionsabout the product before the tooling could be ordered. He delayed for months, blowing up the projectschedule, and screwing up the entire product development calendar for the next 12 months. He was promoted. I was fired.
Third, many CEOs and VPs still view long hours of work as the standard for excellence. Even though I haveoften done the work of two others, I have often been criticized for not working enough hours.
I could go on and on, but I don't want to bore you. Just as 2% of the programmers out there are disciplinedin their approach to product development, I suspect that less than 2% of the companies are disciplined intheir approach to product development. The rest want the benefits, but don't want to change theirorganizations to realize those benefits. They are content to continue hiring young engineers, make themwork way too many hours, and pat them on the back for being such good workers.
– Bill Phillips