Ever wanted to tell your boss the truth about why products ship late? This month, Jack fires off that angry memo you've been dreaming about.
To: Bob Smith, CEO
From: Jake Schmidt
I wanted to respond to your memo of the 16th. Circulated as it was to seemingly half the company, I feel we developers are now operating under an insurmountable stigma. Yes, we all know the product shipped late. Very late. You want to know why the schedule was missed so badly, and want an action plan to ensure this problem will never reoccur.
I was just one of many developers on the project. Others on the team will probably respond with apologetic platitudes. However, you'll note my resignation is attached, so I feel no pressure to paper over the very real problems with politically correct, but worthless, suggestions.
Let me assure you that, contrary to your strongly stated opinion, we were not “spending half our time surfing the 'Net for porn instead of cranking code,” nor was the team composed of “laggards and slackers.” Therein lies the first problem: engineering was treated with contempt and disrespect throughout the task. We're college-educated, highly trained engineers with years of experience, yet we're viewed as cogs in the system, replaceable elements whose sole mission is to respond to marketing's latest harebrained wish-list.
The timecards show we averaged 70 hours a week valiantly trying to tame the beast. So maybe we did check personal e-mail once in a while. Big deal. Yet suspicion reigned: managers, including you, prowled the lab looking for evidence of slacking. When you didn't find us playing computer solitaire, you restricted 'Net access to avoid “extraneous” e-mails and blocked all non-corporate web sites. Professionals cannot work in an environment lacking trust.
W. Edwards Deming, the great quality guru, long ago examined factors influencing motivation. He found professionals get most of their drive from “intrinsic” motivating factors, those that come from within ourselves. Things like feeling part of the team. The desire for the organization's success. Wanting to deliver a great product. Intolerance of poor quality.
He described “extrinsic” motivators as those imposed from on-high, usually without any sort of buy-in from the professionals involved. Artificial measurements ranked high on his list of extrinsic motivators. Deming showed that extrinsic motivators invariably drive out the intrinsic ones. In other words, people shrug in frustration and work to meet the numbers, rather than try to make things right.
A prime example: your imposition of lines-of-code-per-day metrics. You wanted lots and lots of software, and we gave it to you. Oh man, did we crank code! Most of it sucked, of course, and the resulting clean-up is still incomplete.
Joe's suicide and the 45% turnover all stemmed from the lack of control and the frustration we felt. Which leads me to my next point.
Turnover was a huge problem, but almost worse was the company's unwillingness to train replacements. The shortage of decent embedded people meant we relied on headhunters and huge signing bonuses (we called these “newbie bait”). Hiring costs exceeded $50,000 per position, which is actually not out of line in this industry. Yet each new person was immediately demotivated by being tossed into the fray without even a second's worth of training.
We largely succeeded in getting the best and the brightest. At 8 a.m. they reported for the first day of work; by 9 they were expected to be productive. None knew our software standards or our process or our technology. The resulting anger and resentment formed a beautiful feedback loop back into the turnover problem.
Our $50,000-per-person expenditures were for naught. Wasted.
The turnover and lack of training were just another bit of fuel for the impending disaster. Your obtuse decision to create the code in C++ helped us build the bloated monster no one comprehends. I'll quote Joe's suicide e-note: “What kind of an idiot makes fundamental technical decisions based on one Business Week article?”
Yes, the magazine correctly reported that object-oriented programming holds the promise of aiding in the creation of reuseable software components. And yes, that is indeed a very good thing. And C++ is certainly an appropriate language for OOP.
Remember Billy, the new college grad? The one whose fiance arrived late one night waving a steak knife, demanding he get some at-home time? (She's better, by the way, and should be free in a few months.) Well, Billy was the only one here with any OO experience. With the tight schedule we were somehow supposed to master OOP, which is a whole new approach to design and coding, while we created the project. That's simply impossible.
People learn new design approaches like OOP by building small projects and tossing them away. Instead, we were told to turn our first experience with this very different concept into a shipping product. Where's the chance for the mistakes and missteps that are a natural part of learning?
Joe had requested a month of C++ training for all of the developers before we started. You axed that, citing schedule pressure. Now we're a year late and the system still doesn't work. Dare I suggest that the training month would have leveraged the schedule in wonderful ways?
So there's another dysfunctional feedback loop: your lack of respect for us causes you to disregard our expert recommendations (for training, in this case) which makes us late, lowering your respect for us.
We developers were also always puzzled by the surreal management structure that seemed to make all of us report to those 23-year-old marketing whizzes in their fancy suits who kept making astonishing promises to the customers. (I know the Department of Defense is an important customer, but are we still telling them we'll cram Linux into an 8051?) Every whim from these geniuses resulted in a mandatory new feature. It wasn't unusual to see them standing over the shoulders of a developer suggesting-nay, demanding-changes that very instant. We were forced to implement these in an ad hoc manner, with no corresponding schedule changes.
Sales wasn't much better. Why is it they phrase all requests with the disclaimer “I'll never be able to sell that piece of junk unless you add such and such?”
All of us acknowledge that changes are a part of life. We know that it's awfully hard to specify a complex system, so some amount of system evolution is inevitable. But there's always a cost: adding features makes the project take longer. Period. Since change is unavoidable, important, and expensive, we've got to manage it. I suggest instituting standard change control procedures. This means that each request is approved by the project manager after we've looked at its impact, both on schedule/cost and system implications. This would minimize embarrassing moments, like when marketing browbeat the apps guys into adding an e-mail client, when the system never had, nor ever would have, an Internet connection.
As it was, the list of features grew faster than a lottery winner's circle of friends. So fast, in fact, that no one to this day knows what all of the requirements really are. A more disciplined approach is to manage the features and prioritize their implementation. Instead of having hundreds of partially working cool must-haves, we'd get the essentials all working properly, and only then move on to those less important. We could have shipped a bug-free subset of the product long ago, adding upgrades-correct, working upgrades-regularly.
Lest you think this is merely an anti-management polemic, I'd suggest another source of trouble was too little oversight. Twenty developers reported to dearly departed Joe, without intermediate supervisors. No one can manage that many people effectively.
Joe, for all of his virtues, was at heart and in practice an electronic engineer who'd rather be building analog circuits than running a big firmware project. Our previous embedded versions of the system succeeded because they were small. Dramatic overtime is enough to get 10,000 lines of C running; this is simply not the case when you're dealing with 250,000 lines. I suspect development by heroics appeals to management because the engineers seem so obviously dedicated, but the process just does not scale.
Yet Joe knew how to listen. We convinced him of the importance of creating some sort of development process. It was pretty lightweight, actually, a clear coding standard, rigorous code inspections, and a version control system.
Remember the pre-VCS days? When we shipped five different but defective versions of the code to that one customer? No one knew how to get a correct build. Or the time those two ZZ-Top look-alikes lost it, smashed their keyboards on their PCs, and then rode their Hogs right through the cubicle walls, destroying the server? A VCS would have made backups easier, so all that data wouldn't have been lost. I can't believe you hired them back as consultants.
One success was that, once implemented, our VCS worked well.
We tried to institute firmware standards, to ensure that our code was clean and consistent. It failed due to lack of accountability, a critical element of managing anything. In software engineering one important way to ensure things are done right is to review work products, like the code. In the absence of these reviews, there is no accountability other than “well, it seems to work.” So when Joe was busy dealing with the problems of 20 developers, he was unable to shepherd his flock towards doing things right all of the time. Developers, driven to panic by growing product size and your pronouncements of doom if we slipped, dropped the firmware standards and the reviews.
I'd be the last to suggest nepotism, but when you replaced Joe after his sad demise with your ex-auto-mechanic cousin Vinny, all hopes of instituting a development process crashed. Vinny's favorite saying was “yo, this can't be any harder than rebuilding the carb on a '67 Impala. What's wrong with youse guys?” Then he'd stand behind the nervous engineer, swinging a crescent wrench. In a desperate bid to look busy, the developers dropped any pretense at maintaining the process. Vinny wanted to see us crank code.
We needed a coach, someone to help us do the right things every day, rather than a recently paroled felon whose admittedly profound jailhouse skills didn't port well to the lab.
So, with neither standards nor inspections, the system grew creaky and ponderous. Bugs surfaced with alarming frequency, the rate accelerating as time went on. An intriguing study by Caper Jones showed that, in a survey of 4,000 projects, the number one cause of slipped dates was poor quality. Bugs. Sound familiar?
The company's attitude toward debugging tools appalls me. Telling us “instead of spending $10,000 on debuggers, just stop making bugs” was nave. Study after study shows that even with the best development processes, we're going to have to ferret out some bugs. Even decent code inspections-had we used them-would eliminate only 70% of the defects. Traditional debugging is a part of life.
Our choice is simple: buy decent tools or bloat the schedule. A lot. Remember the $50,000 per engineer hiring costs? Tools are cheap by comparison.
As you seem to have forgotten, the delivery date was capriciously assigned by you. Rather than developing one from a list of requirements followed by a careful design, you told us the ship date was to be June 15. To our shame, we didn't walk out en masse, but spent three days nervously moving little triangles around in the project management software to make a schedule that looked believable, all the while knowing each milestone was a sham, the totality a blatant lie.
And so, I close with the suggestion that you replace Vinny with a strong, highly ethical, and experienced manager of software engineering. Someone able to stand up to impossible demands from above, and impose and nurture a coherent development ethic from below.
To: Jake Schmidt
From: Bob Smith
Cc: Vinny DeAlonzo
With marketing making demands for the new version of the product and sales complaining they can't sell the old one, I'm a bit too busy to read your most recent memo, though I'm sure it contains much useful information. I'll get to it soon, no doubt.
However, I've promised the customer we can deliver another new upgrade in three months. Please go ahead and create an appropriate schedule and get started. We'll work out the features and specs later.
Business Week suggests that Java will let us run on many different platforms. So write it all in Java. The magazine said something about garbage collection, whatever that is, but I figure if you don't make any garbage, there won't be any problems. Why can't you engineers think of these solutions? Plan on porting this code to that old 8051 design after you get Linux running on it.
I've been thinking about how to keep this new project on-schedule. From now on, all developers will eat lunch at their desks and keep working. That, and some more overtime, should do the trick.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at .