Embedded systems disaster stories - Embedded.com

Embedded systems disaster stories


To err is human, to learn from other's mistakes divine. Here in the Soapbox you can go one step further—learn and share. We challenge you to share your embedded systems programming mistakes, disasters, and near misses whether they pertain to team miscommunication or technical errors. Reader Mike Moran starts us off with his thoughtful comments on some miscommunications with management in response to Jack Ganssle's “Disaster redux!” Please submit your experiences and we'll post them here.

Letter from a reader:
Management's job is to maximize a company's profits while minimizing liability and keeping an eye of cash flow, so the enterprise doesn't go bust. Engineering's job is to create products on a finite budget in a finite time scale.

Most of the time, these objectives don't mesh, not only due to the pressures of profitability but often caused by breakdowns in the management/engineering interface. Having said this, I realize my statement implies that there actually is a management/engineering interface, which is not always the case.

Some of my own experiences:

On one occasion, I wrote a capital equipment justification for an 8051 C compiler that I thought crucial to getting a project done on time. This was during the era when the debate over assembly/higher level language was far from settled. I was called in to discuss my proposal and was prepared to talk about the productivity enhancement of writing one line of C code versus several lines of assembly language. Instead, I was asked, “Mark, what's a complier?” (spelling intentional).

Later, during this same process, the company CFO who fancied himself quite the computer guru, walked into the CEO's office and plopped down a copy of Borland Turbo C version 1.0, which had just come on the market. (I'm really dating myself, aren't I?) He ordered it from a mail order house for something like $69.95. He loudly proclaimed that he had just saved the company the best part of the $1,495.00 that the Archimedes C compiler for 8051 would have cost us. I explained to these gentlemen that Turbo C emitted code for an 8086, and I needed code for an 8051, which was not the same thing. I was asked “well can't you just fix it up afterwards?” I'm not making this up. Eventually I got the Archimedes 8051 compiler, and the copy of Turbo C came in very handy for running my data acquisition board in the PC, so I guess I made out OK on that one.

Frequently, engineers are pushed by the CEO to “just gimme something for the trade show.” Sometimes you're forced to resort to all but the foot switch under the table to make it happen. On one such occasion I managed to produce something in the nick of time to get it shipped to McCormick Center in Chicago. It “sort of” worked. I had to go to the show myself to babysit the thing and shoo away too-curious interlopers. Upon returning to our plant, I stopped by our marketing office to inquire when the unit would be back. I was told that it had been drop shipped to a customer from the show floor. He had also ordered three more! (Guess I did a better job on it than I thought!)

Finally, I was out of town on a very tough service case that our field service couldn't handle. Our CEO, who had evidently broken his dialing finger, decided during my absence, to stroll out to engineering to get the scoop on the latest project from one of the technicians. I'm amazed he even found the place, but I digress… When asked when the prototype would be ready for customer trial the technician told the CEO: “I'm just waiting for Mark to give me a chip.” Thinking that it was simply an IC on backorder, said CEO phoned the customer to promise that the unit would be in their plant by the end of the week. The chip the technician referred to was in fact, a ROM chip. He didn't have it, because it was to contain the software that I had not even started on, much less tested. Two people with very narrow interests talking about important decisions is a disaster riding in on a pony.

Management has to have a sober, realistic understanding of what engineers really do out there in the back. No, they don't have to be able to write a line of code or design a network, but managers must listen, intelligently, dispassionately, and objectively. Engineers would help their cause immensely by speaking to management in language they can understand.

Real communication between management and engineering maximizes probability of success. Engineers feeling pressure to produce a schedule and budget that will “fly” are, in fact, writing a recipe for catastrophe. Managers accepting unrealistically resource budgets and overly ambitious schedules are inviting failure. Important stuff like code reviews and adequate testing are inevitable casualties of processes like this.

Management is the art of making decisions on inadequate and incomplete information. Engineering is the art of finding one solution out of the many possible that meets all constraints in an acceptable way. Left to their own devices, engineers will engineer a product to the nth degree. Management will ship a product whether its ready or not, after all, product in the lab instead of the marketplace isn't making any money. The trick is striking the balance between these unacceptable extremes.

Leave your ego, emotions, reputation, and other issues at the door. Each side ought to be willing and able to consult independent experts who can provide them with a bird's eye view and some fresh insights before things get to the desperate stage. It's sort of like code review, they don't do any good if you don't do them. It also helps your friendly local consultant make those Florida condo payments.

Best Regards,

Mark Moran
Eastern Regional Sales Manager
IAR Systems Software

Reader Response

I liked and agreed with most of Mark's letter, but as vice president of my firm, I can assure you I do not resemble the management types in your story. You see, we are not all stupid idiot boobs completely ignorant of technology, as you imply. I have a B.S.E.E. from University of California, Irvine, and also an MBA from the same institution. I also have 25+ years as a design engineer and project manager. So you see, us management types are not all the way you have portrayed us.

Kamran Kazem, V.P.
Magnetic Design Labs, Inc.

I've said it before and I'll say it again. Why aren't the managers engineers? Why isn't the CEO and engineer with umpteen years of field experience? Because engineers have given up their pre-eminence. They have no one to blame but themselves. Next time you start a company with your idea, don't give up to an MBA.

Phil Gillaspy

I agree with Mark Moran.

The real problem out there is the disconnect between the front and back offices. Like Mark I have worked in organizations where the management had no idea what engineering is or what it does. Further, they often had no empathy for the engineer as a person. I offer two examples.

I left a large company to work at a small startup company doing oil patch work. A bit later two talented engineers from my old company joined the startup to work in my area. I learned so much from one of them that I later went on to do what I consider leadership work at subsequent companies for which I've worked. One day these two engineers related the story of the day they gave notice at the large company. Both of these men gave notice on the same day and both of them, singularly, were key persons on key projects within the large company. Neither knew the other had accepted the position for which they were giving notice. Upon receiving notice their supervisor became so concerned he escalated the issue up the management ladder. The manager met with them and in separate meetings attempted to counter-offer them. The counter-offer provided to each of them was “Well I know you guys are programmers, and programmers like to have disk space. How about I get you more disk space?”I was astonished but was assured that this was in fact true and had happened to each of the engineers. The manager who told them this was amazingly the supervisor I had initially worked for when I worked at the large company and is now the CEO of another company.

The other example of the disconnect occurred while I was working at a software company. This company was on the trade show circuit and because of that was burning up software engineers like crazy trying to keep up with trade show-driven deadlines. One day an e-mail from above came around that some method to prevent system crashes be found and that this project be given the top priority. About a day later someone responded with an e-mail that read “Please place the power cables in such a way to prevent people from tripping over them”.

I know that these may be mundane examples of management-engineering communication faults, and that every engineer and manager out there has a similar story, but that is the Defect, capitol D. In Japan managers sit with, eat with, and drink with their staff. Their system may have problems but management to engineering communication is not oneof them. So it isn't an endemic problem; it doesn't have to be this way. In America, management, driven by Harvard seminars, are trying to Mcdonaldfy the computing industry, every industry really. IBM has a project out there called 'Autonomic Computing', take a look. But call their help desk, or look at the byline of their redbooks, and you'll find that they rely on foreign talent to pursue their lofty ideas.

Long live Dilbert. May his reign be fruitful.

Wayne Johnson

This is an excellent article, hitting the key point of communication between managers and engineers.

Fuyuan Luo

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.