Catastrophe Disentanglement - Embedded.com

Catastrophe Disentanglement

Correspondent Wil Blake sent along a link to an article with thecompelling name “AnIntroduction to Catastrophe Disentanglement. It's worth reading,especially if you've ever had to bail out a project that's in direstraits.

In it the author describes ten steps to take when a software projectis out of control. Start by stopping development. Yet in my experiencemanagement rarely has the courage to do this. Insteadincreasingly-desperate attempts to reign in a spiraling schedule alwayshappen within the context of “work harder, faster, more hours.”Sometimes features are appropriately sacrificed, but work continuesunabated.

The author's fourth step is to “evaluate the team.” Though that's awise move it glosses over an important part of the evaluation: firesomeone or some group of people. In many cases – not all – part of theteam knowingly made bad decisions.

Management creates schedules based on fantasies or sales squeezes inan impossible load of features. Catastrophes are ultimately peopleproblems (except in the unusual case where one is stumped by apreviously-unknown bit of science) so the people have to change or bechanged.

Such firings are rare, of course, with predictable results. Thepressures that originally created unrealistic estimates are nowexacerbated. The same people tend to respond to the same pressures inthe same dysfunctional manner.

Here's an example from last year. A large company spent 4 yearsdeveloping a replacement for a legacy embedded system. They hadhundreds of thousands of lines of code that “worked” but was riddledwith more bugs than the American embassy in the USSR.

Dozens of developers were each working 60 to 80 hours a weekstamping out defects, but as the testing grew more realistic problemsmushroomed. It became clear that the requirements were vague and poorlyunderstood. The product had to work within regulatory frameworks thatvaried widely in every country and even county to county in the US.

Often some very simpleminded tricks can reveal a lot about aproduct; in this case a grep showed over 2000 conditional compilerdirectives used to alter the system's performance based on locality,yet the test regime only included 150 different conditions. Clearlythere was still an awful lot of unvalidated code, likely to be asteeming with bugs as the rest had been.

For a number of reasons I recommended they toss the entire code baseand start over, this time using reasonable software processes. And Igave them a short list of people to either fire or move to anothergroup.

They did start over. But with the same team and the same management,who immediately proposed an unrealistic schedule and again bypassed anysort of serious requirements gathering and analysis. Last week I heardfrom one developer who told me the project is once again a quagmire.Sadly, I wasn't surprised.

People problems doom big software projects. We have only threechoices: replace, retrain or repeat the whole dismal experience.

Jack G. Ganssle is a lecturer and consultant on embeddeddevelopment issues. He conducts seminars on embedded systems and helpscompanies with their embedded challenges. Contact him at . His website is .


It really pains me to see this post.

It pains me because:

1. It happens all to often

2. Nobody seems to do, or want to do anything about it

3. This scenario all too often leads to the “Failed Startup” syndrome

4. In larger companies/institutions. The 'problem' eventually gets outsourced, and whole divisions/departments get scrapped, and really good people lose their jobs, and really crappy people get promotions.

5. People like me usually get called in to assess the problem, study and recommend. Whereas, I would much rather be part of a successful front-end!

If you wish to point fingers; all I can say, is that brown icky stuff tends to roll down hill!

When I get hired to do these assessments; People usually want me to 'divine' for them where the fault lies. Usually, I do not oblige. Usually, I will attempt at providing some insight on how to get the situation back on track.

– Ken Wada


“and the same management, who immediately proposed an unrealistic schedule and again bypassed any sort of serious requirements gathering and analysis. Last week I heard from one developer who told me the project is once again a quagmire. Sadly, I wasn't surprised.”

This says it pretty much dead on why what I've worked on has not been as good as it could have been, That and trying to do too much with too little funding.

– Bill Wallace


In my experience, the one of the key causes of failure is unrealistic schedules (the other being incomplete or incorrect requirements). The end date is always fixed, due to a trade show or a date promised to the customer by sales and/or upper management. This results in lots of OT (and my company doesn't pay for OT) and unhappy employees, who end up adapted a “couldn't care less” approach just as things get hot.

On one project, sales and management promised a customer a sample within 6 months of a new product in the beginnings of development. Engineers toiled late nights and weekends, technicians were cutting traces and soldering jumpers to modify hardware as a new board layout couldn't be done in time. Some features were barely implemented, but we shipped only 1 day late, which was due to a customs issue (a management oversight, but the engineers got the blame).

Three months later, our management enquired with the customer as to how they liked the sample, what features they were using, what additional features they wanted. The customer informed us they hadn't even powered the sample up, as they thought it was a “physical sample” only and non-functional. When the engineers heard that, there was much grumbling, especially from those that worked lots of (unpaid) OT.

So, the question is, how can we engineers ensure that our management creates realistic schedules? Maybe after that, we can tackle the requirements issue, too.

– John Patrick


I have to wonder if many of these so-called catastrophes arise from the fact that the vast majority of us are paid to manage or spend someone else's money.

Think for a second – suppose you were spending your very own money on such a project – wouldn't there be a greater incentive to get things right in the first place, or come up with an effective solution to fix a problem before it becomes a big one?

That is probably why projects where the developers have a significant stake (typical of smaller companies) tend to do better.

Just a thought.

– Joe Sallmen


(Me too! Me too!)

On top of all that, if you get hired to fix a hardware/software development problem, and if you don't hand the client a working product before the first invoice, even if you take their product farther in one month than it's gone in the last 7 months, the client is still likely to be “shocked” by the invoiced amount.

Thus, I think it's important to keep in mind that clients one attempts “rescue” typically have weary wallets and huge expectations. When considering projects such as these, be on the lookout for the ChroSVoN. That's Chronic Swirling Vortex of Need.

– Greg Feneis


Good advice. Especially in an election year.

G Beattie

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.