How embedded projects run into trouble: Jack’s Top Ten – Number Ten

August 08, 2018

Jack Ganssle-August 08, 2018

David Letterman had his top ten list that he ran through each night. I, too, have a top ten list, but it is a compendium of how embedded projects go wrong. I’ll tease you, gentle readers, by going through it one item per week.

So, here we go. Number ten on my list of how embedded projects run into trouble is…

10 - Not enough resources allocated to a project

Sure, a small embedded project can be done on the cheap, and they frequently are. Stuff a PIC10 with a few pages of code into a simple controller and a single developer using free tools can whack out the hardware and software in no time.

But most embedded systems are sizeable. Starve the effort of people, money, tools, management support, and failure is all but guaranteed.

Let’s start with tools. It’s puzzling that a $30k oscilloscope is not hard to justify, but the same team will be unable to spend a few grand for a software tool. People are expensive. Overhead is expensive. Failure is devastatingly so.

But tools are cheap.

A device that yields even a 5% productivity boost is worth on the order of $5k to $10k per year.

I had this discussion with an old friend this week. The company he is consulting for isn’t willing to buy a commercial IDE for an MSP430, so he’s working with poorer tools. At his $150/hr billing rate any obstacle will cost the company big bucks. But those are future dollars, dollars that they don’t have to account for, at least in the present.

Then there are people. Developers are so expensive that management routinely understaffs. Capers Jones showed that the average embedded project of 1 million lines of code needs about 2500 person-months of staff time, and costs $34m in salary and overhead. But, typically, management will figure on a much smaller group which will only, ultimately, increase costs.

I saw a four-person firmware team trying to get a project of 250KLOC shipped. A year went by. Then another. Promises to customers were made and changed, again and again. Eventually the two-year schedule morphed into seven elapsed years. Their customers were very upset, management nearly apoplectic, and in the end the project cost about twice the industry average. Understaffing always results in higher costs.

Aggravating the problem, developers are often tasked with mundane activities better left to support staff. Ironically, the computer revolution we helped usher in has turned us into secretaries. We do our own typing, filing, drafting, researching parts costs, and all the other activities that were once handled by lower-wage workers.

Then there’s management support. A crucial role of a boss is to shield the team from anything that distracts them from getting the work done. Yet too often marketing droids incessantly tap on developers’ shoulder demanding a feature change. When I was a young engineer, sales would demand some new functionality, always chanting “I can’t sell this thing without that feature.”

We figured that really meant “I can’t sell.”

The boss’s role is in no small part to shield the developers from random feature requests.

Then there’s money. In consulting the money issues are quite stark: bid a fixed price contract and watch the customer faint. They’re figuring on $10K but the effort will be an order of magnitude more expensive. “How can we make it cheaper?” is a valid question, but the truth is, firmware is expensive. Industry figures peg the cost from the start of a project to delivery at $20 to $40 per line of code.

A colleague complained that his company couldn’t afford the $15k needed to buy a commercial RTOS, so his team would write their own. I checked: the commercial version was 6000 lines of code. Multiply that by $20-$40/line and that $15k looks like a blue light special.

Firmware is expensive. If a company embarks on a project they need to admit the costs will be substantial and provide the resources needed to get it done properly. Penny pinching always winds up costing more.

So that’s number 10 on my list. Going forward each item gets progressively more effective at project-killing.

Jack Ganssle ( writes, lectures, and ruminates about embedded systems. His blog is here:


Loading comments...