Old-timers remember the pre-Flash days, when firmware lived in EPROMs. Compile the code. Test it. Ship it. Bugs? The customer lived with them until we were able to Fedex new chips.
Today Flash means we never have to say we’re done. Consumers are encouraged to frequently check for firmware updates. Is your router acting strangely? Download new code. The BIOS interacts with some peripheral? Get thee to the vendor’s web site for a firmware update.
Before Flash, updates were horrendously expensive. The vendors shipped new EPROMs to the entire customer base; to people who were expected to dismantle a complex system, extract one or more socketed chips, and install new ones without bending leads or zapping fragile electronics with static. Today the reverse is true.
Though the vendor might send an email to the customer base about bug fixes, the users must go to the company’s web site and get new files themselves. Somehow the customer/provider relationship has changed. The responsibility for getting a fix installed belongs to the end-user.
I see an anti-parallel to the auto industry of the 70s. Detroit shipped cars full of annoying defects, counting on dealers and customers to resolve the problems. Then the Japanese demonstrated a new paradigm of providing perfect cars cheaply. Today PC users, and it seems more and more embedded systems consumers, are also expected to work their way through a long punch-list of defects.
In the 70s American automakers and consumers thought there was no practical solution to the problem of shoddy cars. Consumers were resigned to a series of dealer visits during the first few months of new ownership. If we today assume that big systems are inherently buggy, will some brilliant upstart prove us wrong? America is the world’s largest exporter of software. Is that hegemony as tenuous as we found Detroit’s to be?
I wonder if in the pre-Flash days the huge costs behind an upgrade made developers more careful in testing their code. Now that it’s easy to do an upgrade, the temptation to ship early is powerful. Only a very strong team, with enlightened support from management, can resist the ship-urge.
NASA routinely sends spacecraft off on long voyages with incomplete code. Why not use a two year coast phase to cheat the schedule? Every brand-new PC’s software is hopelessly outdated the minute a consumer opens the box. And more than a few embedded apps get shipped long before their time, buggy and incomplete.
UPS takes five days to ship a package across country. Does your company use that week to play with the schedule? Ship it, sure, but then use a heroic sprint to really finish the code and email updates?
I know of only one case where this approach was totally successful. We sent a big system to the Middle East by ocean-going ship, a 30 day trip we used to finish the software. But (fortunately) the ship sank, providing us a much-needed two extra months of development time. Rumors that the vessel’s plating had been compromised by the software team were vigorously denied.
But you can’t count on those kinds of providential events. Be suspicious if the boss adds “ship sinks” to week 32 of the Pert chart.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at . His website is .
Ha! I've always suspected that those who make a living creating PERT charts of the perfect project were terrorists of some ilk. This is the first time I've heard about scuttling the delivery vessel though. That's pretty creative. Perverse but creative. Funny thing about PERT charts is that none of them have automatic built in fudge factors that the creator cannot bypass. I try to do what I can by multiplying all my estimates by a factor of four and scheduling a minimum of three turns at normal turn around times (2 weeks)for even the simplist of PCB's. If a specification change occurs that affects the board, then I put three more turns in. (Of course most project managers either ignore my input or come round to threaten me with a career change if I don't revise my estimates to more “reasonable” numbers. This usually requires throwing money at the “problem” by using quick turns. I keep all emails concerning this so I can rub their nose in them on my way out the door at the end of the project, during the “blame fixing” phase.) Whenever a PCB comes in without flaws after the first turn, it is time to go down to the nearest watering hole a buy the whole crew a refresahing libation. I've only had to do that once or twice in thirty years. What has that to do with software and flash upgrades? I suspect they are caused by the same thing, overly optimistic schedules. Or as someone else put it, The first 90% of the project takes 90% of the time allowed, the last 10% takes the “other” 90%.
The IEEE and others are trying to figure out how to get more respect for engineers, but that's unlikely to happen when our own managers have no respect for us.
– Don McCallum
Don McCallum's last comment has hit the nail on the head. Engineers know how long it takes to do things right, but managers march to the beat of a different drum, and don't (in general – there are exceptions) give engineers the respect we deserve. Managers know when it has to be done by, so that becomes the plan, regardless of team size or ability. Engineers understand the problem and the tools available to create the solution (the team size and ability, the technical/tool resources available, stuff that can be re-used etc), tell the managers how long it will really take, then have to take the flack when the manager's plan goes down the pan.
Ah well. I guess I should go cool off a little now…
– Paul Tiplady
I think this one tops the ship sinking:
The past owner of my previous employer once shipped a empty box toour largest customer.
From the outside it had all of the connectors and every thing that should have been there,but on the inside they went no place at all.
The project manger at the customer was aware of the subterfuge, and went a long with it.It got his purchasing people off his back, and ours, for meeting order dead line.
A few days after the dead line the real unit was swapped for the empty box.
– Bob Paddock
I once had to climb a tower in the Everglades to replaced a PROM, and I guarantee the PROM mentality meant better quality. But it also meant longer time to market, and fewer updates. But quit beating up on the managers, folks. 🙂 As someone who was an engineer, then a manger, and then an engineer again, allow me to offer you a three different perspectives. 1) Engineers are known for over-estimate the time it takes for them to complete a task (the so-called Scotty principal). Managers have a tendency to know this, and shave your time accordingly. We engineers did this to ourselves to some extent. 2) There is a great quote from an engineer who worked on the Lunar Lander (the LEM) who said something like “no one knew how long it took to build a lunar lander, we just knew we had six months to do it.” Given a real-world deadline, we engineers always seem to magically make it (actually, it sweat and inspired ingenuity, not magic). The art of a good project manager is to balance real deadlines with engineering estimates, and refrain from giving really smart people really arbitrary deadlines. 3) Engineers sometimes have trouble with knowing when something is done, and so do managers. Engineers think it's done when it's flawless. But I once worked with an executive who wanted to fire all the engineers after we shipped product because “it works, so it must be done.” The truth is often in the middle. I saw to it that no one was fired, BTW, at great personal expense. Not all managers are bad at understanding the issues here. 🙂
Jack has a knack for bringing up issues that make us think, doesn't he?
– Eric Uner