Back in the '70s, when co-development meant fending off dinosaurs as we punched out our programs on paper tape, I worked on a system that used 16 — count 'em — sixteen 1702A EPROMs as the program store, for a breathtaking total of 4k of memory. It's hard to believe that a $20 chip requiring three power supplies held only 256 bytes of code, but the microprocessor industry was in its infancy then.
This was a physically large system that analyzed the chemical makeup of coatings on nylon. The customer was anxious to get the device and our under-funded company was desperate to ship so we could put the receivable on the books. It went early, of course, before the firmware was complete, sent as cargo on a ship. The voyage to the Middle East was to take a month, during which time we figured we could finish the code and then air freight a final memory card to the customer.
The ship sank a week into the voyage. Insurance paid our company and the resulting delays bought us the time we needed to get the code right. We were lucky.
Unfortunately, you can't count on disasters at sea as a scheduling mechanism.
Eventually the program worked correctly and we committed it to masked-ROM to reduce production costs. The stockroom had thousands of sets of chips. The smallest software change obsoleted all of those parts. Though management never understood that firmware changes, feature enhancements, and bug fixes were hugely expensive in engineering time, they were deathly afraid of tossing out so much inventory. For the only time in my career I was in a situation where no one ever said, “hey, it's only a software change.”
Today better technology — like flash memory — has removed the inventory constraint to code changes. It's only a software change; we can download new code in seconds, deliver fixes and enhancements via the Internet, and can even return to the glory days of self-modifying code. We can ship early, before the system works, via UPS Ground, an analog to the old slow-boat-to-China, and email code that works as the system arrives in the customer's anxious hands.
I'd never advocate getting rid of reprogrammable memory devices: they offer astounding benefits. But an underlying cost ignored by virtually all managers is that it's too damn easy to figure a minor enhancement is “only a software change.”
Engineering is generally a fixed expense in most organizations; the cost to develop and maintain a product is rarely amortized over the number of units sold. Using reprogrammable parts means there's no cost to making changes, or at least no cost that's traceable to the product. We developers understand that the very simplest of feature changes might require weeks or months of expensive engineering, but the product manager gets the upgrade apparently for free. Why not endlessly needle us with requests for new features?
I hear engineering managers complain bitterly about the cost of flash memory; in some products it's nearly the largest cost driver in the system. A scarier cost, in my opinion, is that it reinforces the myth of “it's only a software change.”
It's a fact that even a single-bit change to a program may require weeks of validation, integration, testing, and productization. Managers just don't seem to get this. Masked ROM gave them a cost they could understand; reprogrammable chips don't.
The sense is that software changes are free. That implies engineering is free. QED. Yet they're dumping engineers in droves, because we're expensive assets. Where's the logic in all of this?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at . His website is .
My second job was the most dismal for software changes. Senior management of Sundstrand got caught charging all their executive perks to a government contract, and fined 110 million dollars. Rather than sell off part of the company to pay the fine right away, they payed the fine, and then froze hiring. This meant that the project I was on was left to limp along for 7 more years with a staff of 2 engineers.
It was sheer misery doing incremental release after incremental release, rather than staffing up, and doing it right once.