Nikon's brand new Coolpix 5000 has a crippling firmware bug that can render the camera completely useless. According to CNET and substantiated by Nikon, if a user turns the device off with the lens cap installed the camera will fail. Bummer.
Nikon very quickly responded with a firmware upgrade, with extensive instructions about how to download and install the patch via a USB connection. If the bug has already manifested itself, though, there's no way to install the fix. Customers must then return the unit for warranty repair.
There's no assurance that users with functioning cameras won't destroy the firmware by installing the upgrade. The company admits that a PC crash (not the most unlikely of events in the Windows world) during the USB download could trash the camera, again necessitating a return. Also quoted as possible problems during the download: the camera turning itself off, the USB cable coming adrift, or “performing the update in a place where there is excessive electrical noise,” a suitably vague circumstance probably not controllable by most non-techies.
A Nikon spokesperson commented “If you look at our history and our competitors, I think we've all had at least one firmware upgrade in the life of each product.” I'm sure that's true; I have a collection of similar stories in all segments of the embedded world.
There's nothing new here. I tried to explain this to my dad last week when he complained that pictures he pastes into documents appear on the monitor with lines through them. His video drivers were originals, and we know that darn near every PC shipped goes with buggy drivers. He was appalled that the computer industry ships flawed products and treat customers so shabbily. I've gotten used to these annoying upgrades and haven't thought about them in years. But he's right.
In the pre-flash days firmware was all but impossible to upgrade. Fewer products suffered from customer-discovered bugs. Did the horrific cost of a field upgrade make developers more careful in their tests? Are time-to-market pressures pushing products out the door before their time? Or is it possible that our products are so complex now, with so much code, that bugs are inevitable?
I worry that flash might be an enabler. After all, when it's “easy” to do an upgrade, the temptation to ship early is powerful. Only a very strong team, with enlightened support from management, can resist the urge to ship prematurely.
I see a parallel with the auto industry of the '70s. Then, Detroit expected dealers and customers to resolve long lists of problems. The Japanese came to America and showed the error of those ways and almost destroyed the domestic car industry. Today PC users — and, it seems, more and more embedded systems consumers — are also expected to work their way through punch-lists of defects.
In the '70s American automakers and consumers thought there was no practical solution to the problem of shoddy cars. Consumers were resigned to a series of dealer visits during the first few months of new ownership. If we assume today that big systems are inherently buggy, will some brilliant upstart prove us wrong? America is the world's largest exporter of software. Is that hegemony as tenuous as we found Detroit's to be?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at . His website is .
The problem is not just that firmware is too easy to change – just look at this Nikon product. It has several failure modes – many of which are very easily entered – that will render the camera completely useless. How did a product with these deficiencies ever get out the door?
These problems are not just due to firmware – I would bet that no one outside their engineering group or service group ever thought of the inability to reload the firmware in the field as a problem – why should they? Everyone who worked on this design can probably reload the camera with a cable and utility in a few seconds, so why should they care.
Getting back to the basic question, while firmware is a contributor, the real problem is that we dont think before we code. We can code, compile, link, download and test so quickly now, that the basic process of analyzing a problem is quickly being lost.
When was the last time you sat down with a design problem and spent 20 minutes listing alternative solutions and approaches? What problem is not worth 20 minutes of thought? Isn't that what our jobs are supposed to be? Most of us make good money, and we are paid to think, yet the first reaction to just about any problem is how fast can we code something.
I do think we rely too much on firmware to bail us out of trouble – whether its during development, or after the product ships. No product will be immune from bugs as long as humans do the design and coding of software. And you can't test for all conditions – no product would ever ship. You can only test the most critical things. After that you have to rely on the experience, knowledge and the design of the software to prevent problems from appearing later.
My job entails interacting with EE and ME's in the company. In the past few years, these disciplines have seen a tremendous increase in the ability to rapidly prototype both electrical and mechanical assemblies. And they are both starting to suffer similar failures as we have seen in software.
We design mechanical parts using high end CAD tools, and design electronics and test them using built in SPICE tools, and when we build real units, we expect them to work because they worked in the PC tool we used. Engineers are beginning to lose the skepticism when evaluating their work and those of their peers.
This ability to quick turn changes contributes mightily to this problem. The software industry's ability to make compilers generate masses of code in a few seconds makes 'shoot, shoot, look' easier than 'look, shoot, look'. But it sure won't guarantee that we hit the target.
Director, Applications Software Development
Global Payment Technologies, Inc.
I am currently involved with the design of a complete product line for home automation devices. The concept is great, but the time to market is so short that I fear some defects as shown in this article.
Electrical Design Engineer
The Watt Stopper