Firmware MaintenanceAccepted wisdom is that most of the cost of software lies in its maintenance. As far back as the 70s various researches pegged it at 40-80% of the total lifecycle cost, with most of the papers leaning towards the higher numbers.
Nothing seems to have changed; Bertrand Meyer (in Objected Oriented Software Construction) claimed that in 1997 70% of software costs belong in the maintenance phase.
But is this true for firmware?
To my knowledge there are no studies on the subject; indeed, embedded systems are the stealth segment of software which simply doesn't garner the research applied to IT projects. So all we can do is speculate. But if that astonishing 70% number did apply to our projects, the cost implications are staggering.
Embedded code is inherently different than that living in the world of a PC. Despite 'net-based reflashing that some - not many - products support, for most firmware a software upgrade is very costly. The car must come back to the dealer, or an instrument has to be returned to a repair depot. One could argue that these costs change the nature of the code; that the expense of doing an upgrade is such that firmware engineers inherently build better code.
In fact, that's my definition of an embedded system: a computer-based application with a very high quality bar. If the PC crashes it's not a big deal, but having to stop every 50 miles to reset the car computer is completely unacceptable.
Meyer quotes a study by Lientz and Swanson finds that 42% of maintenance costs stem from changes in user requirements. My sense is that there's little of that in the embedded space. MP3 players neither get recalled for bug fixes, nor are a stream of continuing software releases that improve functionality available. Instead, vendors launch a new version of the product, one that has a raft of newer, cooler features. The previous version's hardware and software is obsoleted.
That's very different than the PC, where on might use versions Word for years, contributing to Microsoft's bottom line via a never-ending series of upgrades rather than replacements.
Perhaps in the embedded world maintenance do to changing requirements is really expensed as new product development.
Lientz and Swanson claim 18% of maintenance costs come from changes in data formats. That smells like a problem in the info-processing world than in firmware. 21% are due to bug fixes, a plague shared by all programmers, and 6% from hardware changes. One would think that is entirely in the embedded domain.
Though I do see embedded products that have been in service and continually maintained for decades, most of our products don't resemble those gigantic business apps that have kept legions of aging COBOL programmers employed.
What do you think? How much of your budget goes to ongoing maintenance?
Jack G. Ganssle is a lecturer and consultant on embedded
development issues. He conducts seminars on embedded systems and helps
companies with their embedded challenges. Contact him at email@example.com. His website is www.ganssle.com.
It is a fact that not enough research is being done on the costs of firmware maintenance. I think it would be very difficult to gather any maintenance stats unless there are upgrades to the products without hardware changes (consider the case where the processor and hardware architecture changes in case of mp3 players).
In other cases where upgrades are just made to processor (using a faster CPU), maintenance costs should be very significantly looked into.
Another aspect I would like to point out is about the maintenance cost of firmware when applications are being developed and tested over the embedded system that is being made. This cost maybe during development, but it covers at least 75% of the development lifecycle after the firmware is released (this is purely my personal understanding and it could differ from product to product). Saving over this cost in firmware could be a benefit for early release of the products.
- Saravanan T S
I think that firmware is not of better quality than desktop software as you postulate, but i think that you learn to live with it. On less critical systems, like MP3 players, if it resets when you do some sequence of actions then you will learn not to do it again. Of course, this is not the case with more critical systems, like automotive for example. The firmware in a car is of better quality than most of desktop software. On the other hand in desktop applications you find it the same. Finding a bug in a download manager or word processing tool is very common. Though if you get to more critical desktop application like gigantic apps you mention or may be a flight tracking system, you will find that most of the bugs are cosmetic. Nevertheless you always find bug fixes available -even- to these cosmetic defects, why? Because it's easy to fix it, just run the patch and here you go. My point is, there is no difference between firmware and software in quality, but the difference is in the criticality of the system. A more critical system (either embedded or desktop) means more quality. Another point is that the ease of fixing a bug in a desktop application enables people to report it and get a fix for it, but in an embedded system if they can live with it then they will not report it. When I say "if they can live with it" I mean it. Bugs are not fixed in an embedded system unless it is life threatening or consuming so much money if not fixed.
- Shereef Ibrahim
I think you make some valid points about the lack of software maintenance in most embedded products. I think you're right about consumer products such as MP3 players but not all embedded products fit this mold. I worked for an automated lighting company where I was maintaining the code for the operator's console. Everytime we released a new light, we had to update the console software. I was maintaining code that was 5 years old when I started maintaining it. I won't even go into some of the systems I've seen in the military arena.
- Douglas Lewis
I think we need clear definition of the word "maintenance" as it applies to embedded systems. If it includes new requirements and features after the initial release, then I think I think maintenance costs would be fairly high for embedded systems. If maintenance refers exclusively to fixing bugs, then I would expect maintenance costs would be extremely low for embedded products.
- Richard Rogers
If you look at the case of the new MP3 player from the perspective of the programmer, it seems likely that each new MP3 player constitutes "maintenance" of the existing code base, ranging from installing a new board support package, to porting the code to an entirely new toolset....
- Steve Schilz
How about doing embedded firmware maintenance on a product that is several million miles away? This happens sometimes with space probes and such. A good example of this is the Mars Rover. From what I recall, there were several bug fixes that were patched in while in transit to Mars.
However, in general, most directors and execs frown upon field upgrades. However, most people do insist on field upgradability. From what I have seen, most of the cracker-jack-style products seldom, if ever, get patches and bugfixes. Large network and shelf controllers, that sport fancy operating systems usually will get many many patches and fixes. In fact, most networking companies offer service contracts for just such a thing.
- Ken Wada
Automotive firmware becomes more and more complex and only a few projects can get new software from scratch. Maintenance has become an important cost factor and only lately the maintenability of embedded software became a core requirement. While management likes the idea of reuse, in practice it often is easier to (re)generate software from scratch than maintain an old code base, especially if documentation is poor and personel change is high.
- Bernhard Kockoth
Here is some "factual evidence" that pertains to the current survey about embedded maintenance costs. I though you might find it interesting.
Background: My company builds electric forklift trucks. We typically have 20K " 50K lines of C code per processor, and typically 1 to 3 such processors per truck (plus sometimes a couple of smaller processors). There are about 6 main product lines, and the products have a long life cycle. The working life of a truck is typically 7 " 10 years. Last week we had an inquiry for replacement EPROMs for a 20-year old product.
For more than 10 years, we have had a staff of 6 software engineers on the development side, and 3 software engineers on the health/maintenance side. There has been some minor fluctuation in these numbers over time, but not much. Our engineers tend to have long employment here " the average is about 12 years.
The two groups actually have different reporting chains, so it is fairly easy to keep them separate. There is some cross-over (both ways) between the two groups doing the other's type of work, but it is fairly minimal and probably roughly balances out. Typically, a product is migrated from development to health/maintenance about 6 months after first customer ship.
So based on a simple high-level analysis based on man-power ratio, this says that 33% of the embedded software cost is post-release.
Actually, I believe that the maintenance costs are slightly higher, since there is more involvement from the company's infrastructure (test, field service, release procedures, etc) for health than there is for development. But this may be a wash, since the development side typically has more field test effort than does product health.
- Dave Kellogg
The costs of developing and maintaining software (including firmware) are covered expertly in Professor Barry Boehm's book. A definition of what constitutes software maintenance (corrective, adaptive, perfective) is given expertly in Professor Ian Sommerville's book. Happy reading.
- Martin Allen