I came across two engineers vehemently arguing about buying a commercial RTOS. “It’s too big,” one complained, “we’re already short on flash.”
Engineering isn’t about achieving some sort of technology perfection. And it’s only consequentially about building products.
Engineering is a business proposition. We’re paid to help the company generate profits and increase shareholder value. So a wise developer uses that context to evaluate every decision he makes.
Can’t squeeze some commercial product, say an RTOS or protocol stack, into memory? Don’t start salivating at the chance of writing that code yourself (“I’ve always wanted to write an OS!”). Take your hands off the keyboard and consider the business impact of buying a smaller product from a different vendor… or adding more flash.
We deal with two kinds of costs. Bits and pieces, ICs, PCBs and labor form the cost of goods sold (COGS). Additional memory will indeed inflate the COGS and thus increase the product’s price.
But NRE (non-recurring engineering) is every bit as significant as COGS. NRE is us: our salary, desks, scopes and tools. It’s incurred once, hopefully, during product development.
There’s a tradeoff between NRE and COGS. If you’re building an electronic greeting card even $0.0001 in recurring cost is hugely important. It’s wise to invest lots of engineering to shave every microcent. But other products require different tradeoffs.
Firmware is the most expensive thing in the universe, with most companies spending some $15 to $30 per line of code. The space shuttle’s software runs about $1000/line. Jean LaBrosse’s very popular uC/OS comprises about 6000 lines of C, representing an effort worth well over $100k… assuming average practices. I imagine the cost of this OS to be far more, since it has been used in products certified to DO-178B level A. Jean tells me thousands of pages of documentation were generated to meet that onerous safety-critical standard.
But even at $100k, if you’re building a thousand widgets you’ll save the company serious dough by adding ten bucks worth of memory, rather than rolling your own OS.
A few years ago an engineer emailed me that they were coding their own Internet stack. Management had choked on the $15,000 cost of the commercial package they’d considered. I called the vendor, who confirmed the package had over 100,000 lines of code.
That’s 15 cents per line. A Blue Light Special.
The right business decision would have been to present the vendor with a $15,000 check and a huge grin. That purchase is so lopsided in the customer’s favor it’s akin to robbery.
What do you think? Does your company make reasonable business tradeoffs between NRE and COGS?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at . His website is .
Engineers who want to develop their own O/S, protocol stack, graphics library, CASE tool or design methodology should be encouraged to do so. On their own time!Furthermore, they should be packed off to Tierra Del Fuego where they can get peace to work on their egocentric creations.
– Martin Allen
Mr. Martin Allen has a point about engineers developing their own products, or re-inventing the wheel, on their own time. I disagree with them being shipped to Tierra Del Fuego, however. Many of today's products started off as an engineer's hobby work in his basement. How did uC/OS get started??
– William Nichols
You need to remember who really runs most companies: accountants. Accountants understand real stuff: resistors, PWBs, ICs, desks, computers, keyboards, brooms. The penny-counters always want to reduce those “inventory” costs. But NRE isn't inventory. They really don't care about trade-off between NRE and inventory.
Penny-wise, pound-foolish is an apt description of how most financial departments of a company work. After all, they're already paying you; for you to have to work harder to save on inventory costs only makes sense. Remember, your salary has already been put in the budget and is (hopefully) non-changable. Their variable is inventory.
If you go over your NRE budget, it's almost expected. If you go over your material budget–watch out!
Of course, when you start to through in IP, it really upsets them. “What do you mean we paid $15,000 and don't get to OWN anything?” is what you hear when you try to buy an RTOS, protocol stack, or other IP license.
– John Patrick
Let us not forget another oft-overlooked metric. This one is well-known by those supply-chain people as the Time-To-Volume.
A good design takes into consideration all of the factors, (COGS, NRE, Time-To-Market, and Time-To-Volume).
The best designs are where the organization works collectively as a team to help spell out the tradeoffs between the four issues mentioned above.
>From what I have seen, the manufacturing group very much appreciates anything we embedded folks can do to contribute to
– Ken Wada
>>> But NRE (non-recurring engineering) is every bit as significant as COGS. NRE is us: our salary, desks, scopes and tools. It’s incurred once, hopefully, during product development.
As you know in practice it never is. The all important product support. I can’t think of any case where that isn’t a NRE to be considered.
>>> Can’t squeeze some commercial product, say an RTOS or protocol stack, into memory? Don’t start salivating at the chance of writing that code yourself (“I’ve always wanted to write an OS!”). Take your hands off the keyboard and consider the business impact of buying a smaller product from a different vendor… or adding more flash.
Most of this is scalable and broken down into components. I believe QNX was the first company to recognize this, and advertised their products that way. I would never buy anything from a vendor that was “one large lump of code”.
>>> A few years ago an engineer emailed me that they were coding their own Internet stack. Management had choked on the $15,000 cost of the commercial package they’d considered. I called the vendor, who confirmed the package had over 100,000 lines of code. That’s 15 cents per line. A Blue Light Special. The right business decision would have been to present the vendor with a $15,000 check and a huge grin. That purchase is so lopsided in the customer’s favor it’s akin to robbery.
A simplified case, but then there is always more control when you have written the code. This can be an issue, if you plan on expanding your product line, or reusing code. Some vendors are better at addressing bugs found by customers than others. Basically you need to think in terms of components and consider how likely you’re going to need to change that chunk of $15,000 code. The money is wasted, once you discover it’s not going to meet your needs too soon in the product cycle and you can’t get proper support. OTOH, it’s silly to reinvent the wheel, and the $15,000 code is a bargain if it’s a low-level solid component whose requirements aren’t likely to change at all. Of course I’m assuming this $15,000 code doesn’t come with licensing restraints. What if you have to buy it 10 times? You’re now up to $150K.
The whole open-source issue from a business perspective is a totally new subject onto itself. Perhaps you might want to write an article in that context because the business decision whether to go with a proprietary vendor or spend more time on an open source platform is a common one.
– “Tiger” Joe Sallmen
Licensing an OS from a vendor in many cases is fine.
There are also many chips for which no viable commercial RTOS is available.
There are also still MANY situations where COTS doesn't have a solution that fits my needs – either the product is bloated, overly complicated, or wasteful of other constrained resources.
In these cases, rolling your own is not only necessary, it just plain old makes sense.
All the good RTOS's have yet to be written.
– Andy Kunz
While your comments make sense on the surface, all businesses are not alike.
My company, for instance, cannot evaluate ROI. They don't even have a form. You cannot make a business proposition without explaining what the payback is for the dollars spent.
When a company makes 40 million product items a year, reusing the same code over and over, development costs shrink to niggling amounts.
The example you cite of 'make vs buy' makes perfect sense if the budget for custom software has already been determined, while the budget for manpower is managed separately so that a comparison cannot be made. Or, as sometimes happens, manpower is available at the time, and is not doing anything else productive whilst between projects.
Business world – is a goofy world, is it not?
– George Bennett
Update from Jack Steve Leibson's blog at socdesign.blogspot.com has a fascinating quote from Leslie Berlin’s “The Man Behind the Microchip: Robert Noyce and the Invention of Silicon Valley.” It's about the start of the IC age, and may provide some more insight into how we sometimes make the “build vs buy” decision:
Many engineers, designers, and purchasing agents working for Fairchild’s customers feared that integrated circuits would put them out of work. For decades, these customers had designed the circuits they needed from off-the-shelf transistors [and vacuum tubes before transistors], resistors, and capacitors that they bought from manufacturers like Fairchild. Now Noyce wanted to move the Fairchild integrated circuit team into designing and building standard circuits that would be sold to customers as a fait accompli. If the integrated circuit manufacturers designed and built the circuits themselves, what would the engineers at the customer companies do? Moreover, why would a design engineer with a quarter century’s experience want to buy a circuits designed by [a] 30-year-old employee of a semiconductor manufacturing firm?
Interested reader can [re]read Jim Turley's article “Embedded systems survey: Operating systems up for grabs” (5/24/05 articleID=163700590, on this website).His survey covers twenty-five commercial RTOSes. Twenty-five!
And as a reader pointed out, that survey was not all-inclusive. Surprisingly enough, 17% of RTOSes are developed internally.
There is still somebody “re-inventing a wheel”? Why is that?
– Roger Lynx