Dan O'Dowd, president of Green Hills Software, has been stirring the Linux pot lately. The OSS community is in an uproar over his contention (that Linux is an inherently insecure OS for embedded systems. More interesting, to me at least, was his previous anti-Linux polemic. In “The Myth of the Embedded Linux Tools Market” he wrote, “There is no sustainable embedded Linux tools market.”
Look at the numbers. MontaVista Software, the biggest player in this business, doesn't publish sales figures, but Venture Development estimated their 2002 revenues at $18 million. The company was Silicon Valley's fifth fastest growing company that year, so let's take a guess their sales have doubled, to something like $36m. That's probably not enough money to buy the paperclips used by the Visual C++ developers at Microsoft. Rational (now part of IBM), which basically just sells UML modeling tools, grossed $649m in 2002. Even consistent underperformer Borland shipped $295m of tools in 2003.
But Mr. O'Dowd's observation that there is no embedded Linux tools market is simply a corollary of a bigger rule: there is no embedded tools market.
Wind River dominates this business. They probably own more than half the entire market, yet their 2003 sales were $249m, with a $106m loss. That's a lot of millions to those of us struggling to send kids through college, but it defines a pretty small market space.
Green Hills, Mr. O'Dowd's company, grossed only around $50m, not much more than my estimate of Montavista's performance (though Green Hills consistently runs a profit, unlike most of the Linux vendors). Both companies are tiny. Neither is even publicly held.
Mr. O'Dowd zeroed in on one problem with Linux tools in the embedded space: there are too many distributions. At least six compete for developer's attention, resulting in fragmentation of the market. Linux tool vendors must either support lots of variants (expensive), or address just a minute part of a small market.
But this is exactly the quandary facing the entire embedded tools space: the business is horribly fragmented. Hundreds of CPUs divide a small market into miniscule pieces. An awesome PowerPC compiler appeals only to PowerPC users. It misses customers using Arm, CPU32, Z80, PIC, 8051, 186, 68HC05, ColdFire, Xscale, MCore, SHARC, 68HC11, MIPS, TriCore, MSP430, H8, Z8, AVR, Nios, Xtensa, TMS320, Blackfin, 68HC12, COP8, Rabbit 3000, and a hundred other chips.
Green Hills, for instance, supports 21 different processor families.
Mr. O'Dowd missed a bigger problem with the Linux and non-Linux tool business. Customers simply don't trust vendors. Here are a few samples from emails I've received recently:
- “The company claims 24/7 support but is that marketing hype or is it real?”
- “How do I know this package isn't a bundle of bugs?”
- “We discovered we were the first to use it on this processor. They were sending patches daily!”
- “I worked on a proprietary OS that had a bug. It took them 6 months to release a fix. We got screwed.”
Eighty percent of Linux users hate the tools. So of course they look for cheaper alternatives to those big-buck under-supported commercial varieties.
Most of us old-timers in the industry have been burned by commercial tools. Most of us have had some great successes, as well, but bad experiences loom larger in memory than the good.
This industry has a general mistrust of tools, and even more so of the vendors. That's one reason developers find open source software so compelling. No one wants to actually debug Linux, but users feel that having that option reduces risks. Until vendors find a way to demonstrate their responsiveness and the quality of their products, in a fashion that's so visceral it becomes part of the fabric of people's thinking, the tools market will stay small, and open to alternatives like Linux.
What do you think? How's your relationship with your tool vendors?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at . His website is .
Seems like every project I'm on has to do work around on the tool chain in order to ship product. It's been this way since my 8051 days, and likely will remain that way into the foreseeable future. The first8051 compiler my old boss picked out was horribly in-efficient, and we had to re-code many routines intoassembly to get the project to fit into memory. Then they released the next version which was better in termsof code size efficiency, but changed the register conventions, so this meant editing every assembler routine wehad written. I remember buying a $4,000 PC to work at home in the evenings so I could be with my kids whileputting in the extra hours that this cruddy tool chain caused. We got it all to fit, and work, and hundreds ofthousands of people have been medi-vaced with the system so it was all worth it, but the tools did not make iteasy. I'm now teaching myself about Linux, because the source is all there so you can see how it all works, andyou can ask someone else on the net or at work if you get stuck with something. With a closed system you canliterally be SOL if the vendor goes under, and then comes out of chapter 11 with a whole different compiler. With Open Source, you have the option of bringing someone in to patch your compiler or other tool, or doing thatin house if you have the expertise in house. I do wish that the Open Source community and Microsoft would godown to the FAA building in DC file the freedom of information paperwork and look over the source code for GreenHills Software's Integrity RTOS, and VXWORKS and see how some of their protection mechanisms for memory, andtime domain protection work, and use that insight to make other OS'es more secure and robust and reliable. I'vehad to rebuild an XP machine due to a buffer overflow exploit, and a Linux machine due to a corrupted installerutility this week. (The Linux machine had the smother rebuild, and saved my old data, BTW) (They may have tolicense portions of the technology until the patents expire. Any improvement would be welcome by me)
– Bill Murray
I agree with everything that you say. There is very little money in the embedded tools market eventhough numerous companies have tried to make money out of tools.
Wind River used to have a very clever tools strategy. Their strategy, in the early nineties, was to use otherpeople's tools and thus save on the cost of developing their own. I remeber attending a seminar in the earlynineties where they used the GNU 68K C compiler and the SUN development tools to edit, compile and download codeto a board running VxWorks. All very impressive and all the more so for the way they leveraged what alreadyexisted. The clever bit they did was the code download and integration with SUN's debugger. It looked fantastic.I, and all my colleagues, immediately wanted one; especially since you needed to have a SUN workstation to useit. Back then we all had (very nice) green screen terminals … nowhere near as nice as having your ownworkstation!
In the end we decided to stick with OS-9/68K as its process model suited our development approach and gave usmore flexibility. We even had proper inter-process memory protection!
Of course, WRS grew and grew and grew so it was not long before they came out with their own toolchain. It wasimpressive but not as technologically neast as their earlier approach – let someone else do the hardw work.
And now, WRS have Wind River Workbench 2.0! They have returned to their roots and integrated their best with thecurrent best of breed in IDEs.; in this case, Eclipse.
I fyou have never used Eclipse you don't realise what you are missing. It has fantastic integration with Java(it was designed for Java to begin with) but it has an open architecture which makes it highly customizable. AC/C++ plugin (CDT) exists that follows the UNIX development approach (i.e. edit/make/debug). And it all worksremarkably well.
By giving up on their own editor and all the boilerplate that surrounds a modern IDE, WRS have ensured that theywill deliver a best of bered solution by leveraging what they do well with what others do well.
Now, if only I could have persuaded a former employer to adopt this approach … ah well, c'est la vie!
– Gordon J Milne
The other thing that no one seems to be saying is the fact that if you are worried about security you have tolook at who is working on your software. Yes there could be backdoors, trojans etc. introduced by maliciousindividuals – even at an “American” company like a GreenHills or Wind River. Commercial companies hire the bestand brightest – citizenship is a second concern a lot of times. They may even being doing work 'offshore'. The reality is that a lot of companies don't worry about the loyalties of their Engineers – they are getting thejob done and making a profit for the stockholders.
When you have proprietary code bases which are subject to minimum code/peer review – written by possiblynon-citizens – you have a distinct possibility of malicious code. I'd vote that open source would have a lotbetter chance of being malicious code free versus code written by an engineer who is either foreign based or whohas different loyalties.
– Frank Stepan
Rabbit Semi/Z-World have a non-ansi C compiler with extensive library and examplesfor a very reasonable price. I have found only (1) bug in their code. I have been burned in the past by a toolvendor too.
– Richard Kester
I have followed this topic closely. As a company that transitioned from embedded systems to Linux and back again, I have a great deal of first hand knowledge about this.
I feel many people are taking the issue of tools, profit, and time to market a bit too seriously (bear with me), since these subjects are closest to their daily conscious thoughts.
However, I feel the real issue is that Linux is a threat to the entire embedded space because it suddenly allows Joe Programmer to think he is an embedded systems engineer, and the end products reflect that misjudgment. Developers turn from analyzers and emulators to simulation tools and IDE's with integrated debuggers. Never-mind the quality or availability of those tools, what about the capability? Let's see GDB do timing analysis so we can adjust our rate-monotonic scheduling. Does Joe Programmer even know what rate-monotonic scheduling is?
To say that Linux is unreliable and insecure is an irresponsibly gross understatement. Anyone who disputes this has either not seen the code or is not qualified to judge what they have seen.
I am willing to accept my vending machine occasionally not giving me the correct selection, and I am even willing to watch my set top box mysteriously reset from time to time. But for the love of my family I swear here and now to NEVER climb aboard an airplane controlled by Linux, or drive in a car with anti-locks mechanisms controlled by Linux. Hell, I'm nervous enough that I trust my lively hood to Linux (online banking, etc.), but lively hood is a separate issue altogether than life.
Should Linux gain wide acceptance in the embedded space, I fear that the quality of both engineer (we there still be engineers?) and product will degrade to the state of Linux in the server and desktop world. Anyone who isn't scared by that thought must be considering the primitive life on an island somewhere (I tried that once, but it was Iowa, not an island).
– Eric Uner
I look at embedded tool vendors the same way I look at battery manufacturers: they lie, it's what they do.
My company recently bought two licenses for Protel 99SE. While this is quite an upgrade from what we used before (DCCADD), it is still a far cry from doing what Altium says it will. I don't even bother with the autorouter; a drunken sailor staggering about with a string attached would work as well.
But the thing is, we gotta have these tools. If we pay tens of thousands of dollars for software that is full of bugs and doesn't work as advertised, so what? It's not as if we can go find some shareware solution or do it ourselves (I mean, circuit design on mylar film is one of the reasons FOR the embedded tool industry). We have to have these tools, and the vendors know that.
Alas, when you look at it objectively, it's no worse than any other software. Everything is full of bugs. You can't expect a company that sells 500 copies of a program a year to be competitive with Microsoft. Embedded tool developers have to eat to. What can you do?
Until the embedded software community (primarily management) realizes that software development costs moneyand so do the tools to accoomplish that development, all embeddedd software tools vendors will suffer.
Because engineers and management can see and touch hardware, they think there is value. So they pay and pay BIG for thetools (look at the EDA business $3B+). By the way, HW development tools are not flawless either. Since they can't “see”the software, they think there is no value. Fundamental flaw.
Everyone knows that software development is the most costly portion of building an embedded system. Why don't theyrealize that the tools that go along with it are worth what they pay or more.
My contention is that Linux has cost the embeddded software market money. It gives the opportunity for engineers to dowhat everyone likes to do, play. Too bad we can't calcluate the millions, maybe billions of dollars that companies havespent on engineers wasting their time “playing” with Linux.
– Neil Henderson
In the 80s, I worked on military systems where we were responsible for the entire product, including the”operating system,” and we often developed assemblers and simulators in-house. We even had an ancient Fortran IVcompiler for one system, developed on an ancient Univac and later ported to VAX/VMS, that we maintained. The rest of thecompilers (Jovial/J-73), assemblers, linkers, etc., came from SoftTech and they pretty much sucked. We got new releasesalmost weekly, where they would fix a few bugs, introduce a few new ones, and turn the development environment intoconstant turmoil.
In the 90s, I worked at several different companies with several differentversions of VxWorks, ranging from 5.1 through 5.4. The companies that bought 5.1 and 5.2 source licenses early had agenerally good experience with VxWorks and WindRiver, and generally not much contact with WRS sales or support beyond theinitial sales contract. “Tornado”, or VxWorks 5.3 plus a lot of buggy, ill-conceived GUI tools, was terrible. Everytime we encountered a bug we had to jump through hoops to get the engineers working the problem on the phone with WRStech support and our “contact” engineer, so we could attempt to work the problem. The usual outcome of this was tonegotiate more licenses for something or other in return for getting source code to the part of the system that wasbroken so we could fix their code. I consider WRS to be the epitome of why the “embedded tools” marketplace is goingnowhere.
I've spent the past 4 years now building embedded systems on OpenBSD, Linux, and FreeBSD. I've never needed a “toolsvendor” for any of these, and I don't forsee needing one anytime soon. This is at least partly because I'mbuilding “information appliances”, large systems where you can leave theserver-oriented operating system largely intact. The native gdb debugger, and even the kernel debugger in the case ofFreeBSD, still work. If we need to get onto the box to analyze something, the customer hits a button in their “console”application and we ssh into the box; we can use scp to copy files to and from the system.
My relationship with my embedded tools vendor these days couldn't be better. I developing information appliances on FreeBSD at my “day job.” I'm also a member of the FreeBSD Core Team, and so havequite a bit of say in where my “embedded tools” are going.
I'm fully aware this doesn't cover the entire embedded marketplace. When I have time, I also develop small applicationsusing processors like the AVR and small ARM systems. I do most (approachingall) of that on FreeBSD as well; the embedded developer tools in the open source community are quite good in many areas. If I needed a specific tool, an ICE or something like that, I have Windows machines available, but I prefer to stay inthe programming environment I am most comfortable in.
I think it would be foolish to believe that companies like Green Hills can somehow better serve their customers simplybecause nobody gets to look at their source code. I also think it would be far easier to insert suspicious code into aGreen Hills product than Linux, because in Linux somebody somewhere is going to see what you've done and call you on it. I know for a certainty this would happen in FreeBSD.
– Wes Peters
It could be that the revenues per product/ company are too small to match the markets of tools like MS VC++IDEs etc – having worked with embedded tools for the last several years, a significant part (typically upto 50%) ofprecious development time is wasted in setting up tools that never match their advertized ease of use. Running “out ofthe box” is a virutally nonexistent experience in the embedded development world. Often times it requires constantinteraction with the FAEs and even development engineers of the vendor companies to get a tool running & even get theirhotfixes in line with the development. This applied to practically every embedded tool I had used: compilers, debuggers,IDEs, debug monitors, ICE tools, bus analyzers, embedded OSes, …even Smartmedia used for booting FPGA boards gave medebugging blues for days.
Either this is a huge opportunity for the tool vendors to set things right, or embedded development is a career choice atthe wrong turn, …
– Hari Tadepalli
I think the lack of quality in embedded tools reflects the lack of software quality in the embedded community in general. Is it a surprise that embedded tools are no more reliable than the products they are used to produce?
The percentage of embedded developers I know who use any kind of software engineering practices is alarmingly low. And chronically poor scheduling trumps any attempt to acquire good habits for the future. It's a problem with the culture that will, hopefully, be resolved as the industry matures.
When buying tools, my rule is to *never* be an early adopter. Embedded tools vendors usually make little effort to test what they produce. The customers who buy in the first year are, whether they know it or not, the product test team.
– John Hopson
I don't know why so many people are complaining about their embedded software tools. Either I've been extremely lucky with my tools, or I'm more tolerant than the others.
At my current employer, I've written embedded software for a number of platforms: PIC, 8051, an 80486 SOC, and soon to be a NIOS-II softcore. For the PIC projects, I used Microchip's MPLAB IDE and their assembler. While MPLAB has a new version each month, I find that I can use one version for the length of development, provided I'm not needed features that have been added at a later time. It's true that their IDE and simulator are not 100% bug free (and the simulator does not simulate ALL of the PIC features), but at zero cost to my company, MPLAB works, and works well. I used it to develop a product that we've now shipped over 8000 of, and am developing two new derivatives of this product now.
For the 8051-based systems, I've used Keil compilers and MetaLink ICEs. The Keil compiler is one of the best, and the MetaLink ICE does what we need. The MetaLink IDE isn't flawless, but learning the quirks of any software is part of the job.
The 486 SOC project wasn't perfect, as the company that sold us the debugger closed shop two months after purchase, so there was a bit of a struggle to get things moving. Once that was done, development proceded. This did teach us to research the manufacturers of the tools we buy, and to buy tools that are more “mainstream.”
Remember that embedded tool development and testing costs money, and that the manufacturers need to make up the costs with either price or volume. There's little volume, so it's either more expensive tools or less testing. Given this, is it that embedded software tools “suck” or is it that we wrongly expect that a piece of software that has a limited audience should be 100% bugfree?
– John Patrick