Embedded Systems Programming Editor in Chief Michael Barr recently forwarded me an intriguing letter from a consultant who had attempted to build a business helping companies incorporate Linux into their embedded systems. The effort was a complete failure; a year of work had turned up only two companies actually using Linux in the embedded arena, so the consultant returned to traditional real-time operating systems where there's plenty of work and lots of money.
What gives? If you believe the magazines, it appears anyone not using Linux is a dinosaur, an atavistic throwback doomed to write miserable little 8051 assembly language programs forever. At the local bookstore an entire shelf of computer magazine headlines scream “Linux Forever!” All tout this OS as the solution to everything that ails us, from hackers to global warming.
Unfortunately, Linux is an unassailable icon. Those who cast aspersions on it are immediately branded as wrong-thinking heretics. Yet it's just an operating system. That's it – a tool, something that allows us to build real systems. In the embedded world Linux is even less than an OS; it can't even manage real time events — at least not in the native distribution. Unless Linux is bolted on top of a traditional embedded RTOS, your system might have a killer file system but be unable to handle asynchronous events in a timely manner.
In my travels around the embedded world I see a huge number of developers working with all sorts of real time operating systems, from very simple home-brew taskers to complete top-of-the-line commercial products with protocol stacks and a wealth of features. But Linux seems to be a phantom OS. Plenty of folks have adopted it as a development platform, but only rarely do I see it incorporated into a product. Sure, there are some notable exceptions. Some set-top boxes, PDAs, and other products are being shipped today with Linux kernels. But Linux-based seem to be in the minority, with traditional RTOSes still, by far, dominating the OS space.
Much as politicians are led by their polls, publishers use extensive surveys to see where the market is heading and what sort of editorial content best addresses readers' needs. But surveys are notoriously fickle. The 2000 ESP subscriber study showed that 12% of the 525 respondents used Linux in their 16- (!) or 32-bit systems in the last year. Thirty-eight percent are considering it. With almost 100,000 embedded design starts each year, that suggests a huge market, enough to make any embedded Linux provider salivate and venture capitalists feed the frenzy with wild abandon.
But almost a year has elapsed since that study, and there's no way, based on my imprecise observations, that 30 to 40,000 projects were done this year with this OS. Lineo announced a layoff in September. One of MontaVista's white papers discusses 60-microsecond interrupt latencies on an 800MHz Pentium. Yikes! Is that CPU really indicative of the embedded world? I doubt it.
Linux is a big beast that needs a lot of memory and processor cycles. It does bring some pretty substantial benefits to some systems: communications stacks, a fabulous file system, and Unix-like interactivity and reliability. But is this a substantial part of the embedded space?
What role do you think Linux will play in embedded systems development in the near future?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at . His website is .
I totally disagree. First of all, the very first question we must answer is this: what is embedded? It isn't what it used to be 10-15 years ago. In the past, there was no way you could even talk about an OS with the 8051 or any other processor, because there would be scarcely any memory for the application on a target yet alone for the OS. Now, things are changing and from where I am standing, the line that separates “embedded” and other types of systems is quickly fading away. So, it is no longer a constraint to require more power from hardware, in fact it is a necessity!
My second point is the level of support you would be receiving for any commercial RTOS. Most of the time, you have to deal with the marketing people who don't really know what they are talking about, and whatever you ask, they will answer as they have been taught: a big yes! Wish things were so bright! But when the thing starts breaking down, they will not be there for help and it would take infinitely long for you to get to the real people who know their thing. This is not the case for Linux. You will get help from the user community without any cost, and people will share their code with you. Just for the fun of it! If you want to change things, you are absolutely free to do so. With a commercial OS, would the writer care to give an example on any of these advantages? From my point of view, it's only about one thing: making things better and Linux leads the way on that!
I find it amazing that someone could spend a year searching and only turn up 2 companies doing embedded Linux work. My company doesn't cater to the embedded Linux niche, and over the same time period we've gotten two clients locally in a town of 100,000 that are doing embedded Linux work. In both cases, they came to us…
I obviously can't critique his work-finding plans, since no details were given. It just seems odd.
Politics and perception — from what I have seen, these are the two biggest reasons that Linux is not more frequently being adopted into target embedded systems.
First, politics: I think that you will find many company's will generally be in favor of using a robust, yet free operating system as opposed to paying money for one, hence the high polling numbers. In contrast a high interest from individuals being polled, I think you will find that corporate support will slack off considerably when it means that a corporation will need to come out and support an operating system in direct competition with, say, Wind River. What Company really wants to damage their relationship with their current OS vendor? The company I just left has, in the past two years gone through not one but two Linux vendors and claims to support Linux on their target systems. However, never have they shipped a Linux based product, and in that two years, I believe I have seen 1 small press release on the subject announcing a limited partnership. Not surprisingly, the have a long-standing partnership with other OS vendors who have no interest in putting up with competition.
Second, perception: Linus Torvalds wrote his first post regarding Linux to comp.os.minix on August 25, 1991, so to most of the senior engineers who are making the “what OS do we use” decision, Linux is a baby. These people have cut their teeth on VxWorks, Nucleus, etc. The marketing team for these more established OSes have marketing down to a science. Talk to a senior engineer at your company and ask him if he or she would want to use Linux in the next product you put out. I think you will find that these more senior people are interested in Linux, but are also frightened that Linux will not have the same features or abilities that your more well known OSes do. Its most likely an incorrect belief, but nevertheless, Linux is still new and strange to the men and women making the big decisions. And the Marketing groups from the companies that charge 50K for a yearlong OS license aren't clearing anything up for them.
That all being said, in conclusion: Politics and Perception have one shining attribute that squarely comes down in favor of Linux: they change 10-15 years from now, when you, I, and the other new kids on the block are the ones figuring out what exactly this product should look like, read this article again. My money says you'll wonder why anyone ever paid for an OS.
I am currently working on a Linux based embedded project. It started in March '01. Originally it was to be the development and support system for the deployed LynxOS systems. There were so many issues with LynxOS, that we are looking to replace the fielded systems with Linux systems.
Already the customer has proposed a low cost gateway system using Linux. The LynxOS just costs too much.
There have been issues that we have run into using LynxOS, that we could have probably resolved using the source code, but it wasn't available. Waiting for the folks From Lynx to get the fix take care of is very time consuming. I had an issue with the semaphores in Linux, a couple web searches, and I found the solution, a small patch. Sure I gotta compile it in, but at least I have an answer today!
Linux will happen.
Every time I read an article that questions the future of Linux I ask myself what ax the author REALLY has to grind. In this case, I'd say this author is scared of Linux. Scared of seeing his world change. Scared of having to learn new ways of doing things.
Why do I think Linux is the wave of the future? It's the first universal operating system — the Swiss army knife of operating systems. It runs on more different computer architectures than any other operating system. It has the largest, most knowledgeable set of enthusiasts and professionals who are continually updating and improving it and is continually gaining more and more followers.
The rate at which it improves is almost frightening. Things that it couldn't do a year ago, it does now with ease. Every time a flaw or limitation is bruited about in the computer press, six months later, that flaw or limitation is removed.
Will it dominate embedded computing? Not immediately. But it will dominate those areas of embedded computing for which it is suited, and those areas for which it is suited will grow with time.
There are large areas of embedded computing which do not require a real time system response, and those companies competing in those areas currently using other OSes will find themselves competing with other companies which WILL use Linux and which will benefit from the minimal licensing costs and the ability to modify the source code at will.
In time, everybody who can use Open Source will, because to not use it will give the competition an edge.
Because Linux is much more than an OS, it's a way of doing business. Every distribution comes with much more than an OS, it has languages, databases, GUI software, development tools, editors, desktop software, networking software, office suites, file systems, image editors, browsers and every other type of software imaginable- all of which can be customized, edited, or torn apart and put together in new and useful ways. You can even, if your heart desires, BUY software to work with it.
Any way you cut it, it's one hell of a deal.
Embedded Linux is an interesting phenomenon. Where is it going to go? I think it is still too early to tell. Too many factors (such as the dotcom bubble bursting and a recession) are skewing things. I think the there are too few points in history, too close together, with too much noise to predict a future.
Look at, say, ARM and small footprint Linices. Things have changed so much in the last 6 months, improving things greatly. New GUIs (no more fat X), better driver support, better stability, etc. It will be interesting to track the new Sharp Linux PDA and see if that takes off.
The hardest parts for Embedded Linux right now are I believe:
- The dotcom bubble burst has made people get all conservative. Remember the saying “Nobody ever got fired for buying an IBM”? Corps want to play with conservative, “solid” suppliers – not exactly the Linux character. Significant support from IBM etc will probably start to sway them in the future.
- “Linux is free”. People feel they're getting screwed when they pay for Linux development services – even if it is a good deal. I've done WinCE and Linux consulting and people definitely feel better pulling out a wad of dollars for WinCE than a small amount for Linux. Linus=fun, Bill=dollars. This is making it hard for embedded consultants to get going.
How is Linux consulting biz stacking up against other RTOSes? Dunno yet. Do Wind River dev shops have it easy? I'm not sure. I do know that WinCE shops are struggling: BSQUARE laid off a bunch of staff. Eclipse (probably the best at one stage) went belly up. I don't know any WinCE shops that are thriving.
I have a hunch though that a lot of embedded Linux is in-house using SBCs with off-the-shelf distros or the distro that came with the SBC or off the www. This is so very easy to do. Don't need a crate load of gurus. Makes the biz hard for gurus.
I think some of your tech assertions are a bit off: * Size: You can make a useful Linux way smaller than a useful WinCE, though not as small as a VxWorks system. That's using standard Linux. uLinux should even do better. * Latency: I have only done limited latency measurements on a StrongARM. Didn't lose any interrupts at 50uS. While that's not great compared with an RTOS it beats WinCE. 50uS is fine for a lot of soft RT.
Things I like about Linux:
- Source code is available. Can see what's going on. Can't in most proprietary OSes –especially WinCE.
- Freedom to use the stuff as you want to. WinCE imposes many restrictions apart just the licensing cost. Unless you sell 25,000 per annum you're unlikely to be able to get some of the software (eg., pocket apps).
- Apart from platform dependant code, the same drivers/kernel that run on an ARM7 also run on an IBM mainframe. This makes for well-tested code.
- The shear energy and excitement. Many people program Windows to feed the body and Linux to feed the soul. They care far more about what they put into Linux.
- Linux isn't helping the Evil Empire. I dislike what M$ is doing. Linux feels morally right.
As system complexity increases and HW costs decrease I think there's more reason to shift towards Linux (or less reasons not to). 32MB of SDRAM costs less than $5. ARM core is only a few bucks. Lots of cool stuff is just sitting there to be used. The main motivation to use an RTOS is not having enough RAM & CPU [hey remember when that was the main reason to use a roll-your-own and not use an RTOS!].
Anyway for now, I'm getting into Linux. I expect it will be a while still before the embedded side really takes off.
I just completed a project using Linux on a PC-104 board. This project is used to monitor seismic activity and look for potential underground nuclear tests.
The Linux part of it assists with validating and digitally signing the data stream before it is sent to international monitoring agencies.
This is embedded, and mission-critical.
David F. Skoll
You have a point for deeply embedded apps. If you need a system for $2.50 or you need hard real time, Linux isn't it. But a lot of modern apps require Ethernet and windowing and big memory and horsepower. I think a better term for where Linux is going is appliances. Video, networking, web based, things that are a snap on PCs but people want mobility. And, it's not the traditional embedded crowd that's doing it because they are still doing deep embedded stuff with tiny resources.
For example, take the project I founded. Yes, you can build a PLC without Linux, obviously. But PLCs today are doing Ethernet, Fieldbus, web access, database access, etc. Bolting all that onto a low resource start, you quickly get to the point where things would be much easier if you started with something where those aren't add-ons. The development is much, much, easier with Linux tools than an assembler. And it's royalty free. Linux is well known and not at all esoteric. With Linux, almost any good programmer can do embedded. The SOC makes a lot of this possible. You're looking at the old class. This is a whole new class of net things and digital video and the next big thing. A lot of these will fail as people learn something about embedded and not every brilliant idea is going to pan out, but I think Linux will be ubiquitous in the “post PC” age.
Curt Wuollet — who will hang on to his Linux PC for dear life
Heartland Engineering Co.
Jack replies: Agreed that Linux is just a tool that's appropriate for some apps and not others. And your comment about the Post-PC age is probably right on. At the moment we're grasping in the dark to see just what “Post-PC” will mean. It's a fun time to be in the business.
Embedded OSes are not a “one size fits all” proposition. In certain cases Linux makes sense from both the business and engineering perspectives. It's well tested, inexpensive and offers a wealth of services. In these days of ever declining memory prices, if Linux can meet your real-time requirements, it certainly makes sense to evaluate it for the project at hand.
From my perspective, it's certainly better to have an OS littered with features, some of which I may not need, than it is to have an OS which has few features, a few poorly tested extension modules (which generally cost extra) such as TCP/IP or a disk file system. With the cost of suitable hardware ever declining, I think Linux will increasingly be the OS of choice in the embedded market.
Integrity Data Systems, Inc.
I have read Jack Ganssle's articles in Embedded Systems Programming over the past few years. For most part, I find that his comments are well founded. However, there are times when using a crystal ball is not appropriate. The only way to see the future is to look at what's happening now and hope that it shows where we are headed. All the time remembering that any road will have curves somewhere.
From Jack's articles, you get the feeling that all embedded systems must be “hard” real-time. With a few years under my belt, I have yet to see a system where “soft” real-time didn't amply meet the requirements. We are currently shipping a product that was designed over two years ago using Linux. It's not really an embedded system in the normal sense of a micro on a board, but it's an embedded PC in the sense that the users are not aware of its presence. Isn't that where MS is going with WinCE and embedded XP? In this case, Linux was chosen because of the lower cost, no licensing hassles AND the rich set of software available. The final product had more flexibility than we had hoped for and has been enhanced to meet continually changing needs. (What more could you ask for?) We never regretted the choice.
We are currently in the prototype stage of a new product. Linux, uClinux specifically, is the choice because it runs on the Coldfire (wanted for its low power and rich set of peripherals), has a small footprint, no licensing fees/hassles, etc. In this specific case, a full-featured IP stack drove the choice. This allowed us to eliminate a hefty IP stack license fee, which was required for its predecessor. To indicate our level of success, the board was first powered up just two days ago and today it's running Linux!
We are a custom electronics design and manufacturing firm. As such, we are very much driven by our customer's desires, we have worked with VxWorks, WinCE, DOS, Windows 3.1, Win95/98, Win2000, QNX and many proprietary OSes. With the maturing of uClinux, I see Linux as becoming a very dominant contender in the embedded marketplace and do not hesitate in recommending it for various designs. At the same time, I am ready to admit that there are situations where Linux is not my first choice.
I think to ask if embedded Linux is a bust is very narrow minded. Success is not measured by the number of publicity banners that shout “runs Linux.” Success is measured by the number of teams that have quietly put it to use where it runs reliably and serves the purpose well. As more people implement such systems, the confidence level will rise and rather than being a wave of success that busts in a froth of foam on the beach. It will be a ground swell of a sure thing that's here to stay.
David J. Pfaltzgraff
Jack replies: Dave, you make an important point in that most of us work very quietly, with little publicity. It's hard to really know who's doing what.
Since I make my living from the Linux embedded world I would have to disagree that it does not have its advantages vs. traditional embedded environments. It has significant advantages over traditional systems in certain areas.
However I do agree that there are probably not as many embedded Linux projects as some would have you believe. The problem is a lack of skilled people to use in such a project. The amount of skill required to actually make modifications to a kernel and all the resulting infrastructure is quite great, and most programmers simply do not meet the muster, as it were. Since Linux itself and all the components around the kernel were not designed specifically to be used in an embedded environment, they do require modification for most projects.
The infrastructure for “Linux embedded” is emerging, and has been emerging for quite some time. However, it is an operating system, not some sort of dream answer like “42.” It works for many project types, but it does not and will not work for everything. The OS itself is too large to fit inside of some device constraints, much less all the applications and the OS.
Since it can't work on every project, should it be regarded as worthless? No, clearly not, and that is why I think your article is biased. However, I understand the cynicism since many people touted Linux as the dream answer. As a person who is working on leading my second Linux development project, I can say with 100% certainty that there are good reasons to use Linux, and there are good reasons to not use it. Any sensible management team would evaluate the issue on a project-by-project basis. What OS meets the needs of this project best? Do we have the requisite in-house skill to use Linux? What will it cost us to gain that skill? And will it pay off in what we would gain in licensing fees we won't have to pay?
There are very few people that are truly understand Linux embedded development. If you are looking to MontaVista, Bluecat or Lineo, you've already shown that you don't understand Linux embedded development. If a company has to depend on one of those providers, then they don't have the inside resources, and they should look for something else to meet their needs, or find the staff that will be able to meet their Linux needs. If they don't have those internal resources, then their project is doomed to fail.
Anyway, as with anything, if you're looking for the Holy Grail, then go to Church or the Mosque; don't go to an operating system.
Chief Kernel Architect
(Former CTO of Agenda Computing)
I don't have a barrow to push on this issue, as I don't program using RTOSes. However it strikes me as dangerous to base a line of thought on a letter by someone who appears not to have done the appropriate market research before trying to build a business.
The Bionic Ear Institute
East Melbourne, Australia
VxWorks was for sale in 1986. WRS went public in 1993. It takes years.
I've had more (free) interactive support from MontaVista and Lineo (both have superb engineers) than I've ever seen from WRS, in the newsgroup or their paid services. To be honest, I got excellent support from OSE, maybe it's just WRS that has poor support?
Magazines which seem to be lining up with WRS can't hold a candle to the technical content in Linux Journal . The bar is so much lower with proprietary OSes like Windows XX and VxWorks.
Jack, how dare you assail the “unassailable”? 🙂
Seriously, write something critical about VxWorks — if you've ever used it. And if your editors dare allow you.
The old context switch and interrupt latency canard is an extremely marginal issue compared to OS platform stability for application development and deployment.
We won't mention the huge base of tools and applications that totally swamp the commercial offerings.
The person in the article must live in Redmond!
At two different companies I work with there are at least six different embedded Linux systems in development or production. These range from radar systems to digital radios to network monitoring equipment. Distributions run from Yellowdog and HardHat to Redhat, and run on everything from Altivec G4 to PowerQuick PPCs to 1U rackable Intel-based to something called uLinux in a soft-core part. The embedded world is not all ABS systems and microwave ovens. Nor is it all hard real-time.
Every one of the projects in the past (some of these are second or third generation products) was previously or would likely have been done with VxWorks, or not done at all.
2 Tigers Consulting
Jack replies: Actually, I live in Baltimore, about as far in the USA from Redmond as it's possible to get. Coincidence? Maybe not.
I plan to use an open source operating system at some point in the future for mid-scale embedded systems use (PC-104), but whether it is FreeBSD or Linux is at this point immaterial.
The larger point that Jack misses here (by half a million miles) is the issue of open standards, open source, and open tool-sets, versus proprietary. Linux is a part of the sea change going on in the industry, but it is not the whole issue.
If anyone hasn't noticed, you can now download the complete QNX (Neutrino) SDK and start using it to prototype and play around with, for free. Do you think that would have happened two years ago? And let's compare with the hype about Embedded NT and Windows CE. If you wanted to print a story about products that launched with lots of hype followed by astounding number of aborted and failed projects that tried to use Windows CE and failed, then you'd have a point.
Any embedded systems engineer who has been stuck dealing with a proprietary real-time OS, when serious problems are discovered after a device is in the field knows that the power of open source solutions might still be a ways away from making practical business sense for them yet, but Linux and open source are not in the least bit over . Heck, it hasn't even really started yet. There are still people using DOS in the embedded systems world, for goodness sakes!
Embedded Systems Developer
ZTR Control Systems
For the record, I can testify from both personal knowledge and hearsay that there is a LOT of embedded Linux stuff going on. Linux is an excellent embedded platform IF your real-time demands are not hard real time, and there are extensions to Linux that do hard real time as well.
I do have a guess about the “phantom OS” facade [the author mentions]. A lot of companies, especially in this field and after the great Linux hype came and went, simply refuse to admit they are using Linux. We had the same thing happen in the server market two years ago and now it's in the embedded space. Companies who use Linux on their products are afraid, for many reasons, to mention [that they are using Linux]. The reasons are many — fear to be thought of as a Linux company, fear to be thought of using “new” technology in a field where reliability is a must, and a lot of misconception about the definition of a real-time OS and real time requirements of products, as opposed to embbeded OS.
Software Team Leader
Kagoor Networks ltd.
Embedded Linux is definitely not a bust, though I don't expect it to dominate embedded space in the near term. It's not the best choice for every embedded project and, like many new things, media hype has created expectations of instant success. The reality is that it'll take time for Linux to become a major embedded force. My guess is that by the second quarter of 2003, the hype will have spent itself and embedded Linux will be an accepted and frequently used tool in the embedded systems toolbox.
Linux is currently suited for a number of high end embedded applications, but it still has to prove itself for others. Real time support is still immature, but is “good enough” for some applications, and not all embedded applications need an RTOS. As more embedded projects use Linux, its weaknesses in this area will be highlighted and addressed. I expect to see multiple approaches to be tried, this will include some that _are_ busts, but others will be successes. Lessons learned from both the failures and successes will evolve Linux into a better embedded OS. Faster and cheaper processors and memory won't hurt either.