Blind Men, Embedded Platforms, and RTOS Elephants -

Blind Men, Embedded Platforms, and RTOS Elephants


The buzz in the embedded world recently has been about “platformization” — the development of a common software development platform, application programming interfaces, and languages. The concept emerged from two decades of the desktop PC, where platformization was thought to be the key to explosive market growth.

What the platform concept allowed was the shift from a vertically oriented market dominated by one or two companies to a horizontal market, in which dozens of smaller, aggressive companies could enter, take those common building blocks and go up against the giants on roughly the same footing.

Making this shift possible was a common set of tools and building blocks that could be used across a broad range of application segments with not only the OS and IDEs specific to developing code, but also all of the ancillary code packages for protocols, etc.

This sort of platformization usually happens in one of three ways. The first way is for one company to grow faster than the others, become the de facto standard, and through its market presence and muscle impose its view on the market. The second is for a company to develop a specification, but let it be widely used either on a royalty free or at a modestly charged basis.

The third is for the participants in an industry segment to come to a common agreement on an industry-wide standard. That's what is going on the embedded networking and network processor segment, driven by efforts such as Network Processing Forum with its common API proposal and the ForCES initiative.

Several companies with open source Linux OSes, such as Monta Vista Software, which allow complete access to the kernel's source code, and proprietary RTOSes vendors (some of which also allow access) are jockeying for a lead position in accelerating this platformization. In November, both Wind River Systems and Green Hills Software introduced “superbundles” targeted, in part, at the embedded networking space as bid to play a role in this platformization.

Wind River has chosen to create subscription-based “superbundles” aimed as several specific segments, of which network equipment is just one. The others are consumer, industrial, server appliances and military/government applications. Green Hills has chosen to offer a single superbundle to all segments. The biggest difference between the two bundles is that Green Hills offers subscribers complete access to the kernel source code. In the Wind River bundle, while many elements have accessible source code, the kernel is off-limits, as far as I have been able to determine.

While some technical publications and business journals have heralded the superbundles as augurs of a fundamental change in the embedded market, my opinion is that such hyperbole is out of proportion to their ultimate impact. This new way of packaging software tools and building blocks may generate another more sustainable revenue stream for the companies that's a bit less volatile than their current pricing structures allow. But that is about it. No more, no less. No sea changes here.

Engineers in the networking and communications space that I've talked to — admittedly a small sampling of all developers — lead me to believe that RTOS suppliers such as Green Hills, Accelerated, ExpressLogic and QNX, who offer access to the source code of their RTOS kernels, are the ones who will succeed in driving platformization, whether offered in bundles, superbundles,or no bundles whatsoever.

To be fair, Wind River representatives have assured me that they surveyed their entire customer base and virtually no one asked that access to the kernel source code be a part of the superbundles. John Carbone, vice president of marketing at Green Hills is puzzled by this information. “We are in many of the same markets they are, and the people we talk to give us extra points for the fact that we offer access to the Integrity kernel source code. And the same is true for ExpressLogic's ThreadX.”

Moreover, he said, there must be a reason that companies with proprietary closed RTOSes have not been able to make a dent in roughly 40 to 50 percent of the total available embedded control market, despite almost 30 years of trying. According to Carbone, for a variety of reasons most embedded system developers still prefer to write their own kernels, use an open source version, or use a proprietary, supported OS where they have access to the source code for analysis and/or modification.

Why is access to the kernel source code so important in accelerating the trend toward horizontal platformization in the network and communications segment? What I have been told is that because none of the RTOSes available, even the fastest and most deterministic of them, can keep up with the line speed of an NPU at the network core, access to the source code is vital for optimization purposes. While virtually all of the vendors of proprietary RTOS vendors have sold into the space, they are usually confined to the slowest portions of a line card's operations for non-real time, non-deterministic management and supervisory functions.

But the silicon vendors who are building software programmable NPUs are intent on providing customers with a complete and familiar tool suite. They want access to the source code to determine what to put into the silicon, what to leave out and how to structure the hardware so that it does not conflict with the RTOS.

Although network processor vendors and their customers may have licenses to a number of RTOSes for use in various aspects of their design, the one OS they always mention as part of their arsenal is Linux. The reason is not its speed or determinism, or because it is “free,” but because its source code is accessible for modification and analysis.

The conclusion I draw from this is that Linux has a fair chance to gain a significant foothold in this segment, bundled or unbundled. As the bandit from The Treasure of Sierra Madre might say, Linux don't need no stinkin' bundle.

It's Wind River's point of view that these particular users constitute only a small portion of the NPU market, an even smaller portion of the total telecom/networking market, and even less of the overall opportunities in networking, consumer, industrial, and server appliances.

However, Udi Yuhjtman, vice president of operations and business development at Jungo Software Technologies, which focuses on the residential network gateway market, sees things somewhat differently. The company claims 3,000 design wins since 1998 across a broad spectrum of applications. In addition to its own implementation of Linux within its OpenRG framework, it also incorporates interfaces to most of the major proprietary RTOSes.

While he agrees with Wind River that most developers across the broad spectrum of applications don't want access to the source code, the issue is just not that simple. “There is in almost every segment in which we participate about 20 percent who are the most important to the growth of any market,” he said. “And they are the ones who demand access to the kernel's source code.”

There are practical reasons customers prefer to use an RTOS which allows them to access the kernel source code, according to Bill Peisel, CTO at Netsilicon, which licenses ThreadX for use with its complete hardware/software solution,. “And if they did not want it initially, ” he said, “they usually wish they had it when they get the first message from the proprietary RTOS at 10 p.m. Friday night that error #3321 has occurred and have to wait to hear from the OS support team to find out what the error is. They don't want to modify the kernel. They just want to know what the heck is going on.”

It's at times like this that I feel like the reporter who has been sent to interview the six blind men of Indostan who went to investigate the true nature of an elephant and came up with different views based on their “hands on” experience. The result, according to the 19th century American poet John Godfrey Saxe, was that they

Disputed loud and long,
Each in his own opinion
Exceeding stiff and strong,
Though each was partly in the right,
And all were in the wrong!

I think what I need to get a better idea about the nature of this particular elephant is a broader sample. So, write or call me and let me know what you think.

Bernard Cole is the managing editor for embedded design and net-centric computing at EE Times and the editor of iApplianceweb. He welcomes contact. You can reach him at or 928-525-9087.

Reader Feedback:

When an article can motivate someone to respond right after reading it, or even after taking a walk for a second to cool their jets, it must be a good article. To that end, Bernard Cole's article is a winner.

Platformization in the technology space is very much akin to platformization within the automotive space, and shares some of the same advantages. By developing a platform of RTOS, Silicon, and associated dev and debug tools for which multiple products can be developed from, OEM's come closer to reaching economy of scale and enjoy lower per unit costs. This is not some black art, but rather, business applied to science. This article seems to convey platformization from a narrow scope, and doesn't really take into account the commercialization of products that OEMs have to face.

Another piece of misleading info offered in this article is the inference that the applicability, or future success of an RTOS provider is directly related to the degree of source that they open up to the world. The reality is that most OEM's do not have the staff with the right experience to really make a difference if they have access to the kernel, nor does they problem normally have to do with any kernel, or source-protected, code. It normally has to do with the way in which a developer utilizes the facilities offered by the RTOS provider. So. If we agree for a moment that having source is not a major necessity for developers, why do we cling to this notion?

I've got two theories. Firstly, there is an overwhelming concept that RTOS companies offered terrible support, or at least do not offer the right kind of support (WindRiver, we are talking to you here). If you can't get support to your issue, and you can't push your project/product manager to push off the delivery of your product to be more aligned to the RTOS provider's schedule, then of course you would need to look at the source for the underlying code for which your code body is based.

My second theory has to do with perception. The perception is that if your work is transparent to your customer, you have nothing to hide. This is certainly true of businesses post-Enron, but is this transferrable to the world of commercial RTOS providers where the bulk of their IP is in their source? Heck no. This is what keeps them in business to allow you to buy their products, which in turn allows you to build yours and sell it to your customers. Asking a commercial RTOS provider, the operative word here is commercial, to lift up their komono and risk losing their IP is like asking Cisco to show Juniper just how they have architected their packet distribution technology in their latest core router.

So … what am I saying. I'm saying that we need to hold RTOS providers to better support. I shouldn't just pooh-pooh on WindRiver here, this should be a call to all OS companies. Secondly, I'm saying that 'platformization' is equivalent to the development of a reusable component, which means that reliability should go up, time-to-market should go down, and costs should go down. All good stuff. And lastly, I'm saying that the concept of open source is highly altruistic, does have its' place, but not in an environment of taking innovative ideas and commercializing them for sale to the open market. No … I'm not saying we should all be like Microsoft. Instead, what I'm saying that we don't go to work for free, we don't give our products (for which we put 1000's of person hours on developing) to our customers for free, and we don't shout from the rooftops when we have to pay $2 for a cheeseburger. Why then should we expect to ask skilled developers to work hard, be creative, and developer and we want only to have to share their IP with their competitors and be forced to try to make money on services.

Greg Fields

My experience has been that having access to source for the OS has been crucial to solving difficult bugs that involve the kernel. This has been true for both RTOSes in embedded environments and non-RT OSes in new platform development, like device drivers or BIOS routine development. You don't always need it, but for those times that you do, it is absolutely necessary for timely resolution of problems. My rule-of-thumb is that needing support from a vendor in time-critical situations is never going to be fast enough, so you have to rely on your own resources. Having source code gives you a possibility for success at least.

Jeff Galloway
Senior Member Technical Staff
Hewlett-Packard Company

Once circa 1986, I was compiling some of vendor one's RTOS code using a cross compiler from a second vendor and I got the wonderful error message “This error can't possibly occur. Contact 'second vendor' error 6002.” We had a support contract in place with the second vendor and so called in with our problem… several days later they sent back a fix… of course the fix they sent back also broke the functional logic and rendered it useless but it provided the needed clue so we could fix it ourselves.

Ever since I have had a very strong desire to at the very least have read access to the source code so in desperation I can search out what a error message means.

The other use I remember we made of having vendor one's RTOS source was to add some tiny tweaks to the task switching to provide a function we (and most likely only we) needed for some performance problems.

The “This error can't possibly occur” still sticks in my mind after all these years.

James McMechan
Electronics Engineer

I don't like other people's cooking. I'll stick to writing an AS-RTOS (Application Specific RTOS). The highest quality embedded systems seem to have the “design as needed” approach anyway. If an RTOS is tailored to be part of the application, the system tends to run much smoother and specifications are met with ease.

Steve King

I wrote a proprietary RTOS before I switched to commercial RTOS. One major reason was the effort of support. When the proprietary RTOS was in use, my coworkers kept asking me for technical support. Nobody bothered to look at the code! The accessibility to source code doesn't matter. The technical support matters.When I first adapted commercial RTOS to our product, we got caught in the middle of cross-fire between venders. Compiler, debugger, and RTOS are from three different venders. Compiler vender started offering RTOS. Therefore, the RTOS was no longer compatible with compiler. None of technical support from those three venders was able to help. That was nightmare! Company only make money by adding features to products, not solving integration problem. I like the idea of platformization. I don't care if I could access source code or not. What I really need is an integrated platform and good technical support.

Vincent Wang

I agree with Bill Peisel's statement. That was our experience. The software crashes in a system call and the only thing you can look at is assembly code. You can reverse engineer that piece of the kernel (probably illegal) or try to get help from customer support (often slow in responding). So the best thing is to have the source code. It happened to us very few times, but it would have made a big difference.

Diego Warszawski
Embedded SW ENG
Eastern Research Inc.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.