Alliances Drive Embedded Linux Toward Prime Time

Alexander Wolfe

Kerbango's stab at applying a downsized version of the traditionally hefty Linux OS to the world of embedded appliances may be an early indicator of what's being replicated at dozens of software developers. That's because purveyors of operating systems and tools are just beginning to roll out a rash of products billed as either "embedded" or "real-time" Linux. According to Kerbango's Hewitt, that wasn't the case a scant 12 months ago, when work on embedded Linux platforms was at a far earlier stage industrywide.

Essentially a clean-room version of Unix, Linux was conceived by then-University of Helsinki computer science student Linus Torvalds. (Today, he's an employee of Silicon Valley startup Transmeta Corp.; see "The Software Side of Crusoe," April 2000, pg. 85.) Like Java, Linux initially made its appearance in the bloated world of Enterprise computing. Lately, however, a plethora of embedded Linux implementations and toolsets are beginning to make their way to market.

"I think Linux is in a footrace," said Michael Tiemann, chief technology officer of RedHat Inc. He pointed to VxWorks from Wind River, Windows CE from Microsoft, and the BeOS from Be Inc. as some of the contenders.

Outside operating systems apart, at least five and perhaps as many as 10 vendors within the Linux community itself (in this often secretive corner of the software world, no one is quite sure of the exact number) are going embedded systems code one better and attempting to roll real-time versions of the OS-even though the basic kernel was not designed to support such operation. (Included among that number are Lynx Real-Time Systems Inc., Lineo Corp., MontaVista Software Inc., QNX Software Systems Ltd., and others who either haven't fully disclosed their real-time plans or could not be reached at press time.)

"You have to understand that this is an open source piece of software and people each have their own enhancements," said David Ditzel, president and CEO of Transmeta.

And because Linux is "open source"-a concept pioneered by the Cambridge, MA-based Free Software Foundation-which is famous for its GNU compilation and editing tools, any Linux updates become (theoretically, anyway) available to the entire software community. In many ways, today's rush toward Linux by a herd of companies is reminiscent of Java two years ago. That is, technical efforts are taking center stage while business models-how to make money-seem to be something of an afterthought.

"It's so early," said James Ready, president and CEO of MontaVista Software Inc. "The wheat will eventually get separated from the chaff both in terms of real technical capability, both in the product itself and in engineering services. In other words, does the company have an understanding of Unix and real-time issues." Another big differentiator is that Java had a major corporation behind it (albeit one that was largely pushing an Enterprise version of that platform). Linux's biggest corporate support comes from IBM Corp., which has pledged to support the OS in everything from full-blown mainframes all the way down to Internet appliances. To help the company deliver on that promise, IBM recently formed an internal Linux group.

However, most of the names heard on the Linux battlefield are either new companies or the traditional purveyors of RTOSes. And even there, many of the old-time RTOS players are taking a careful wait-and-see attitude before they divert resources from their primary, proprietary OSes and jump full-tilt into the Linux fray. Indeed, most industry experts agree that the next year will be a telling one for Linux in the embedded arena.

"We are seeing a serious groundswell in terms of using Linux in embedded systems," said Doug Locke, president of TimeSys. "I'm talking about a wide range of applications, stretching from very small things one might find in an automobile to much larger military-control apps."

There's just one major stumbling block on the road to a Linux that's useful for developers of all stripes. Namely, Linux can clearly be better downsized to fit the bill for tight-footprint embedded applications. However, whether its kernel can be extended to deliver real-time deterministic performance is another question entirely. At issue: the native Linux kernel is designed for coarse-grained synchronization and system tasks can't be preempted.

Kernel characterization
Still, several companies are laying down bets that they can deliver real-time versions of Linux. Such an OS would obviously have a jump on an embedded-only offering, which would at best support "soft" real time or, if not that, then "human" time operation such as might be required in an automotive entertainment system.

Two of the more notable participants in this rapid rush to real time are newcomers Transmeta Corp. and MontaVista Software Inc. In that regard, a question had surrounded Transmeta Corp., which earlier this year released two embedded processors powered by a downsized version of Linux. At the time, it wasn't clear whether the company would open up its "mobile Linux."

Ditzel of Transmeta intends to put those questions to rest. "Our mobile Linux will absolutely be open source," he said. "What we're doing is, number one, adding power management and number two, adding a compressed file system called CRAMFS (for 'cram' file system)." This enables programmers to shoehorn files into about half the space they would normally require. "In an [Internet] appliance, where you might want to store your program in a flash ROM, rather than using a hard disk, you'll need only half as much flash ROM-and that's a big deal," he said.

Contenders abound in what may be an early sign of polarization in the Linux marketplace. But one of the big questions is just exactly what the memory footprint must be, at minimum, to meet bare minumim embedded requirements. Tiemann of RedHat floats a figure of 4K as his low-end target. However, such a setup offers very limited funtionality-basically a kernel with no file system, no networking support, no user interface (which must be tacked on separately), and only bare-bones I/O.

Indeed, others disagree that a 4K profile is capable of providing what will be required for a total solution. (Even Tiemann's breakout of the user interface as a separate number would seem to validate such a conclusion.)

More than anything else, Linux's legendary size-it has hundreds of services and calls-has made it a bear to wrestle to real-time ground.

"To do anything really useful including a kernel and reasonable functionality, you have to go to at least half a megabyte," said Ready of MontaVista. "Much smaller than that is not going to be interesting." However, he notes that more complex systems, such as those that run multiple applications, will have to have even bigger footprints. For example, adding a sophisticated file system could boost things to the 4MB range.

Kerbango, which has used MontaVista's HardHat Linux in its Internet radio, says the kernel, TCP/IP stack, and support for virtual memory mapping require about 600K. The user interface and software codec that enable the radio to play MP3 audio files aren't included in that count.

Figure 1: Lynx Real-Time Systems will use its "Blue Cat 2.0" offering to deliver hard real-time Linux support.

Real-time or just embedded?
"Our unique positioning is that we're doing 100% pure embedded Linux," said MontaVista's Ready. "Here, I use embedded as the highest level of characterization; embedded requirements are many things including real time." One result, Ready said, is that his company has been able to reduce interrupts-off times in the kernel. MontaVista is now shipping that feature and also has folded it back into the open-source version of Linux.

Indeed, the most serious functionality requirement that separates the "hard" kernels from their pokier embedded cousins is true real-time. That's been a subject of much debate in the Linux community and it's not a subject likely to be resolved anytime soon. "To make it real-time, what you have to do, fundamentally, is deal with memory, CPU, and I/O resources in such a way that when you 'speak' with them for parts of the system that have to share those resources, they can actually meet their time contraints," said Locke of TimeSys. "Linux as it exists doesn't really do that."

Another route to hard real time is coming from Lineo Corp., which acquired Zentropix Computing earlier this year. The latter's real-time application interface (RTAI) will be used by Lineo to bring determinism to the Linux kernel.

Competing flavors
Despite the fact that numerous software distributions will carry the Linux name, peel back the onion skin and there will be some major differences between the implementations making their way to market.

Unlike Java, Linux is not driven by a group of developers intent on forging a firm standard-despite the fact that it sometimes seems that way. Strictly speaking, Linux is open-source code. Perhaps that's why there seems to be reasonably cooperative competiton in the Linux community. That stands in marked contrast to Java, where Sun Microsystems holds all the cards. (Of course, competition could get far more intense once products start to hit the market in force.)

"I think the Linux community is much more diverse than anybody realizes," said Tiemann of RedHat. "And there are people connected with Linux who are real embedded gearheads."

Regardless, sometimes standards are in the eye of the beholder. "Using the word 'standard' feels funny, because it connotes something that is not the way Linux works," said Ditzel of Transmeta.

"Some people would like to have their particular ideas as a standard because then they have other products surrounding that, which will fit nicely," said Paul Zorfass, analyst at First Data Inc.

For one, RedHat Inc. is pushing ahead in its bid to nail down its homegrown Linux application programming interfaces (APIs)-called EL/IX-which it hopes will come to become dominant.

Tiemann of RedHat doesn't see anything nefarious in that. "The underlying principle that [Linux creator] Linus Torvalds preaches is that the best code wins," Tiemann said.

EL/IX, he explained, is an attempt to pare down Linux so that it's manageable for embedded apps. "We took the APIs that Linux programmers use on a daily basis-for example, printf is from the C library, sleep is from the kernel-and we looked at the Posix 1003 standard and we looked at how that standard defines bundles of functionality and how it defines application and operating system profiles," Tiemann said.

"We created a definitive list of ways that you can slice and dice all the entry points in these various APIs," Tiemann said. "So that you could, from a top down perspective, define a set of functionality that was consistent between two different people working from two different source [distributions]."

RedHat's EL/IX draft is available on their Web site. Their EL/IX distribution will be available by mid-fall, Tiemann said. Separately, Ready of MontaVista noted that he took a different approach, by helping to found the Embedded Linux Consortium. What's the purpose of that group? Linux promotion, pure and simple, he says.

For real-time developers keeping their eyes focused on the open source distribution, the Web site of choice remains www.rtlinux.org. The central repository for Linux remains the kernel.org site.

Making a dime
The final question for developers-though perhaps the first one for marketers-is just how Linux's purveyors are going to make a profit. As with the Java world, the big question remains of how the Linux players plan to make money selling a product that typically ships for under $100.

"It's not on the operating system," said Darrin Skewchuk, an executive at QNX. "It's on the support and services."

Indeed, some stalwarts of the RTOS world are waiting to see whether Linux will make as big a splash as some expect. "Right now, Wind River is genuinely trying to determine the appropriateness of Linux for the embedded space,"said Curt Schacker, a vice president at the company.

"Embedded is a lot of things," said Schacker. "It's very broad, there are many different applications, and I think that Linux has gotten attention in some of those spaces-most particularly in the Internet appliance space where people are interested in using the Intel x86 architecture."

Ready of MontaVista believes Linux could prove complementary to Java. "We're finding that Linux is becoming a very strong platform to host Java," he said. "That is, you don't want to run Java on a bare machine, so you have to run it on something. And we're finding extreme interest in Linux being the engine for Java. We're working with three or four Java implementations right now-that's a huge trend."

For now, however, Schacker of Wind River plans to keep taking reality checks on the software flavor-of-the-month. "I think a very important question is, will Linux find acceptance in the broader embedded space. Is it going to move into networking, industrial control, and military/aerospace," he said. "I don't think there are answers to those questions yet. So what we're trying to do is study the trends and understand where Linux is going to go."