Open Source is Already Delivering -

Open Source is Already Delivering

To read original PDF of the print article, click here.

Internet Appliance Design

Open Source is Already Delivering

Bill Gatliff

Read the article Are Open Source and Innovation Compatible? by Niall Murphy

One need not speculate about the benefits of open source software. The future holds more of the same.

I wish to provide a counterpoint to Niall Murphy's views on the present and future of “open source” software for embedded systems. Mr. Murphy makes some good points but has overlooked much of the value of the open source movement to embedded systems designers.

Open source business models
The key concern of the open source business model, as I see it, is how to remain profitable when users of your product have significantly more rights than under other models. Because the end user can obtain your product or service elsewhere, a revenue stream is not assured unless users find that you are a better choice than a competing supplier. The solution: be better, just like a plumber or race car driver has to be better than the competition if they want to stay in business. Simple enough.

Mr. Murphy is correct that this model doesn't break any new ground. But the concept clearly hasn't been practiced in software industry until recently, and its practice still isn't widespread.However, it's my opinion that because the open source model works for the good of the end user, and because the alternative is so bad, success of the open source movement is without doubt. As if to help motivate us, history provides examples of what the alternative outcome is. And it isn't pretty.

Think back to the late 1970s, before automobiles with computer-controlled engines were popular. At that time, garages and repair shops were almost as frequent as gas stations, and, through documentation and basic ingenuity, these shops could accomplish anything from serious diagnostics to complete engine overhauls. Prices were kept reasonable because competition was everywhere.Today, primary automobile maintenance is performed almost exclusively by factory-authorized service centers, also known as dealerships, at prices that clearly are not competition-driven. And the corner garages? They just change tires and oil now. To make matters worse, they compete with dealerships that offer those same services as loss leaders.

The reason for this change isn't that cars got more complicated. While it is true that cars did get more complicated, their sophistication was, and still is, managed at the dealerships by means of explicit service procedures and specialized diagnostic equipment-tools that would be equally effective if used at the corner garage.

The reason all those corner garages were reduced to tire and oil changers is because the major automobile manufacturers created “closed source” products (the electronic engine controller), and then controlled access to essential product information and training in order to create advantage for their own service centers. The major players in the software industry have clearly followed this same strategy, and have arrived at exactly the same conclusion: it works well for them.

Even the parallel in product release cycles is disturbingly similar. Consider this: in the auto industry, the typical way to stoke a bottom line when profits from existing products start to sag is to update the product line. By forcing the market to transition to this year's more profitable models, and phasing out support for previous models, the process renews itself.

Does any of this sound familiar? And don't get me started on the similarities between automobile leasing and the latest software distribution model: application servers.

The exit from this scenario, as proposed by the free software movement, is to do what the automobile and software industries currently won't do: offer basic freedoms to the end user, and give those freedoms “teeth” (source code, documentation, and unencumbered licenses) so that the user can exercise them by taking your product elsewhere when it makes sense in the user's mind to do so. When choice is truly an option, power doesn't consolidate unless users say it's okay first. And the decision is always revocable if the arrangement doesn't meet expectations.

Indeed, often only one or two controlling interests are behind all the dealerships in town. This isn't even permitted in some other industries like commercial radio.

Mr. Murphy asserts that open source has little to offer engineering as a software construction model. Quite to the contrary, most open source projects actually demonstrate how product development must be managed in order for the software industry as a whole to graduate from its current chaotic state. The idea that open source would lead the way into the future of our industry is a profound one indeed.To prove my point, consider the open source industry's most popular version management system, a product called CVS.1 CVS's client-server capability and non-locking development model make it the perfect tool for large, geographically diverse teams, which has always been the way of business for open source development. As the software industry as a whole makes increasing use of outsourcing and telecommuting, it's good to know that the open source community is already there showing them how to do it.

But a large, distributed development effort needs more than version management. It needs automated bug tracking as well. For that, there's Jitterbug,2 another free utility that allows users-or the software product itself-to submit bug reports via e-mail and have them automatically added to a database that can be viewed and managed with an HTML browser.

These examples show that if you want to see the next 10 years of software development practice, just look backwards at the last 15 years of the free software movement. And in case you don't want to set up and use these tools yourself, there's SourceForge, a free service that will supply them for you-but only if you're writing free software.3

A free software innovation that is currently in the works is EL/IX, a Red Hat-sponsored project to specify a common Linux-derived RTOS API that can be implemented by any RTOS vendor, free or otherwise.4 Although counterintuitive at first, both open source and commercial RTOS vendors stand to gain a lot in the success of this effort, because their EL/IX-compliant products become drop-in replacements for other EL/IX-based components. This makes it easier for a vendor to unseat an established competitor in situations where their product is clearly a better choice.

But consumers of EL/IX-related products stand to gain the most from EL/IX, because the cost of changing an EL/IX-compliant component for another is significantly reduced. When EL/IX hits the streets, RTOS vendors will finally be forced to market themselves based on their true strengths and weaknesses, instead of winning designs by pedigree, because they won't have the inertia of the installed user base they have today.

Mr. Murphy seems concerned that the success of the open source movement spells the end of contributions to the free software community from academia and other nonprofit sources. Again, the evidence doesn't support his fears.

Much free software was, and still is, written and maintained by members of the academic community, including instructors, graduate students, and undergraduates. One of the most profound examples of this is the Coda File System, which is the heir-apparent to the popular Network File System (NFS) as an approach for distributed data management on an enterprise scale. Coda is free software.5

Although technically still under development, Coda has been deployed at Carnegie Mellon University and elsewhere, and is maintained by members of the computer science staff there. The number of graduate students whose master's degrees are based on their contributions to Coda isn't listed anywhere, but is likely a significant number.Yet another example is Mosix, a novel clustering solution that lets networked Unix-like computers share processors and other resources without user or application intervention.6 Like Coda, Mosix is in routine use although it is still under active development, and it is also a product of academia.

For something with more of an embedded influence, there's the LART project, a StrongArm-based system that runs Linux.7 Among its strengths are that it offers about 250 MIPS on a measly one watt of power. And yes, schematics are included.

Free software morality
Unfortunately, I must agree with Mr. Murphy on one point-many free software and open source community members do mistakenly claim the moral high ground over their commercial brethren.

I participate in the free software movement because I don't need, nor do I want, the assistance of a “closed source” license in order to be successful-professionalism, skill, and proper marketing are enough. Releasing a product as free software demonstrates my commitment to that and isn't a morality issue at all.

My position on this subject is probably common, but my situation isn't-I don't have to convince a corporate boardroom to adjust their way of doing business. I only have to encourage a handful of clients, who are usually progressive enough to go along willingly after I explain the alternatives (which occasionally include finding another supplier). Go easy on the people that don't have such a handy lever to pull.

When you use open source software, you support those who wish to deal with you forthrightly and who allow you complete access to information needed to evaluate their strengths and weaknesses. When you choose free or open source software over a closed source alternative, you demonstrate your preference for vendors who want to portray their capabilities in the most unambiguous way possible-as source code.

If you think you gain a certain moral advantage by participating in this process, you've missed the point entirely. Free software isn't about giving users their rights back, it's about not taking them away in the first place!

Embedded code
Mr. Murphy states: “Encouraging embedded engineers to release code as Open Source rarely has benefit unless the hardware design is included.”

This statement isn't altogether true, but it is clear that providing the source code for an OTP or masked-ROM embedded system is of little benefit.

Or is it? Advanced users and analysts could use the source code to track down bugs and compare features of competing products. And in cases of product obsolescence (planned or otherwise), that source code could also be used by motivated third parties to populate replacement devices.

Furthermore, new coders would get an example of how a particular problem was solved. So open source software also provides a valuable and inexpensive training opportunity. How can this not be of benefit?

Unfortunately, both the FSF and the Open Source Initiative are understandably short on ideas for how to protect user's rights related to embedded systems that are field upgradable, and I have personally run headlong into this problem in my own work.

Furthermore, the liability issues that arise from an end user's incorrect programming of a piece of critical hardware should not be underestimated. But some products, like factory automation equipment, are built with this explicit purpose in mind (save the “incorrect programming” part, perhaps), so working out these details in an embedded product is not without precedent nor indication of reward.

And finally, there's the debugging opportunity that showing your code to the world offers. End users of technical products are often technically competent, and I always get beneficial feedback and ideas for improvements after my code (embedded and otherwise) hits the 'Net.

Embedded Linux is bogus
Thanks, Niall, for saying that.

But seriously, we've seen all this before, with Windows CE, with Java, and countless ideas before them, including C. One would think that we would have learned to tune this stuff out already.Embedded Linux is clearly a great idea in some situations, and a lousy choice in others. We're hearing so much about it now because some are trying to find an alternate means to market dominance than by the simplest, most obvious way: being better than the alternatives.

This tends to happen when the ratio of investment capital to good sense is what it is right now. As with all previous examples, in another six months or so Embedded Linux will assume its proper place alongside, and not in place of, other tools developers can use to solve problems. Until then, we'll just have to suffer through the normal progression of things.

The cost of free software
To the casual observer, free software can legally be had at no cost. But to the hardcore embedded users who need to tailor products before use, the cost of free software can't be ignored.

In my own experience, it is often about the same cost to download, learn, adapt, and use for the first time a free utility like newlib (a C runtime library)8 as to purchase a similar commercial offering.So why do I bother? First, note that I didn't include the “learn, adapt, and use” steps for the commercial offering. Those steps are often still necessary, and in many cases made more difficult by a lack of source code or experienced user community to consult for help.

Second, with open source software, the setup cost isn't repeated when I change processors or architectures, because free software tends to be more portable than commercial equivalents. So the time spent up front is really an investment that pays dividends on the next project.

And finally, the only way to make sure that future newlibs will be available is by showing an interest in the current one. That isn't always enough, but it is certainly a start. And when that's all you can offer, it's clearly better than nothing, and the true free software community will be grateful for it.

Making money
Okay, so if the free software model is so great, how does somebody practice it in a way that feeds a family? Here's my recipe.First, find someone-hopefully your current employer-who wants you to perform a software technology service, say, develop an application.

Embedded or otherwise, free or not, isn't important yet.Second, gain the client's approval to build the product using free software. The product itself isn't required to be free for this step to work, but keep in mind that certain free software can only be used to construct free software. Check licenses carefully.Next, build the product. Avoid skipping this step.

Finally, repeat this process starting at the first step, using your new skills as a developer of similar applications, and/or as someone skilled in tailoring and using the tools that went into the product. Season to taste.

This approach is working for me, anyway.

Free software
The truth is that the free software movement is a long overdue course correction that reverses the software technology industry's progression towards a state that holds the rights of software vendors in higher regard than the rights of software consumers. Furthermore, products of the free software movement provide models that demonstrate how software should be designed, managed, and marketed in the coming years.

By my count, that's two heaping helpings of positive change. All for the price of, well, free software.

















The Last Word
Niall Murphy responds

Mr. Gatliff has made some good points about the contribution that open source has made. However, I remain unswayed in my thoughts on its prospects for the future. In starting his argument with an analogy of car dealerships, he points out that the dealership model makes it difficult for the small corner garage to compete. This model is not in the best interests of consumers, but it is in the best interests of the large dealerships. Surely the point that should be derived from this is that big business serves consumer interest less well, and since open source has now become big business, we are not going to see them work to the advantage of the small guy as much as in the past.

One of the examples Mr. Gatliff uses for the future prospects of embedded Linux is the prospect of a common API for reduced versions of Linux. If Red Hat is the only player in the EL/IX game, then EL/IX has nothing to offer, since no one else is going to be a guest at a Red Hat party. And why should they, they are all trying to innovate in ways that make their product different enough to be able to make money out of it. Did I say different? Oops, I meant to say proprietary. Unless the other major Linux players buy into EL/IX, it is more likely to cause fragmentation of Linux than to prevent it, and so far Red Hat has not convinced any major players to publicly come on board.

On the question of whether embedded software for custom hardware can be released as open source and whether there is an advantage for the consumer, Mr. Gatliff makes the valid point that having the source may allow a technical customer to track down a bug more quickly, or more accurately. This is a model that will find favor with few. Are we encouraging suppliers to push the onus of finding bugs onto the consumer, rather than leaving it with the supplier? I wonder where that will end. Maybe you do not need to document a device's behavior if the customer can merely read the source to figure out how it works. In any case the debate on releasing this kind of device-specific code is futile-in the real world it is not happening, mainly due to concerns of company secrecy and competition.

From Mr. Gatliff's description you would assume that the open source products mentioned were assumed to be of higher quality than the commercial equivalent, or that they preceded the commercial competition in some way. But if you look at the Web site for the Coda file system, you will find it described as an experimental file system. In more detailed documentation it states “the rough edges, which inevitably come with research systems, are slowly being smoothed out.” While this is a straightforward description of a product in its early stages, it is apparent that this is not something you can depend on for industrial strength development.

Mosix is another product mentioned, but the latest version is 0.97.6. Can you take risks on a product which is not even considered a 1.0 release? These two cases underline a common problem with open source development. They have great difficulty getting releases out the door. Open source projects tend to be so engineering-driven and technology-oriented that the feature list grows endlessly. This leads to situations like Mozilla, which has been waiting two years for a release since it became open source. At the time of this writing the next release date is uncertain.

I will finish with a glance into my crystal ball. While I am genuinely thankful to the free software movement for the tools they have made available so far, I predict that in five years time most embedded developers will not find themselves using any more open source products than they do today, and the number of such products, which can reasonably compete with fee-based products, will not have significantly increased.

Bill Gatliff is an independent consultant specializing in free software tools and techniques for embedded systems development. He is a member of the Embedded Systems Conference Advisory Board and a frequent contributor to Embedded Systems Programming magazine. Bill welcomes comments via e-mail to .

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.