I just completed a project doing embedded Linux on a PowerPC based system, and it was an eye opening experience. I've been interested in embedded Linux for a few years now, invested some time to learn the technology, and when the opportunity was presented to me to do a project with Linux from “soup to nuts,” I jumped on it. As a consultant, I've spent the last 15 years making my bread and butter doing VxWorks-based projects, and relished at the chance to deploy a project with Linux. The founders of the company I went to work for were particularly attracted to Linux due to the pricing structure (at first glance, free), not to mention the numerous benefits of working with Open Source.
First, a kernel was not available for our particular strain of the PowerPC, so we had to investigate our options. We looked at vendor support from several of the established embedded Linux vendors, and their schedules showed availability after our product was to ship . To accelerate development, we'd have to spend a lot of cash. The group I was working for had considered having me do the port, but it's not my area of expertise, and further, I had other pressing matters to deal with — our actual application code! Fortunately, I was able to bring in a consulting group I had worked with in the past to do the port to our processor (for a reasonable fee).
Embedded Linux was no longer free. We did get stabilized Linux in the time frames which we required, but we had to rub money on the project.
Our Linux port was stabilized on a vendor supplied evaluation platform. Unfortunately, our proprietary hardware did not follow the configuration of the evaluation platform, so we had to port. This itself is not an unusual situation for many, even those who use commercial solutions such as VxWorks. The trap we got into was we ported the first release of Linux we had to our hardware, and then each time our consultant group released a new version, we had to do the port again. Developers using commercial solutions such as VxWorks seldom have this problem, as VxWorks is stable, and it's not often they have to deal with new releases from Wind River in the middle of a project.
Managing our embedded Linux environment started to take time. I'd guess that over a two month period, while stabilizing our hardware, and receiving updates to our Linux kernel, 30 percent of my time was spent managing Linux. There were at least five updates to the operating system over a two month period, all while we were trying to ensure the stability of our proprietary hardware. At this point, not only had we paid for a kernel port, we now paid in time.
Keep in mind that in the middle of all of this, we are booting Linux over the network, and mounting the Linux root filesystem NFS. We are testing our application by putting it in the NFS mounted area, and invoking it from telnet sessions. We're not even in ROM yet, we are feebly running, and in two months, we've got to drag this box up to a potential investor for a demo!
After a couple of months, our hardware is stable, our Linux port is stable, and our application is somewhat operational (which, after all, as my boss often reminded me of, this was the point of the project — make the application run!). And our investor demo looms in a week. If our path had been VxWorks, we'd already be ready for ROM; the Wind River supplied build environment allows you to easily boot over the network, or simply embedded the image you have booted over the network. With Linux, and the NFS root filesystem, we had a huge directory structure that needed to be tweaked to fit into our 4Mb flash device. Since we had gone the route of no vendor support, we were on our own.
The Linux build environment, at least for the PowerPC, does support building images suitable for ROM. However, what is not supported is creating the ramdisk image which contains all of the executables in /bin type directories, or /etc and /lib content. Thus, I had to cobble together a set of scripts to do this. These scripts allowed us to pull in a sub-set of the existing /bin and /sbin type directory binaries, and then through old world craftsmanship (by hand), I established what libraries needed to be included. A couple of weeks at this, and we've got a ROM based image that our application does not run under, and it's slightly bigger than 4Mb. Would vendor support have helped? Most likely, but that comes at a price. We went to 8Mb flash, and continued to struggle to configure the environment to allow our application to run.
Even at this point, our system did not auto-boot; it ran from flash, but we still had to start it by hand. The next step was configuring system initialization files to start everything up, and make sure the application ran as a daemon. Not a lot of work, but again, I'll compare it to the VxWorks environment; the development environment rolls right into flash pretty painlessly. We got it going, eventually.
By the way, we missed our demo for the investors.
Other trappings? Well, Linux is a protected namespace, MMU based environment. You can't see your devices from the application-level code unless you write device drivers. We did. They took work. In VxWorks, it's a flat model, and often, device drivers are extensions of the application. In our case, I was the “kernel man”, so a lot of this low-level work got serialized through me. Sure, we had examples due to all the Open Source, but we didn't want to give our application away, so had to write modules, which did not link statically with the kernel, in order to retain our proprietary engineering. For some of the chips on our hardware, we had Linux drivers from the vendors that worked under x86, but not PowerPC. Trying to port them was work, as Linux is not Linux (everybody has a different version of the kernel on their desk), and we had endian issues.
Cross development tools? I had to build them. Not a big deal, but not for the faint hearted. I was able to find a web link on how to build the PowerPC cross environment with GCC, but ran into hassles on building glibc. Persistence paid off, and in the end, we had stable cross tools. Certainly, these would have come from a vendor, if we had gone the route of vendor support, but vendor support costs money.
Is embedded Linux viable? You bet. Is it free? No way. If time-to-market matters, you get a lot from companies like Wind River with stable environments, or perhaps the current embedded Linux vendors. Drawn from my long experience with VxWorks, I concluded that we had added 30 percent to our schedule through going the Linux path, and keep in mind, we spent some cold, hard cash in the process to get the kernel stable. Yet now that it's done, the actual cost of goods is kept down, as there's no license fee.
There numerous hot spots developers should be aware of should they choose to go their own path? Here are the key ones are:
- Basic system configuration, such as getting the cross tools going, and getting their development environment setup, takes time, experience and specialized knowledge.
- Installing Linux on proprietary hardware takes work, and without knowledge of the kernel environment, can be a tough road to go.
- Building ROM/flash based images is not trivial.
- Understanding of Linux drivers is essential.
- Vendor support. Evaluate it carefully; it could save you a lot of time, and get you running quickly, but make sure you know what you are buying.
Embedded Linux is real, it's viable, but it's not free. It's easy to get side tracked with embedded Linux, and find yourself in the OS support business instead of the application development business. Would I do it again with Linux? You bet! But this time, with eyes wide open.
Thomas E. Besemer
The author is an independent consulting who specializes in Embedded Systems. You may contact him through his web site: http://www.tbcorp.com.
Apparently holy wars are waged still by holy warriors in the embedded community – with little regard for the underlying reason the audience is granting them the forum to speak in the first place (we are looking for solid and specific information to get our job done). Reminds me of the assembly vs. C and C vs. C++ holy wars.
I read this article, like many I presume, because I am embarking on my first embedded Linux project. But sadly, the resulting wars did not offer but very few specific inputs for someone, like Mr. Besemer, who clearly WANTED to use Linux for (apparently) the right reasons.
Now, for those EMBEDDED Linux types who responded to and trashed Mr. Besemer, if you have several embedded Linux designs (for major score – $$) under your belt, I ask your specific and practical input on how, knowing what you know now, would you approach the issue of building your FIRST embedded Linux product?
Let's define this as a product that requires a simple yet elegent GUI, a fast external interface pipe (USB 2.0?), and speeds in the 10's of usec range to process inputs and carry out requested hardware tasks.
Let's assume the product will be designed by a small team of embedded engineers with misc RTOS's and several non-trivial embedded product designs under their belts.
Of course, just to sweeten the challenge a little, it's a low cost product that must be delivered in 12 months, on a low budget, using an off the shelf expandable floating-point CPU board, board support package, and tool chain.
(Mr. Besemer, my hat's off to you! Took guts to open up and expose your mistakes, trials, and tribulations in an open forum. Reminded of Ganssle's writings. It really was helpful to me to read about your first Linux experience, as I consider mine.)
Principle Software Engineer
Solid Core Technologies
Besemer's article doesn't critique “Embedded Linux” so much as the rhetoric that surrounded Linux as it entered the embedded market over the past several years. Of course, any salesemen will offer you the world, but the Linux-hype was particularly difficult to filter because the actual product comes in many different forms and is constantly changing — to say nothing of the passion with which it has been evangelized.
Embedded Systems Programming
Although Linux is free, every company has to invest in new tools or systems. At Chess we calculated that new Linux projects will cost about EUR 28k for setting up Linux.
But, if you know that you're going to use Linux, you can select supported devices, like uarts, i2c RTC's and that like. If you don't, plan some time writing this yourself.
We have had WindRiver software, but even then you have to invest in knowledge and writing drivers. And, WindRiver is NOT free @ all.
Now, when using Linux, more than 70% of the drivers were available.
Bernard C.M. Wesselius
Senior Software Engineer
Chess Linux Group
I think, that it is your own choice – what to use…. The cost structure differs on your demands. If you have to do long period perspective some technology development – Linux almost have no competitors. But you have funded your team education for Embedded Linux. If your project is hard sceduled and you know nothing about Linux – you may loose your money by a lot of different ways:-)))))
We use Linux in our embedded projects and understand, that we use the free platform, but we have to pay for it by our self education.
If you pick a piece of hardware that the system does not support, what do you expect?? Do you really beleive vendor supplied drivers are going to be great?
Also, don't badmouth linux or open source simply because someone wrote a bad driver that relied on endian-ness … linux kernel uses architecture independant routines for accessing hardware.
Any project is to have a learning curve, if I were to do projects on VxWorks it would take me a lot longer since I don't know how to use it at all.
Ken Emmons Jr.
This is the most ridiculous thing I've ever read. Embedded linux is free, but you still have to know what your doing.
As an engineer I sympathize with his plight of being thrown into a situation that he has no experiance in and being expected to master it in a couple days. But he obviously doesn't know what he's doing. Some notable flaws in the article:
There were at least five updates to the operating system over a two month period, all while we were trying to ensure the stability of our proprietary hardware.
Of course there were! this is linux, it's always evolving, but you don't have to install them all!
You can't see your devices from the application-level code unless you write device drivers
This is simply not true at all. User space drivers in linux are both possible and widely done. Even real time user space drivers are available with the RTAI patch.
Drawn from my long experience with VxWorks, I concluded that we had added 30 percent to our schedule through going the Linux path.
Of course you did, as you just pointed out you have 15 years experiance with VxWorks, and zero with Linux. How could it possibly not have added time to your schedule?
This is the most common and disgusting attitude in engineering today. That naive belief that there is no such thing as expertise, that everything is either easy or hard, it doesn't matter who you are or what you've been doing for the past 15 years.
This article was misleading and false, and shame on whoever published it.
Embedded Linux developer
I always enjoy reading an article on peoples experience using linux for embedded systems, so I apreciate that you'd share this with us.
But, to be frank, overall the article seems to be trying to paint a negative image of embedded linux, and unfairly so.
Throughout the article you consistently compare embedded linux to vxworks, and always find e-linux coming up short. But you could at least admit that you can't possibly be objective with the comparison. As you say in your article, you have 15 years of experience working with vxworks, and seeminly no real experience with e-linux at all prior to this project. Of course e-linux is going to seem lacking to you.
Also, you tone seems to penalize embedded linux for several problems that were more design errors than anything else. Of course if you decide to use hardware not supported by linux it's going to cost you. Any bad choice is going to cost you. To select hardware that doesn't already have linux support, and then expect everything to just work, that's truly ridiculous. Would you build hardware that isn't supported by vxworks and then expect vxworks to just run on it? Ridiculous.
As for the problems you bring up in regards to the constant upgrades you received from the consultants you contracted for support, that has little to do with e-linux and more to do with your choice of consultants. To be fair though, linux in general is known to be a bit of a moving target when it comes to development, which is why most successful embedded linux projects take a snapshot of the kernel and then freeze it, updating with only the enhancements they need.
And really, that's the bigger issue. Embedded linux is not easy for people who don't know the platform well, and don't have enough linux experience to make good choices. The idea that you could use embedded linux as some sort of silver bullet is wrong, as is the idea that somehow embedded linux failed you and was responsible for missing your product demo.
All in all though, the article was a good and interesting read.
Embedded Software Engineer
Total crap !
This article is the best reference not to engage him and his company services.The best joke is his homepage, where he praises himself an “embedded linux expert”.
He should have done his homework before using technology which is obviously way out of his “expert knowledge”.
Why do so many people assume free means something for nothing? The Linux community is not at your beck and call. Free means freedom. The freedom to do what you choose with your software, in any way you choose to do it. Heaven Forbid you be required to do some work to make your product jump thorugh the hoops you want it too. If your system isn't supported out of the box, of course you need to do work to get it all put together, and that costs money, regardless of what OS you use. I would argue that every problem and obstacle you encountered in the course of this article is attributable to your novice standing in Linux development. As you repeat this process, and your familiarity with the system grows to match that of your VxWorks skills, I would wager you will come to find the problems you encountered here are actually very usefull features. And when you do, you'll make your demo deadline.
I'm glad to see that, in the end, Mr. Besemer expresses confidence in Linux as an embedded platform and an interest in using Linux in future embedded projects. However, it is clear to me that Mr. Besemer approached Linux with certain attitudes that I have seen again and again in the three years I have been working with embedded Linux. All too often I have seen developers hold Linux to a higher standard than VxWorks while refusing to embrace the strengths Linux has over VxWorks.
Mr. Besemer begins by telling us of how his client had to pay to have Linux ported to the specific PowerPC family in use. My question is, what would have happened if VxWorks had lacked support for that processor? Mr. Besemer's client would have either had to 1) wait for WindRiver to support their chosen platform; 2) pay WindRiver to do a custom port; 3) choose another CPU; or, 4) abandon the project. These are the same options available with Linux. At least with Linux it is possible to “shop” the port to multiple vendors or to do it oneself.
Mr. Besemer goes on to complain that he had to keep porting his board support as new kernel versions were released. One has to ask what compelled him to be on the latest kernel release available? He should not have started the project if he needed unreleased features, so we must assume that these releases were bug fixes? Would not he thankfully apply bug fixes if they were released in so timely a fashion for VxWorks? Either way, as one who has done suport for more than half a dozen PowerPC Linux platforms, I would be very surprised if moving board support between kernel versions took more than an hour each time. I highly doubt that was a significant drain on anyone's schedule.
Mr. Besemer also states that booting from ROM was a problem. While building the requisite RAM disk image is not an everyday task for the average user, it is a task that bears little difference from building an initrd or a chroot-jail in a desktop Linux environment. Either of those tasks are reasonably well documented in HOWTOs and other publicly available sources. I suppose the lesson here is that one should not expect to use Linux in an embedded environment without knowing how to use Linux in the first place.
Nevertheless, one has to wonder why using an NFS-mounted root filesystem would prevent a successful demo of the application. After all, this is Linux where (for the most part) one filesystem is as good as another. Using a laptop (running Linux, of course) as an NFS server should have been a small hardship for an investor demo.
Mr. Besemer goes on to warn us of the need for init scripts and proper device drivers. Of course, the use of a script to start the application provides a lot of flexibility that can be easily debugged without the code/build/load cycle needed with C code on a VxWorks system. And, device drivers are a feature of the memory protection sorely lacking in a normal VxWorks system. Do you really want your application accessing hardware from just anywhere?
We are even warned that some Linux drivers have endian issues. Have you ever seen endian issues in embedded code? (sarcasm intended) I find the availability of a plethora of drivers for common hardware ample compensation for some occasional debugging. At least with Linux, I CAN debug faulty drivers…
Finally, we hear of problems building the cross-development tools. I will concede that this task is not as simple as it probably could be. However, options exist even if one does not want to pay for a proper consultant. Denx Software Engineering (www.denx.de) makes their Embedded Linux Development Kit available for anonymous download with support for both PowerPC and ARM processors. I would advise one new to Linux to make use of this resource before going it completely on one's own.
With all that said, I think Mr. Besemer did a fine job of relating his own experience in using embedded Linux for the first time. However, I do wonder about the story of his using VxWorks for the first time. I often think that VxWorks developers forget that even VxWorks was new to them once. The fact is that Linux and VxWorks are very different environments, and skills in one environment do not directly transfer to the other environment.
VxWorks has been ubiquitous for many years, and many active embedded developers are quite comfortable with VxWorks. However, Linux has many qualities that have caused both developers and managers to begin embracing it. Linux is a great system with many strengths. However, much of that strength cannot be unlocked by treating Linux as a drop-in replacement for VxWorks. Instead, we must let Linux be Linux.
John W. Linville
LVL7 Systems, Inc.
It's not clear how the author can compare a project:
- using an established commercial RTOS
- in which he already has considerable expertise
- and having support for the processor he is using
- using an unmodified desktop-oriented operating system
- that is not ported to the processor he is using
- him having little experience with that operating system
- without the support of one of the embedded Linux toolkits
The only surprising thing here, in light of the stacked deck, is that it took him only 30% longer with the Linux approach.
Also not clear is why he kept applying (and presumably paying for) each new kernel provided by his vendor. Did each one of these upgrades fix a specific problem for his device?
Finally, the comment about being required to “give our application away” is misleading. It is well known that the GPL would require him, at most, to release the code to the device drivers, not the application.
I felt Mr. Bessemer's article was very useful and made some important points that are relevant to organizations considering switching from some other embedded programming environment to Linux. It seems that many of the people who have worked with embedded linux took the article as a put down of embedded linux. I did not reach that conclusion reading the article. It was a roadmap to some of the issues and biases that one deals with in switching from anything that is familiar to anything that is not.
While those in the Linux community may understand that free means freedom, many of us outside the community, in particular, management types(think the Dilbert variety) read free and think that they are going to reduce the cost of their next embedded programming project.
Mr. Bessemer's article was useful to point out that in switching from familiar ground to unfamiliar ground, there are hidden costs that must be taken into consideration so that there are fewer surprises along the way. While perhaps that may seem obvious to some, many of us need to be reminded of that. I look forward to doing my first embedded Linux project armed with some useful information regarding what to plan for and anticipate as part of the process of switching tools.
Great article! I think it does *much* more good for embedded Linux than it does bad.
Anyone who's considering using embedded Linux for a project thinking it's free is going to be in for a big surprise. And will likely have very negative stories to tell of embedded Linux when it ends up costing more than they expected. This won't help embedded Linux make inroads.
As someone who believes in embedded Linux, it's best to avoid this happening by making sure people who are considering using it are aware of the real costs involved.
Principal Software Designer
AMIRIX Systems Inc.
I think this article was informative and well-written, reminding us that the freedom of Linux comes with a cost that needs to be accounted for. It is always interesting to read about real life experiences in putting Linux into work. I do not think the author has a negative attitude towards Linux, he points out real problems. On the other hand, some of the response given by blue-eyed penguin fanatics makes me wonder whether they have ever used embedded operating system at all.
Thomas Besemer Responds:
I wish to thank each and every one of you for your feedback. The passionate responses show clearly that embedded Linux is alive and well, but still maturing. I would guess that Wind River would love this sort of passion towards VxWorks.
For those of you who didn't feel I was qualified for the job, I beg to differ; I'm one of the early adaptors of embedded Linux, have done consulting work for the vendors, and walked in with a sense of knowing what this was going to be about. Yet this was the first “soup to nuts” project I had done, and no matter how qualified I was (or thought I was), there was a learning curve. It's a reminder to all of us that no matter how much we think we know, until we actually do a project from “soup to nuts”, we don't know. In regards to the question of how hard was my first VxWorks project? It was very hard. I have years invested in VxWorks, and still learn on every single project I do with the Wind product line.
I wish I could report that this was my first experience with doing embedded Linux, but it was not. I've been “Unix” literate for years (what's Windows, again?). Yet I got bit, and it wasn't just Linux; it was poorly supported hardware (and no, I did not pay for each release of the OS; it was very interactive with the consulting firm — I tested, I reported, they fixed, they released, and they were highly competent people with a lot of experience; any mistakes were not theirs, but ours — we had picked a processor that was not currently supported under Linux, and that was a mistake).
For those that have not done a project with Linux, in an embedded capacity, I assure you it's harder than it looks. I'm not anti-Linux. In fact, I'm pro-Linux. I think it makes a lot of sense. I have a friend who is very wealthy, and an active VC kind of guy. He reviewed the comments on what I wrote, and offered up the following: “Sounds like most of them drank the kool-aid.” No doubt. Embedded Linux has almost a “cult like” following, and from that, you find fanatical perceptions on it.
My comment to the zealot followers and supporters of Linux? Keep it real, keep the business model in mind. And keep in mind that I was only a reporter in my article, reporting only on my experiences.
Thomas E. Besemer