A company, let’s call it Acme Products, had a very popular product which many of us use on a daily basis. Powered by a single 16 bit processor, all the code fit into 256k of ROM. The code had evolved over decades; patch after patch added to the convoluted mess. Maintenance costs escalated yearly.
Engineering convinced management that a complete rewrite was needed. Not unreasonably they elected to go to a 200 MHz high-end 32 bitter. Perhaps unreasonably, succumbing to the siren call of cool , the engineers elected to replace a traditional RTOS with Windows CE, an extensive graphics library, and many layers of middleware to insulate CE’s API from the application, and to provide all of the resources needed to do, well, anything imaginable in the future.
Five years of development and $40 million in engineering crept slowly by. System ROM went from 256k to a meg, to two and more. When they called me the application now consumed 32 MB, yet included only half the functionality in the original, 256KB, 16 bit version. Button presses, which previously offered instantaneous response, now took seconds. The safety aspects of the system were in question.
Grand dreams of vendor-neutral code mandated so many layers of middleware that the system burned too much CPU time, consumed huge amounts of code space, and ate so much engineering time they ultimately canned the project and started from scratch.
Then there was, let’s call them ABC Security, who built a secure communications device upon a Pentium Pro. A quarter gig of RAM holding a version of UNIX, middleware that again aimed to provide a vendor-neutral API, more API-neutral access to a flash file system, a chain of device drivers so deep, so convoluted, that was sure to baffle Linus himself, led to a product that took full ten minutes to produce a dial tone when the unit went off-hook.
Also cancelled, after burning $2 million in engineering dollars.
Then there was the data acquisition product that migrated from an 8051 to a Power PC. Engineers replaced the old brain-dead idle-loop design with an RTOS from a well-known vendor, and the simple data storage structure with a real filesystem. TCP/IP replaced the previous incarnation’s proprietary synchronous communications mechanism. And this system, too, never saw the light of day as a real product, as it could no longer keep up with the input datastream.
Magazine ads scream the benefits of higher levels of abstraction, of complex middleware, of languages guaranteed to provide immediate and painless reuse. If you’re not working with a UNIX variant or some flavor of Windows your products are dinosaurs sure to fail in the marketplace.
In the olden days assembly programmers could count T-states to predict execution time. RAM and ROM needs were absolutely clear. The move to C did obscure this somewhat, but with a little experience developers could reasonably estimate real-time performance and memory requirements. That’s not possible anymore when we bundle huge chunks of middleware into our products.
Management finds it hard to resist the promises of vendors, who earnestly show how just one more layer will insure the product will outlive capricious decisions made by other vendors, of how converting to a popular API will let us replace expensive embedded experts with dollar-an-hour Visual C++ programmers.
Yet the costs, in terms of memory and CPU cycles, never seem to enter these discussions. It’s not till the system has grown into a bloated morass of code running at a crawl, that the realities become apparent. At that point there’s usually no recovery other than a complete recode. Expensive? You bet.
Middleware, high-end languages, and complex OSes are awesome tools and capabilities that many embedded systems absolutely need. They’re rarely the panaceas promised by anxious vendors. We must demand quantitative overhead specifications in place of the all-too-often vague promises used to sell this stuff today.
Real time resource-constrained embedded systems are still the norm, and still fundamentally differ philosophically from PCs. Yet too many vendors seem to view the embedded space as an extension of the desktop market.
Jack G. Ganssle is a lecturer and consultant on embedded developmentissues. He conducts seminars on embedded systems and helps companies with theirembedded challenges. Contact him at
Correct me if I'm off here. Basically, middleware should prevent one from having to re-invent the wheel. For example: Suppose “YourWare” needs to communicate to “BobsWare” and to “JuansWare”. Yet BobsWare and JuansWare do the same thing… they just have different communication interfaces. Sort of like speaking English versus Spanish. Middleware should make the translation from Yourware to BobsWare and JuansWare.
If “we” would try to “Standardize” communication interfaces… we might not have so many problems. I vote for more standardized communication interfaces, rather than each company making its own interface. Oh crap, we can't do that. That would make too much sense.
– Steve King
Not only that, (and here I'll say it for you) but couple these thoughts with the fact that most firmware is coded by amateurs who are not given enough guidance on the art of programming (middleware coders included) and you got a recipe for disaster. Being, or starting to become, a programmer is a far cry different than being a mere
coder. It takes many, many years to achieve.
– Dave May
“Yet too many vendors seem to view the embedded space as an extension of the desktop market.”
Let's face it, that's where the glitz is, even though regular readers of ESD know that desktops represent only a tiny fraction of the processors sold every year.
Why does Microsoft want the embedded market? Because there's no room to grow in the PC market. Why does anyone else want the embedded market? Because Microsoft does. (Okay, I'm engaging in a bit of hyperbole. There are good vendors with good products in the embedded space already, but their ad budgets aren't very big.)
As embedded developers we shape our solutions to fit the problem. When you already have a solution, you need to shape the problem to fit the tool. Microsoft already has the solution; they don't want to spend millions more to make another one for us. Unless you folks want to pay $100 a pop for a license.
– Dave Hinerman
This is so true. Everyone madly chases after technology with no thought for practical considerations. Nobody wants to say 'well, *why* do we need to be prepared to integrate with every concievable future developement…*why* do we need more layers of abstraction? Abstraction is not always good.
– Dan Winslow
In the old days, when at school students start off using “vi” and program in 'C' or assembler.
They programmed closer to low level system calls & h/w and eventually got a feel how a embedded system worked.
Today a student at school is programming with a pull down menu's, large middle layers of abstracted api's and which produces bloated s/w and requires Ghz and GigaBytes of resources to produce a “hello world” program. The student feels they accomplised a lot so quickly compared to someone doing the same in “Vi' + 'C'.
The end result is that we now have fewer young engineers working at low level s/w & h/w.
I like productivity tools which does not try to restrict my design or creativity.
It's like trying to put some extra seats and flat beds in fighter planes, like in commercial aircraft. Firmware/embedded software – like fighter planes – are very specialised things and are not for everybody to try their hands on.
– Srikanth Varma
There are many factors contributing to the current status of the embedded market. What seems to be the most obvious to me is the lack of engineering common sense. And it starts from the university, where students are taught various things but not to use their brains. There is not course in the curricula teaching common sense unless you are lucky to get a good lecturer with commercial experience to disseminate the common sense.
Then there is this: it takes much longer to train and embedded software engineer than any other software engineer. Anybody can start programming a PC these days, but in order to do embedded programming you need to know hardware as well as software, whereas a PC programmer can start using Visual Basic “for dummies” and crop up a program in no time at all.
If I were to write a piece of PC software I'd never get it done because I would like to know every bit of detail of how a class is implemented, how does it interact with the rest of the modules until I create a whole dependency tree so that I can have a thorough understanding of the libraries I'm using. In the meantime, a PC programmer would have developed the program and had a holiday in Ibiza. This just shows the level of detail required when one does embedded programming
– Romeo Ghiriti
Your article has hit upon a weakness of the industry trade journals.
I find good technical information on some micro aspect of a project relatively good and easy to find. How to effectivly use OO concepts, evaluating the RTOS capability of an OS, or a useful C/C++ programming tip.
What I find much harder to grasp and find are good systems design articles that bring together hardware and software.
I'd like to see some good articles on how one should design a product to meet specifications using the minimum resources possible. In other words, build the product the cheapest way you can get away with and still meet the requirements. Size all appropriate resouces – ROM, RAM, I/O, CPU. Decide on algorithms, features needed in O/S, development platform and interfaces. And naturally do this both ways, iteratively. I'd like to see some good case studies illustrating successful projects, giving me good acts to follow.
– Tiger Joe Sallmen
I agree with everything I've read so far, but I have to admit it's from a distance. I'm in Automotive in Europe, where we are spending considerable effort (650 engineers, equivalent to 175 full-time staff) on creating a common platform (specification, conformance, integration…), including what could be considered middleware, for use across the whole industry (see www.autosar.org for details).
Now I admit that automotive is not the largest market, but it is significant. Maybe other markets could take a leaf out of our book, form a consortium, and specify the OS & middleware that's actually needed for that market, then allow anyone – open source? – to implement that specification.
– Paul Tiplady
Most of the time the management argees with the Engineering's request for a complete rewrite if the management have not much technical background in the current system. Secondly, who recommend it. If they are those from PC world with extensive C++ background or they are just hired on board by the new manager. They are likely recommend to rewrite it. It is just my experience
– Tom P
Response to Paul Tiplady:
I'm in the Aerospace Industry… and all of our SW tools MUST BE “FAA certified” right down to the compilers we use (for commerical) — military certification is a bit more lax. Maybe… just maybe if “we” set some standards we could standardize and certify middleware components.
Open Source sounds nice, but there is really no “quality control” on how things interconnect and its standization.
– Steve King
Abstraction – one of the essential concepts all engineers work with. I bet that all embedded engineers and product builders will have enough understanding of the system only near to the age of their retirement (I am including all the learning they have to pass through to get into the cutting edge technology market these days).
Abstration cannot be ruled out. Abstraction is one of the things that we thrive and survive on.
When I order a chip for USB PHY from the vendor, I do not care about the implementation details – I just reuse it to build my product with the standard interfaces it provides. Of course – provided it satisfies all our specifications on the performance requirements.
What makes us think that abstraction is a bad idea then? I can launch my products faster than ever now.
What our industry is lacking is PROPER guidance to engineers to accept a level of abstraction based on the requirements of the system at hand.
NO TWO EMBEDDED SYSTEMS ARE ALIKE. All embedded systems have their own persona and their very own performance requirements.
Hence, middleware that is chosen and OS (OR NO-OS) decisions should completely be driven from requirements.
Ask yourself – Do I need this component to be future extensible? In that case – how much extensible? And then choose the appropriate middleware. In all of my experience now, I believe that the choosing of middleware and technology should and MUST be driven by requirements.
The MANAGEMENT as opposed to ENGINEERS should make a decision of what is required in the final product.
Unfortunately, today many companies seem to have managers having technical insight into products and take the right decisions on cost/performance/extensibility tradeoffs.
Middleware and abstraction are required – they come with a trade off and all that we need to decide is – Is this tradeoff acceptable?
It just needs the engineers and managers to sit and talk together spiced with some common sense. That's all to it.
– Saravanan T S
“Yet too many vendors seem to view the embedded space as an extension of the desktop market.”
I recently built a prototype device using Embedded XP and ran across a “feature” that tells me that MS has no idea of what “embedded” means (nor does my management that insists that we use the logical successor to DOS; there is nothing in this device that requires DOS compatibility).
On the very day I was planning to ship the device to the customer, I powered it up just to make sure everything was working… the device wouldn't come up.
I plugged in a monitor (and had to reboot since the VGA driver appears to disable the video circuit if no monitor is connected at boot time) and instead of my application running, there was a prompt informing that the password was going to expire and asking if I'd like to change it now as if the change would stick with the EWF running.
There's no setting for this. There is a work-around.
XPe is too complicated for most embedded apps and the docs (that exist) are terrible. Most engineers don't have the time to surf the web for hours looking to see if someone has posted some little tidbit.
I sure miss the days when I used pSOS with it's small number of functions that were well-documented and easily understood.
An earlier XPe product I worked on would have never gotten off the ground if we hadn't retained the services of one of the few people in the world who knows how NT works from the inside.
– Robert T
I think the time has come for experts like Ganssle to redefine embedded systems, embedded software and firmware. Abstraction, middleware, Linux hype..there's just too much contamination and confusion.
– Srikanth Varma
I escaped from Microsoft/PC coding hell about 10 yrs ago—one of my primary reasons to move to embedded world was I hated the bloated OS/middleware/microsoft out-of-the-HW-nitty-gritty mind-set. Seems as though my jump to real-time embedded as a means of escape will turn out to be futile–but, hey, it was great while it lasted…
– paul calvert
The answer is right here. POSIX.
If the OS vendors support the POSIX, I see no reason for Abstractions. Atleast, the majority of the middleware goes into OS abstraction.
– RadhaKumar Pulyala