The article “Embedded Developers Do Some Sole Searching” describes how mechanical engineers at Adidas made a new type of smart shoe.
“Adidas engineers said creation of the software algorithms was among the most complex challenges, in part because the design team consisted of mechanical engineers who had scant programming experience. Thus the team went back to school, signing up for a course in programming Microchip's PIC16F88 MCU.”
“We didn't have much choice,” DiBenedetto said. “We flew to Chicago, took classes and started writing code.”
I'm sure this is a cool product, and I salute the engineers for creating something so complex. But at the risk of sounding elitist, the story sends goose bumps down my spine.
Programming is a specialized discipline. Many people can crank code—heck, even teenagers manage to infest the 'net with their evil creations. But it takes years—not weeks—of training to learn how to build great code.
I can use an X-Acto knife to perform minor surgery, say to remove a splinter. But only a fool would attempt an appendectomy without years of specialized training. An EE like you or me might build a treehouse for the kids in the backyard. But we need civil engineers to do a skyscraper's loads analysis.
One of the great ideas behind Dartmouth BASIC was to popularize programming. Practically anyone could create simple chunks of code that did useful things. We quickly learned, though, that most of that software was awful. It might work, sort of, but was typically neither reliable nor maintainable.
The PC changed all of the rules of computers. Everyone has one on their desks. We all have at least some sort of programming tools at hand, whether Excel macros or scripting agents. These tools bring great power to the average user and are a Good Thing.
But sometimes they create the perception that it's easy to create programs. A five-line macro or 100k LOC embedded system—what's the difference?
In my opinion one of the great problems in our business today is an undercurrent that we're not really professionals. “Let 'em crank some code,” the boss intones. “Don't have any CS people? Give those mechanical and chemical engineers a few days of classes and let them help.”
But with a few days, or even years, of training I can guarantee that I'd be an incompetent nuclear engineer. There's no way I'd be able to do the vibrational analysis of a spacecraft or design a refinery. Those people are experts.
As are we.
A great deal of skill goes into building extensible, maintainable, and reliable software. So maybe the reason there are so many buggy apps out there is that we're not using the highly trained professional software folks on these projects.
Some might say the Adidas people were only designing a simple shoe. It's probably not an application that will grow and evolve much. But an expert software person knows that nearly all software is organic. The programmer black-belt has seen a dozen jobs, all promised to be throw-away prototypes or simple apps never requiring support that turned into major product lines. A decade later the same code is still being improved and shipped, and has now been ported into many other products.
As long as the software community is viewed as a set of interchangeable cogs, fungible assets easily replaced by a co-op student or the secretarial staff, we'll be saddled with buggy, unmaintainable code.
I work with embedded systems developers all over the world and see projects of all stripes, from brilliant successes to those dismal failures that destroy jobs and companies. I've observed that despite years of training and experience at this work, many of us are in desperate need of even more book- and experiential-training in the field. This is not simple stuff.
What do you think? Are we experts who apply years of knowledge to solve problems in a clean and extensible fashion? Or can someone with a quick class or two replace us?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at . His website is .
You're correct: anyone who can think logically and express that logic on paper can write code. The problem is writing good code.
Embedded software has an additional problem: interfacing with hardware. I have friends that aren't EE or CS degree holders (but do have degrees in other engineering fields) who have written good programs. However, all of their programs are desktop (Windows, Linux, FreeBSD, etc) programs, where the UI is often stdin/stdout, and the program only had to interface with other programs. There was no worry of hardware interfacing, program memory usage, RAM usage, etc. Writing code that fits in 2kB and can do something–and do it well–is what separates embedded developers from desktop developers and hackers (used in the good sense).
A class or two can teach architecture of the processor, maybe touch on interfacing with hardware. I'm not sure there's a one-week class that can teach good embedded programming.
To me, programming is like driving: almost anyone can do it, but not everyone can do it well. You don't hire just anyone to drive in the Indianapolis 500, you hire an expert. The same should be said for embedded development.
– John Patrick
There's “programming” and there's “programming.” One means writing small programs that start, do a job, and end (like the desktop apps Mr. Patrick metions, or a single function in a larger system); the other means to design and implement a program that does a job (and keeps doing that job) with awareness of the environment in which it functions and the dirty tricks that environment may pull. I try to distinguish the two by calling them “programming” and “software development,” but my management doesn't seem to get the distinction.
I've seen programming compared to the process of building a machine. Well, maybe in school it is. But the nontrivial projects I've been involved with were more like building a factory – a whole bunch of machines that must work together in the presence of no input, too much input, wrong input, untrained operators, earthquakes, tornadoes… you get the picture.
In the factory analogy, the person who designs the machines isn't the same person who designs the factory. The skills don't scale upward. I think the same is true in software development. Somebody who is skilled at developing functions isn't automatically skilled at developing the entire system.
This isn't to say the person designing the functions can't become the system designer, but there's a lot of learning that has to happen along the way.
– Dave Hinerman
I suppose anyone who is helping embedded computing to become as ubiquitous as hoped, and doing it in the U.S. is not too terrible. I only object when it threatens my job… UNTIL, those newbies begin to take their “experience” to Lockheed or some medical equipment maker. You get the idea.
– David M. Tomer
Jack, I generally agree with you, but I think in this case your're a bit too concerned. I would be surprised if you went back to ANY of those engineers associated with the Adidas project and they told you that thye were ready to tackle an engine control system or the latest Space Shuttle control software. It looks as if the big problem was defining the parameters of the show operation.Now to bring some balance, many years ago, I was working for the hybrid products division of a large semi manufacturer. One of our parts required a test at 125C, and the mech dep't was tasked with making this machine. Finally, after many months of problems, the EE department was called in to write the software to run the machine. The computer used was a TRS-80 clone (can't remember the brand) but we had a cool mass-storage device – the Exabyte Stringy Floppy! (do you remember that one?) Anyway, the other guy and I managed to get the motor driver function & the main control loop working (in Basic & Z-80 assembler!). The real problem wasn't programming knowledge per se, as the ME on the project knew the fundamentasl, the problem was that the operational parameters weren't adequately defined. Once we got through that, it was fairly simple.However, I think that if the shoe works, then they did OK. Why not check back in 6 months & see if Adidas wants to integrate a GUI & web browser?
– Dave Telling
You're right again as usual. I teach a few graduate courses for the National Technological University (www.ntu.edu). One is on embedded systems software, another on software engineering. In my SE class, we have been discussing a similar topic about comparing software engineers (and other types of engineers) to doctors and lawyers. To use Mr. Dangerfield's phrase,”We don't get no respect.”
In addition to teaching, I run an embedded design department for a Long Island electronics company. My group gets the respect we deserve from senior management, but often times not from others in the company. I have heard that we are just code jockies, or we can afford to be sloppy because we can always fix the problem later unlike our hardware brethren who must get it right before the unit ships. The latter may be true to some extent, but it's not appreciated just the same.
We have also been discussing the topic of certification and licensing as a way of ensuring a minimum acceptable level of experience. Would this result in making our profession more respectable? Perhaps.
– Mitch Alexander
Embedded programming is different. I once spent three days helping an intern with VAX/Pascal experience write a simple program that would have taken me about 4 hours.
Nothing against the kid, but he had no conception of memory-mapped hardware interfaces, among other things. Hardware was “just there” and should have been supported by library routines. The concepts that would have enabled him to write those library routines just weren't there.
I've been doing this for over 20 years, now, and there are still a lot of things I'm not familiar enough with to feel comfortable doing … how can someone think taking a couple classes is sufficient? I suppose it also relates to the recent study that showed that incompetent people weren't competent enough to recognize their incompetence.
– Steve Wheeler
The problem with software is… it's called software. Many people think this is something easy to create and easy to modify.
Unfortunately, -especially for people who are not from the field- not so many understand the hidden complexity behind the word software.
Programming a 4 bits microcontroller is a feasible task by a novice kid, who already owns a much more complex calculator. But things become totally different when our software needs to controle a system based on a RISC/DSP Multi-core processors, running RTOS that support a telecommunication stack on one side, and an embedded OS for multimedia purposes on the other, all this with power consumption constraints and bottle neck avoidance!
There is a big difference between a program performing some simple calculation with printfs, and the one orchestrating a complex system.
Unfortunately, we will still face this misunderstanding, as long as any software is called software. Worse, today everything that is not Silicon or HDL is already called Embedded Software, from ROM located boot SW or HALs till final applications.
– Imad MIKAIEL
I am not sure about the highly specialised qualifications part, especially because I have seen far too many EEs who are not capable of understanding the system level perspective needed for writing and enhancing embedded code!
I must wholeheartedly agree that we need people who understand the idea of embedded software and not just overnight operators who punch in lines on emulators and IDE to design a 'working' solution!
The end result of amateur programming in professional environment is that bug fixing leads to bugs elsewhere!
– Deepak Alse
Well I think a lot of us are leaping to conclusions without the details – namely the spec.
Should we presume this is was a difficult over trivial application?
Is it possible it's another sneaker with blinking LED(s)? Perhaps it's even something fancy like piezo stability arch control system (PID) – not exactly a challenging concept for an experienced ME or any talented system process engineer.
If we assume there were no embedded experts at the helm perhaps a 4 bit state machine and a few dozen lines of pseudo code would have fit the application. Perhaps, along with training, our friends at Microchip provided more support than we'd need as their value added to the sale? After all they are great at that. 😉
We just don't know so it seems a little early to comment without specifics. Surely we professionals, along with our embedded experience and sensibilities, know to ask the right questions before critiquing, quoting, or allocating resources to the job… yes yes?
– Wm. Groves
Can anyone do it?
Yes & no.
The Civil engineer that does skyscrapers is measured. His stamp.
Other experts as well are measured.
The software engineering industry needs a measure. A public measure.
Can anyone build a toaster? yes, but only UL (or CSA) certified are accepted by the public.
So a does software meet a standard? There are specific standards for certain types of software, like aviation.
A more general standard for the public is needed.
Could this be done? There is going to be lots of pressure not to implement general standards by large corporations (like MS).
– Tim Flynn
As for the poll.
It doesn't matter. As long as you can code to the standard.
headline : Anyone can do it
– Tim Flynn
Without a doubt, Jack's point that embedded software engineers are skilled, valuable contributors is right on. Can those not trained in software engineering or embedded systems write code and ship product? The Adidas case shows they can.
Like any skill, those with training and experience will produce a better solution. “Better” might be defined as quicker, cheaper, more robust or more maintainable in the future. It's safe to bet that if Adidas would have hired an experienced embedded systems programmer the product would have shipped faster and possibly had lower direct material costs due to cheaper electronics. Would that have been the correct cost/benefit tradeoff for that product? Who knows. If the Adidas product is successful and spawns a full product line, the cost/benefit ratio will be important to understand going forward.
As a mechanical engineer with a master's degree in control systems, I must write embedded software to practice my art. To that end, I've taken four semesters of formal CS training (one in data structures and algorithms, two in computer architecture, and one in operating systems). Does this qualify me to write embedded software? Given that my code has generated no field issues in millions of units of product shipped over six years of work, I'd like to think that I'm doing OK.
I've seen a lot of control algorithms implemented by CS types that have no formal training in control systems. Should that be allowed? The fact of the matter is most of these control loops were low performance and worked great. When the problem got harder due to performance or resource constraints, I was called in due to my more specialized skills. Similarly, I try to steer clear of systems programming issues, and get support from “real” embedded programmers when avoidance doesn't work.
I believe that the cost of entry for embedded systems programming should be like that of any of the other engineering disciplines. If you have the skills to do the job adequately, great, go do the job. Someone with more skill should be able to provide a better solution. For each problem, management needs to look at the cost/benefit ratio of using a specialist to decide the best course of action. The one caveat is for applications with serious safety implications. In that case, someone with the proper credentials such as a P.E. in the proper discipline needs to sign off on the final design.
– Doug Harriman
Did the code work? Did Adidas make a profit? Nothing else mattered. If so, they'll simply do it again. Welcome to the new paradigm.
I come from an embedded hardware design background, but seem to have drifted into software by accident over the past ten years. I seem to be stuck there now!
Back in the “good old days”, I designed a variety of amazing pieces of hardware kit, often on my own, and I sometimes even consulted the softies on the way!
The softies understood memory capacity, but not much else. When asked “how much?”, they always answered “As much as possible”, and in reponse to “how fast?”, they always said “as fast as possible”.
In the many years I was a hardware bod, it was always a constant frustration that many (most?) of the supposed embedded softies glazed over when they encountered the dreaded
My experience of these pseudo-embedded softies is that they would typically hammer away at the UART code until a byte popped out of the transmitter, and then hurriedly bury the source code in case it broke. Never mind that they were programming the baud rate generator with the receiver enabled, they just threw values at random control registers until it “worked”. I have seen this time and time again.
In 24 years on the circuit, I have never managed to persuade any embedded softy that actually reading a programmer's guide to some peripheral chip is actually possible, let alone a good idea.
But then I guess that is how I ended up where I am today (penniless, actually). No softy could possibly be trusted to program the UART sensibly, so I, the Hardware engineer, took over the responsibility.
Another relevant anecdote.
A year or so ago, a client asked me to vet 150 CVs, in order to source a graduate embedded software engineer. No problem, I thought.
Here in the UK, we have many very old, highly regarded universities. However, the government, in its wisdom, created dozens more “universities” by re-labelling some of the lesser educational establishments. We are therefore awash with supposedly university educated scholars, who are (in my humble opinion) complete bozos.
Of the CVs I saw, most made absolutely no mention of embedded subjects. They were full of “Access” this, “Excel” that, “Website” the other.
About half went into the bin because the standard of written English was so appalling. A useful, but arbitrary way to slim down the pile.
Many more went the same way because of the complete lack of relevance to my customer's subject.
Of the 150 applicants, only one was recommended for interview.
Embedded electronics (or software at any rate) doesn't seem to be taught very much these days.
– Chris Brown
This is a recurring issue. I always end up referring other people –including myself– to Dijkstra's article “How do we tell truths that might hurt?” (http://www.cs.utexas.edu/users/EWD/transcriptions/EWD04xx/EWD498.html), written back in the days when mostly scientists and engineers had access to programable machines. It's amazing how little things have changed and how many of that 'truths' still apply. Perhaps they are immutable laws of the universe. As you state, computer accessibility has only made things worst, acting like an amplifier for our bad habits.
Will we one day, finally, *learn to learn* from our mistakes? Still, to err is human but to really screw up you need a computer… and each day there are just more of them –for the good or the bad.
– Pablo Bleyer
I'm with you all the way on the general principle of embedded software being a non-trivial task. But there are exceptions, and I think the shoe is one of them. I just don't want to have to draw the line, because drawing lines is not my specialism…
Also, I'm facinated by the poll. I wonder if it reflects the qualifications of your respondents (it does for me!)?
– Paul Tiplady
When I came on this company I got a job in an IR&D group and was given the task of writing the code that talked to the hardware (FPGAs) and to the overall system controller. I had just finnished college and the only programming course I ever took was in Pascal. All my “experience” (class projects) was in writing VHDL and assembly code. The code I wrote worked and the system was delivered to the field testers. Now a year and a half later we are developing version 2 and I am having to rewrite everything because it hurts me to look at how bad it all was.And I know I still have much more to learn.
I like the factory analogy.
– Chris Michael
Yes and no. I don't mind a handful of engineers from another discipline writing code for my shoes. Exactly how much can go wrong go wrong go wrong go wrong with a shoe?
OTOH, the guys doing flight controls in my airliner have more experience and would probably be lost (for a while) on the mechanics of shoe design.
BTW, who designed the electronics around the Adidas PIC? Was it a ChemE?
– Andy Kunz
Thanks for speaking up about this subject. When news of this “product” first appeared in this newsletter I had a hard time believing it was real. My comments to the editors reflected as much. I am glad to see there is someone with some sense of practicality on the staff of this publication. Anyway, when I read your comments I had to smile. I am a BSEE that has been in school for the last 6 years working on a Computer Science degree preparing for an embedded career. Even that does not prepare you for this field. My “library” on the subject has become such a problem that my wife has moved most of it to the garage. So yes, I have to wonder about how much these “embedded engineers” picked up in a week at some seminar. We need to be glad that the only thing they have released for production is a pair of shoes.
The world needs a voice of reason. Keep up the good work.
– Joe Hall
I at least partially dissagree with you on this one. I think the onus is on the project management to be sure that their team's capabilities match the project requirements. If you are building an air traffic control system, the team members must have a strong theoretical background. I would have had the Adidas team's software used as a prototype and handed it to proven programmers to rewrite but hey, it is only sneakers (probably $200) – how much could it cost the company if it is discovered that 6 hops on the left foot without stepping with the right will ruin them? OK, so an outside design review was warranted.
I am more impressed by someone's capabilities than their degree. There are times when the educational foundation is necessary but some of the best engineers I know were self-taught and held way back due to a lack of degrees. I watched a team of three guys – not a B.S. among them, building a bit slice stroke display that could draw 50K vectors/sec in the early 80's. It was the flagship of a 400 person company. They were unhappy but couldn't leave because they lacked paper. Happily, the new engineering director realized how valuable they were and doubled all their salaries and isolated them from all the company politics. I've met self-taught mechanical engineers. They tend to talk more to other fields – machinists, vendors, consultants – to verify their thinking. I think that is just as easy to fault the degreed people for their inability to realize when there are people who have more practical experience to assist. It depends on the person involved. I guess that is my most basic complaint with your assumptions. It is too wide a generalization since very creative people can accomplish many things and learn many things on their own.
– Steve Nordhauser
I usually make it a point to read your articles. Although I like some of your viewpoints, I am usually amazed with your “keep them out” attitude.
Lets look at what programming is really. Its just writing a set of instructions to a dumb electrical machine, telling that machine to do something useful. Nothing more, nothing less.
Lets also look at what people are creating these embedded systems for: really simple applications, like probably measuring the pressure your heel exerts, so that the sole can appropriately compensate according to some new concept.
I really don't think there's any reason to make such a big fuss if some non-ee writes code. (Although I understand that an EE-grad is losing a job here, but, programming is really not an ee domain. Its the domain of the computer engineers and scientists.)
And for heaven's sake! No body is born perfect. Do you think you wrote bug free code? Do you think reading and studying all those principles of software engineering and program design are going to help you write bug free code the first time through. I can assure you: most definitely NOT!
Writing software is like walking. As a kid, no matter how much you try and how much your parents tell you about the steps, you are bound to stumble on your first steps. But, as you continue to walk, over time, it comes to you so naturally that you don't even think about it.
programming is like walking. It wouldn't matter how much your read and study good programming principles. Till you are writing a trivial piece of software (like a “Hello, World!”) you are bound to make mistakes. And you are supposed to learn from those mistakes. And if you continue to write programs, and practice the art/craft/science of programming as you write your programs, there will come a time when programming will come to you naturally. Your code will be bug free, maintainable, and all that and more.
Computers have become too omnipresent. If we have to realize the dream of ubiquitous computing: computers everywhere and in everything, we will have to get over our prejudices of “non-ee should not write code”. Maybe we need to work really hard at helping the non-ee-s write better and more useful code.
I would like to congratulate the Adidas engineers: bravo guys, well done. Programming is really simple. Welcome to the new world. And please, don't be disheartened if you had bugs in your code. Its natural. Fix them and move on towards developing more and better programs.
– Mahersh R S
I am a big fan of bringing people up to speed fast. It is a benefit to any field when the doors are not closed. Besides, computers have a history of development from all disciplines, not just specialists. Of course, expert status will still speak for itself..
– Ed Dench
Well it seems everybody here is so jealous of their long years of study and work experience (which is a good thing, of course) to forget that often the best things are those made by passion, a true vocational predisposition, a deep understanding (by practical sperimentation and personal need) and, moreover, the attention to those little details that separate an artwork of technology from a marketing pressured job duty.
So, please, don't kill the entusiasm of newbies for embedded engineering.
– David G.P. Tamagni
I agree that in the past due to processor, memory and programming language constrains, a serious embedded system required the use of 100% experienced SC or EE graduates. Today when memory is cheep, processors are fast and embedded code can be written in C++, only a core of few serious embedded engineers is required in order to deal with stuff such as interface to hardware, hardware debug, task scheduling, concurrent access to resources and some other more complex staff. “Programmers” using desktop like programming style can do most of the code. It wouldn't be, as efficient but will probably fly. After all if your embedded Linux was well ported to your hardware, you are programming now for a Linux environment rather than for a hard real time environment, so you can use Linux programmers.
Jack, you've hit on my biggest frustration in embedded software development (for systems far more complex than a pair of sneakers): the primary reason for poor-quality embedded software is that most of it is written by people who have little or no formal training in software development. Most of them are EEs. While I'm sure there are EEs out there who are skilled in SW development, the reality is that it's not part of most EE curricula.
Hacking code until it “works” does not constitute software design and development. The former does not produce reliable, testable, and maintainable software. Anyone can hack code; SW development takes a trained person.
A related problem is that most SW managers also have no formal training in SW engineering.
– Jason Dougherty
I read one response from Chris Brown and I agree with him on a few points. Getting the right education and the right amount is important. Coming out of a college that really focused on hardware and software interfacing, I still was a babe in the woods compared to what technologies and established techniques that were already out there in the real world. Designing hardware to fullfill the customers requirments is easy. There are standards out there pointing out how the hardware should be placed. The true magic is the software technique used to make that hardware perform as it should.
– Dale Arnold
It's true that experience with something helps breed expertise/competence, but that is just a general rule. I've seen people with several years of experience who never develop a high level of competence because of certain limiting factors (I.Q., work ethic, attitude, etc.). I have also seen some really amazing newcomers who right bullet-proof code and are amazing problem solvers (incidentally, these guys are good at very many things, not just coding – probably an indication of generally high intelligence).
The differences between a not-so-competence-yet-experienced developer and a competent-yet-newcomer developer are usually evident in projects that are very challenging (in depth of complexity, not necessarily the size of the task). In