Advertisement

The demise of the embedded generalist

November 02, 2004

Jack Ganssle-November 02, 2004

Click here for reader response to this article

What's your system's clock rate? About a third of the firmware developers I poll can't answer this question. Yet in my twisted EE way of thinking the clock rate is a basic aspect of a system. Firmware people use that figure to program timers, serial channels and PWMs. Sometimes we're required to select a clock frequency as part of the setup procedure for our debugging tools. And it's tough to do performance estimations without some idea of the system's speed.

But the shifting landscape of embedded systems development has changed everything. The olden days of 4k programs is largely history. Reiner Hartenstein, in the April 2004 Proceedings of the First Conference on Computing Frontiers claims that embedded software doubles in size every ten months, and will reach 90% of all software being written by 2010. Other numbers suggest that, today, firmware consumes 70% of the project's development time. It's clear that firmware is the foundation upon which civilization rests.

Developers are changing, too. In my firmware seminars I'll often ask how many attendees are EEs. A few years ago more than 50% answered "aye." Last week, for the first time, not a single hand shot up. CS and CE majors are slowly crowding out the EEs in software development.

We're specializing. That's often a sign of a healthy, maturing market segment. Networking experts don't dirty their hands with device drivers so just don't need to know the system's clock rate. Application level developers never delve beneath a high-level API, and might not even know what CPU is used. Some signal processing experts are more mathematicians than programmers and work at a very high level of abstraction indeed.

In Controlling Software Projects, Tom DeMarco urged software developers to embrace specialization. Some people are terrible coders but great testers. Some fumble in debugging while others zap bugs lightning-fast. Isn't it silly that each engineer analyzes requirements, designs, codes, inspects, debugs, tests and documents — and then provides tech support? How many of us are really experts at so many diverse specialties?

Doctors have an arguably much less creative profession than we embedded people, yet you'd never go to a GP when confronted with a diagnosis of cancer or heart disease. Specialties abound: cardiologist, podiatrists, gynecologists, sports injury practitioners, dermatologists, ophthalmologists, and plastic surgeons are just the tip of the iceberg. Consider the vast array of docs trained to work on, in, or around your brain.

The field of medicine is so huge that only by specializing can a doctor know enough about one single area to provide good care.

The embedded world is nearly as big: networking (TCP/IP, I2C, CAN, Bluetooth, Zigbee, Wi-fi and a hundred others), network management (SNMP and more), data acquisition, signal processing, C, C++, Java, any of a hundred different assembly languages, motor control, electromagnetics, PCB design, Verilog, ASIC design, and more. Each demands a high level of expertise that can take years to acquire. Any one could consume an entire career.

Yet in the embedded world resumes are expected to be acronym soup; we're specialists who have to be experts at everything. Realistic? Probably not.

IBM's Chief Programmer Teams of the '60s and '70s tried unsuccessfully to divide a team into a group of developers each practicing one specialty. Developers rebelled and the practice soon lost cachet. We like doing it all. I rather miss the early embedded days. It was a kick to design the hardware, tape out PCBs on great sheets of mylar (that was before CAD and autorouters), troubleshoot the boards while simultaneously writing assembly-language device drivers, and then implementing the rest of the firmware. Unlike the cardiologist who wearily sees the same handful of problems day in and day out, building an entire system means exercising a wide array of skills. There's never a chance to get bored.

But engineering is a business affair. Managers only care about the most efficient way to create embedded systems. Do we rely on broadly-educated developers doing all phases of the project or specialize?

Windows CE and Linux have already made big inroads into this industry. For managers these desktop OSes remove the need for highly skilled embedded developers. Millions of competent desktop programmers can write code for an "embedded" system layered on top of Windows or Linux.

I predict that the embedded world will bifurcate. Real-time, hardware-intensive work will be marginalized by the use of abstracting tools like UML, Java/C++, canned protocol stacks and more. Companies will hire a few great traditional embedded people for the low-level work, using lower-paid IT-like programmers to build the bulk of the code.

In my opinion, such specialization will save the company money, but sure sucks the fun out of projects. What do you think?

Jack G. Ganssle is a lecturer and consultant on embedded development issues. He's conducting a Better Firmware Faster seminar in Las Vegas on Dec 10. Contact him at jack@ganssle.com. His website is www.ganssle.com.

Reader Response


Jack, I have to entirely agree with you. Today - abstraction is the name of the game in software. I have been doing development right from the firmware to the GUI on the host application. But wait - today I can design systems just because I know what is involved in all these domains. I do not need to be an expert, but still I need to know what it takes to build a system out of all these discrete components, from the firmware to the application. Do whatever abstraction you can, still we need to know the internal details in order to design a flexible and efficient architecture. Specialists can't manage to do that!

So we need a few system architects who are experts in all fields and core embedded engineers specialised in their specific fields and then those IT programmers for rest of the application development. The best team a manager can have today is a mix of today's specialists and yesterday's generalists.

- Saravanan T S


Yes you are surely right when you say doing all that part of the product, from spec through pcb to code to debug and deliver was great fun.Specializes really takes out the fun out of the product developement. But thats what the market needs and thats what the manager needs.

- James


It only "sucks the fun out" if you want to do the same thing for your entire career, always relying on the same skillset.

I started out as an EE, designing hardware a decade and a half ago. By the time I interviewed with my current employer six years ago, I told the first interviewer that I wasn't interested unless he could guarantee I would spend at least 50% of my time doing software development because that's where my interests were leading me. We both decided that the Software department could make better use of me than the EE dept. My primary responsibility became writing software interfaces to the hardware that was being designed: I would have been the guy in your article who abstracted the hardware away. A few months into that job and my interest in software development had increased to the point that I returned to school part time to get an MS in Software Engineering.

Working for a medical device manufacturer, I have some exposure to biotech and I'm becoming more interested in the intersection between biotechnology and software. Perhaps that will lead me in a different direction in the coming years, who knows.

We do need to specialize in order to build complex systems efficiently, but there's no reason that anyone has to hold on to the same specialization for an entire career. Life is short; enjoy the variety that's out there.

- L Walker


I would rather hire a "broadly educated" developer than a specialist to get the job done. Engineering in my opinion is a conglomeration of various skills applied in a cohesive manner to build a product. Therefore at some point there has to be somebody to guide the various skills to get the product built. I believe a broadly educated developer is better able to bring these skills together than a specialist who has to be guided by somebody with broader skills anyways.

- Siddhartha Ray


Being an EE I have foreseen the trend and learned about Software Engineering at times when embedded software was a fringe activity of the electronics parts industry.

Today advanced software production methods exist for embedded devices but have been ignored and hindered by older business models.

For their Personal Computer, people are used to aquire software as a separate item while vendors of soda machines or cars include the software in the end product price. Ok, PC software changes the behavior of the machine, but why not have a soda machine serve as MP3 jukebox, Internet access point, or whatever?

All cost-conscious industries force the embedded software to be invisible in pricing. While the car may be made in Germany, its software often comes from other parts of the world.

With advanced Hardware Abstraction Layers it will become possible, even in the hardest of hard-real time, to separate application programming from the bits and bytes of the electronic hardware.

In the early 90s we did 32-bit systems development with 3 hardware and 3 software guys. Today there are still the 3 hardware guys, but rather 200 software guys, plus a few dozens managers. This way of creating embedded software will not prevail over time much like we do not raise our own cattle any more.

- Bernhard Kockoth


"IT-like" programmers are generic enough that the work can be outsourced, even off-shored. So remote development team management will be a skill that the few great embedded guys will eventually need to have.

- Doug Arnold


It is interesting to hear about spcialization in the field. I have worked at places where everyone was specialized. I did video and video controllers, others did arithmetic processing (mostly firmware), and a larger group did software. At another place, I did PCI bridge boards, ethernet, video, and a Celeron/chipset CPCI board, and other people wrote the software. I did help, because some required the EE mind to read the data sheets. I have also contracted me out to design data acqusition boards, and wrote the original PC software in C to test them. I tried to learn and use C++, and JAVA, but haven't succeeded (YET). I also have some fundamental VHDL in me. I recently took the path of licensure to be more of a generalist, and respected as one.

I find that I am asked to do FPGA, microcontroller hardware, AC power, firmware, software, interconnects, USB, RS232, RF, and compliance engineering and testing, etc. Generalization is what I do. I don't know how specialists can do it.

- Douglas Datwyler, PE


I partially agree with some of the comments in here. I see specialization already thru off the shelf middleware (e.g. residential gateway and setop box), commercial OSes (e.g. vxworks, linux), and heavy use of class libraries (e.g. J2ME). The higher you move up the chain the more it becomes an issue of specializing in knowing how to use some particular programming API.

But you still need somebody to integrate everything and debug it. For embedded devices (especially with custom hardware) that guy will most likely be an embedded guy. Why? Because embedded guys tend to be better at debugging because they understand what's going on under the hood. For example, your java guys will be dead in the water if the problem occurs at a lower layer. The best embedded guys are excellent at debugging which is not easy to teach.

I agree the end result will be a reduction in the need for a lot of developers. On the other hand eventually problems will occur due to lack of differentiation. If every box uses the same middleware package it becomes harder to differentiate using features. Then you end up with a commodity market. Commodity markets eventually lead to dominance by a couple big players who eventually stop innovating because they don't need to anymore. This leads to startups which come up with some new technology or new strategy to obsolete the dominate one guys choose to specialize in.

So from a career management point of view I would never encourage specialization. In the high tech world everything you know becomes obsolete within 5 yrs. History has shown that specialization is the kiss of death for engineers. Just ask the guys who used to specialize in X-windows, C++, Ada, Cobol, Fortrain, DOCSIS software for cable modems, SNMP, vxWorks, etc.

- Earl


I concur with your evaluation of the trend. I think there are two specific areas where the trend won't hold: system architecture (as mentioned by another reader) and small companies. In an engineering group of 6 (1 HW, 1 FW, 4 SW), the software group is CONSUMED with keeping up with the latest incompatibilities in our PC software provided by Microsoft. That leaves a team of two to do all the hardware/firmware development, which means spreading ourselves among analog and digital electronics, PCB layout, FPGA design, assembly language programming, RTOSes, device drivers, networking, and embedded applications.

I think this remains critical because the bulk of the innovation in the market comes from the many thousands of small companies, rather than from the few dozen big ones. Small companies also make the tools the big companies use -- development tools, instrumentation, etc.

So although the market for generalists may not be expanding along with the cell phone or PDA booms, it will (I hope!) always be there.

- Greg Nelson


Another side-effect of using generalists to build software (embedded or otherwise) causes a problem I call pragmatic programming practices. Pragmatic programmers fiddle with broken code until they get it to "work" without uncovering the underlying root cause of the problem. Such practices provide a very brittle foundation on which to base a product. PPP is all the more scary since many of these embedded programs are now part of the vehicles on which we depend for safety!

- Paul Wilt


I am fortunate that I work at a company that is sized for both engineering colaboration and the full range of product development involvement. This is the type of environment that I enjoy and hope to be able to continue working in. My degree is BSET with a minor in CS. This give me the advantage of being very applied in my electronics and more than on-the-job training for my programming. We have about 10 software design engineers. I am 1 of only 3 that also do hardware design. This is in an environment with no embedded PC's, all 8 and 16-bit processors and only our largest products make use of a commercial RTOS.

With increasing capabilities in smaller/cheaper processors and the wide spread use of larger OSes, I am not surprised at the trend for specialization. By my definition the "embedded" market is shrinking while the "specialized PC" market is exploding.

- Joel Morton


I think that smaller companies and startups are the turf of generalists. When I work for a company with under 20 people, there is no luxury of having a specialist for every function. As an EE, I've done my own prototypes, purchasing, schematics, some PCB layout, even production setup. My background includes data acquistion, embedded software, real-time video, FPGAs, motor control and a bunch of other stuff. Plus I'm inquisitive enough to stick my nose into anything (hence the occasional burn marks on my nose). Sure, I've had to write off technology areas that take too much specialty to really understand, but I think I'm pretty valuable in general.

I think where specialists come from is that when you get out of school, your first job at a major company can give you some depth. It is easy to follow that path since you now have some intrinsic knowledge value and keep going deeper. The next thing you know, you know everything about one thing. As someone said here, that sucks all the fun out of engineering to do the same thing all the time. It probably sets you up to be obsolete if that area goes away when you aren't looking.

- Steve Nordhauser


I have to agree that it is sad. It's progress, but it's sad. The days of doing it all yourself are just history. As I was reading your article, it made me think of Henry Ford, and the assembly line. Slowly but surely, the majority of folks working on techy projects will be just like assembly line workers. Each becoming highly specialized in one small aspect of the overall project; without ever gaining the overall picture.

This is great for business. For each detail of the project someone will have been assigned the magnifying glass to develop and debug it. If you want to put a positive spin on this, you could say that its great for developers too because it allows them to really get to know the problem (a luxury we seldom get nowadays, because as soon as it works we need to move on to something else). If you want to put a negative spin on it, you could say that it pigeon-holes developers never allowing them to get the whole picture; hence, making it difficult for them to move into other areas.

Personally, I have to admit that I would love to have time to focus on one thing long enough to really get to know it intimately; however, I also have to admit that I truely enjoy the ever-changing aspects of day-to-day work and the challenges it brings.

As usual, what is better for business, isn't necessarily better for the individual. Unfortunately, business usually wins out, so its in our own best interest to adjust and make the best of it.

- Ed Sutter


I think that you're confusing two different trends of specialization. Here you only talk about specialization in a phase of SW development. I haven't seen much of that.

However, in the last 20 years, I've certainly seen SW engineers focusing on problem domains, as well as employers desiring employees who specialize in those domains.

- Anthony Wong


An aeronautical engineer who quit working at Boeing to go and work as a software developer once explained to me the reason he quit was due to specialization; Boeing engineers are assigned to a single part of an aircraft (or two) for the entire careers. Not much excitement there.

Often an engineer's value rests on their ability to remain diversified, and to not become too specialized in an area that could one day be extinct. While it may be difficult to extract oneself from the path to specialization, it's what engineers ought to do best.

- Matt Staben


The middleware and the OS make it possible for the application specialists to concentrate on application development in a backward compatible fashion while providing new fuctionality.

However, it is fact of the day that newer HW interconnect standards, CPUs & Instruction sets, newer platfors are still evolving. Cases in point: PCI Express, Hypertransport, Bluetooth, Zigbee, UWB, USB2.0, PowerQuicc, ARM & ARC & many DSP processors. Add various protocol acceleration engines with their specialities. As the HW/Silicon developers pile on morefeatures, functionality & efficiencies, it takes the embedded generalists to bring those features into the realm of the application developers: the folks who do audio/ video/ networking & browsers - which is a lot of development & debug work. In my experience, one is constantly on the run for better tools in the embedded world. Imperfection & coexistence with one's tool vendor's development/debug cycle are hard realities of embedded development. 60% of embedded development is getting one's tools to perform as promised. All these job fuctions require generalists who had "seen it all" in their previous years.

With these observations, I disagree with Jack.

IMO, employers tend to have a split personality about specialization, depending on the time of hiring & time of employment. At the time of hiring, they want to hire the guy "who knows it all". In other words, specialization is a disqualification. When it is time to start working, managers & employers to pigeonhole their people into specialization. When the time comes to reinvent the company & layoff, those very specilzations encouraged by the company will be slashed off.

In these turbulent tech employment times, specialiation is a suicide pill. My view: Start with a broad general experience & specialize afterwards.

- Hari Tadepalli


Don't put all your eggs in one basket; else your ability to communicate with other engineering groups within your company will become ineffective. Frustration mounts and you risk having your engineering groups alienate each other. This can lead to a high turnover rate. Hire Multi-talented engineers. Cross Communication is important.

- Steve King


This explains why I get those blank stares when I ask that we observe the timing relationship on an o'scope of an I/O bit that we toggle in the firmware. I traditionally have used similar techniques to verify interrupt timing and latency. I never realized that only EEs thought like this. Oh well.

- Michael Weisner

Loading comments...