The demise of the embedded generalist -

The demise of the embedded generalist

Click here for reader response to this article

What's your system's clock rate? About a third of the firmware developers I poll can't answer this question. Yet in my twisted EE way of thinking the clock rate is a basic aspect of a system. Firmware people use that figure to program timers, serial channels and PWMs. Sometimes we're required to select a clock frequency as part of the setup procedure for our debugging tools. And it's tough to do performance estimations without some idea of the system's speed.

But the shifting landscape of embedded systems development has changed everything. The olden days of 4k programs is largely history. Reiner Hartenstein, in the April 2004 Proceedings of the First Conference on Computing Frontiers claims that embedded software doubles in size every ten months, and will reach 90% of all software being written by 2010. Other numbers suggest that, today, firmware consumes 70% of the project's development time. It's clear that firmware is the foundation upon which civilization rests.

Developers are changing, too. In my firmware seminars I'll often ask how many attendees are EEs. A few years ago more than 50% answered “aye.” Last week, for the first time, not a single hand shot up. CS and CE majors are slowly crowding out the EEs in software development.

We're specializing. That's often a sign of a healthy, maturing market segment. Networking experts don't dirty their hands with device drivers so just don't need to know the system's clock rate. Application level developers never delve beneath a high-level API, and might not even know what CPU is used. Some signal processing experts are more mathematicians than programmers and work at a very high level of abstraction indeed.

In Controlling Software Projects , Tom DeMarco urged software developers to embrace specialization. Some people are terrible coders but great testers. Some fumble in debugging while others zap bugs lightning-fast. Isn't it silly that each engineer analyzes requirements, designs, codes, inspects, debugs, tests and documents — and then provides tech support? How many of us are really experts at so many diverse specialties?

Doctors have an arguably much less creative profession than we embedded people, yet you'd never go to a GP when confronted with a diagnosis of cancer or heart disease. Specialties abound: cardiologist, podiatrists, gynecologists, sports injury practitioners, dermatologists, ophthalmologists, and plastic surgeons are just the tip of the iceberg. Consider the vast array of docs trained to work on, in, or around your brain.

The field of medicine is so huge that only by specializing can a doctor know enough about one single area to provide good care.

The embedded world is nearly as big: networking (TCP/IP, I2 C, CAN, Bluetooth, Zigbee, Wi-fi and a hundred others), network management (SNMP and more), data acquisition, signal processing, C, C++, Java, any of a hundred different assembly languages, motor control, electromagnetics, PCB design, Verilog, ASIC design, and more. Each demands a high level of expertise that can take years to acquire. Any one could consume an entire career.

Yet in the embedded world resumes are expected to be acronym soup; we're specialists who have to be experts at everything. Realistic? Probably not.

IBM's Chief Programmer Teams of the '60s and '70s tried unsuccessfully to divide a team into a group of developers each practicing one specialty. Developers rebelled and the practice soon lost cachet. We like doing it all. I rather miss the early embedded days. It was a kick to design the hardware, tape out PCBs on great sheets of mylar (that was before CAD and autorouters), troubleshoot the boards while simultaneously writing assembly-language device drivers, and then implementing the rest of the firmware. Unlike the cardiologist who wearily sees the same handful of problems day in and day out, building an entire system means exercising a wide array of skills. There's never a chance to get bored.

But engineering is a business affair. Managers only care about the most efficient way to create embedded systems. Do we rely on broadly-educated developers doing all phases of the project or specialize?

Windows CE and Linux have already made big inroads into this industry. For managers these desktop OSes remove the need for highly skilled embedded developers. Millions of competent desktop programmers can write code for an “embedded” system layered on top of Windows or Linux.

I predict that the embedded world will bifurcate. Real-time, hardware-intensive work will be marginalized by the use of abstracting tools like UML, Java/C++, canned protocol stacks and more. Companies will hire a few great traditional embedded people for the low-level work, using lower-paid IT-like programmers to build the bulk of the code.

In my opinion, such specialization will save the company money, but sure sucks the fun out of projects. What do you think?

Jack G. Ganssle is a lecturer and consultant on embedded development issues. He's conducting a Better Firmware Faster seminar in Las Vegas on Dec 10. Contact him at . His website is .

Reader Response

Jack, I have to entirely agree with you. Today – abstraction is the name of the game in software.I have been doing development right from the firmware to the GUI on the host application.But wait – today I can design systems just because I know what is involved in all these domains. I do not need to bean expert, but still I need to know what it takes to build a system out of all these discrete components, from thefirmware to the application. Do whatever abstraction you can, still we need to know the internal details in order todesign a flexible and efficient architecture. Specialists can't manage to do that!

So we need a few system architects who are experts in all fields and core embedded engineers specialised in theirspecific fields and then those IT programmers for rest of the application development. The best team a manager canhave today is a mix of today's specialists and yesterday's generalists.

– Saravanan T S

Yes you are surely right when you say doing all that part of the product, from spec through pcb to code to debug anddeliver was great fun.Specializes really takes out the fun out of the product developement. But thats what the marketneeds and thats what the manager needs.

– James

It only “sucks the fun out” if you want to do the same thing for your entire career, always relying on thesame skillset.

I started out as an EE, designing hardware a decade and a half ago. By the time I interviewed with my currentemployer six years ago, I told the first interviewer that I wasn't interested unless he could guarantee I would spendat least 50% of my time doing software development because that's where my interests were leading me. We both decidedthat the Software department could make better use of me than the EE dept. My primary responsibility became writingsoftware interfaces to the hardware that was being designed: I would have been the guy in your article who abstractedthe hardware away. A few months into that job and my interest in software development had increased to the point thatI returned to school part time to get an MS in Software Engineering.

Working for a medical device manufacturer, I have some exposure to biotech and I'm becoming more interested in theintersection between biotechnology and software. Perhaps that will lead me in a different direction in the coming years, who knows.

We do need to specialize in order to build complex systems efficiently, but there's no reason that anyone has to holdon to the same specialization for an entire career. Life is short; enjoy the variety that's out there.

– L Walker

I would rather hire a “broadly educated” developer than a specialist to get the job done. Engineering inmy opinion is a conglomeration of various skills applied in a cohesive manner to build a product. Therefore at somepoint there has to be somebody to guide the various skills to get the product built. I believe a broadly educateddeveloper is better able to bring these skills together than a specialist who has to be guided by somebody withbroader skills anyways.

– Siddhartha Ray

Being an EE I have foreseen the trend and learned about Software Engineering at times when embeddedsoftware was a fringe activity of the electronics parts industry.

Today advanced software production methods exist for embedded devices but have been ignored and hindered by olderbusiness models.

For their Personal Computer, people are used to aquire software as a separate item while vendors of soda machines orcars include the software in the end product price. Ok, PC software changes the behavior of the machine, but why nothave a soda machine serve as MP3 jukebox, Internet access point, or whatever?

All cost-conscious industries force the embedded software to be invisible in pricing. While the car may be made inGermany, its software often comes from other parts of the world.

With advanced Hardware Abstraction Layers it will become possible, even in the hardest of hard-real time, to separateapplication programming from the bits and bytes of the electronic hardware.

In the early 90s we did 32-bit systems development with 3 hardware and 3 software guys. Today there are still the 3hardware guys, but rather 200 software guys, plus a few dozens managers. This way of creating embedded software willnot prevail over time much like we do not raise our own cattle any more.

– Bernhard Kockoth

“IT-like” programmers are generic enough that the work can be outsourced, even off-shored. So remote developmentteam management will be a skill that the few great embedded guys will eventually need to have.

– Doug Arnold

It is interesting to hear about spcialization in the field. I have worked at places where everyone wasspecialized. I did video and video controllers, others did arithmetic processing (mostly firmware), and a larger group didsoftware. At another place, I did PCI bridge boards, ethernet, video, and a Celeron/chipset CPCI board, and other peoplewrote the software. I did help, because some required the EE mind to read the data sheets. I have also contracted me out todesign data acqusition boards, and wrote the original PC software in C to test them. I tried to learn and use C++, and JAVA,but haven't succeeded (YET). I also have some fundamental VHDL in me. I recently took the path of licensure to be more of ageneralist, and respected as one.

I find that I am asked to do FPGA, microcontroller hardware, AC power, firmware, software, interconnects, USB, RS232, RF, andcompliance engineering and testing, etc. Generalization is what I do. I don't know how specialists can do it.

– Douglas Datwyler, PE

I partially agree with some of thecomments in here. I see specialization alreadythru off the shelf middleware (e.g. residentialgateway and setop box), commercial OSes (e.g.vxworks, linux), and heavy use of classlibraries (e.g. J2ME). The higher youmove up the chain the more it becomesan issue of specializing in knowinghow to use some particular programmingAPI.

But you still need somebody to integrateeverything and debug it. For embeddeddevices (especially with custom hardware)that guy will most likely be an embeddedguy. Why? Because embedded guys tend tobe better at debugging because theyunderstand what's going on under thehood. For example, your java guys will bedead in the water if the problem occursat a lower layer. The best embedded guysare excellent at debugging which is noteasy to teach.

I agree the end result will be a reductionin the need for a lot of developers. On the otherhand eventually problems will occur due tolack of differentiation. If every box usesthe same middleware package it becomes harderto differentiate using features. Then youend up with a commodity market. Commoditymarkets eventually lead to dominance bya couple big players who eventually stopinnovating because they don't need to anymore.This leads to startups which come up withsome new technology or new strategy to obsoletethe dominate one guys choose to specialize in.

So from a career management point of view I wouldnever encourage specialization. In the hightech world everything you know becomes obsoletewithin 5 yrs. History has shown that specializationis the kiss of death for engineers. Just ask theguys who used to specialize in X-windows, C++,Ada, Cobol, Fortrain, DOCSIS software for cablemodems, SNMP, vxWorks, etc.

– Earl

I concur with your evaluation of the trend. I think there are two specific areas where the trend won't hold:system architecture (as mentioned by another reader) and small companies. In an engineering group of 6 (1 HW, 1 FW, 4 SW),the software group is CONSUMED with keeping up with the latest incompatibilities in our PC software provided by Microsoft. That leaves a team of two to do all the hardware/firmware development, which means spreading ourselves among analog anddigital electronics, PCB layout, FPGA design, assembly language programming, RTOSes, device drivers, networking, and embeddedapplications.

I think this remains critical because the bulk of the innovation in the market comes from the many thousands of smallcompanies, rather than from the few dozen big ones. Small companies also make the tools the big companies use — developmenttools, instrumentation, etc.

So although the market for generalists may not be expanding along with the cell phone or PDA booms, it will (I hope!) alwaysbe there.

– Greg Nelson

Another side-effect of using generalists to build software (embedded or otherwise) causes a problem I callpragmatic programming practices. Pragmatic programmers fiddle with broken code until they get it to “work” withoutuncovering the underlying root cause of the problem. Such practices provide a very brittle foundation on which to base aproduct. PPP is all the more scary since many of these embedded programs are now part of the vehicles on which we depend forsafety!

– Paul Wilt

I am fortunate that I work at a company that is sized for both engineering colaboration and the full range ofproduct development involvement. This is the type of environment that I enjoy and hope to be able to continue working in. My degree is BSET with a minor in CS. This give me the advantage of being very applied in my electronics and more thanon-the-job training for my programming. We have about 10 software design engineers. I am 1 of only 3 that also do hardwaredesign. This is in an environment with no embedded PC's, all 8 and 16-bit processors and only our largest products make useof a commercial RTOS.

With increasing capabilities in smaller/cheaper processors and the wide spread use of larger OSes, I am not surprised at thetrend for specialization. By my definition the “embedded” market is shrinking while the “specialized PC” market isexploding.

– Joel Morton

I think that smaller companies and startups are the turf of generalists. When I work for a company with under 20 people,there is no luxury of having a specialist for every function. As an EE, I've done my own prototypes, purchasing, schematics,some PCB layout, even production setup. My background includes data acquistion, embedded software, real-time video, FPGAs,motor control and a bunch of other stuff. Plus I'm inquisitive enough to stick my nose into anything (hence the occasionalburn marks on my nose). Sure, I've had to write off technology areas that take too much specialty to really understand, butI think I'm pretty valuable in general.

I think where specialists come from is that when you get out of school, your first job at a major company can give you somedepth. It is easy to follow that path since you now have some intrinsic knowledge value and keep going deeper. The nextthing you know, you know everything about one thing. As someone said here, that sucks all the fun out of engineering to dothe same thing all the time. It probably sets you up to be obsolete if that area goes away when you aren't looking.

– Steve Nordhauser

I have to agree that it is sad. It's progress, but it's sad. The days of doing it all yourself are just history. As I wasreading your article, it made me think of Henry Ford, and the assembly line. Slowly but surely, the majority of folksworking on techy projects will be just like assembly line workers. Each becoming highly specialized in one small aspect ofthe overall project; without ever gaining the overall picture.

This is great for business. For each detail of the project someone will have been assigned the magnifying glass to developand debug it. If you want to put a positive spin on this, you could say that its great for developers too because it allowsthem to really get to know the problem (a luxury we seldom get nowadays, because as soon as it works we need to move on tosomething else). If you want to put a negative spin on it, you could say that it pigeon-holes developers never allowing themto get the whole picture; hence, making it difficult for them to move into other areas.

Personally, I have to admit that I would love to have time to focus on one thing long enough to really get to know itintimately; however, I also have to admit that I truely enjoy the ever-changing aspects of day-to-day work and the challengesit brings.

As usual, what is better for business, isn't necessarily better for the individual. Unfortunately, business usually winsout, so its in our own best interest to adjust and make the best of it.

– Ed Sutter

I think that you're confusing two different trends of specialization. Here you only talk about specialization ina phase of SW development. I haven't seen much of that.

However, in the last 20 years, I've certainly seen SW engineers focusing on problem domains, as well as employers desiringemployees who specialize in those domains.

– Anthony Wong

An aeronautical engineer who quit working at Boeing to go and work as a software developer once explained to methe reason he quit was due to specialization; Boeing engineers are assigned to a single part of an aircraft (or two) for theentire careers. Not much excitement there.

Often an engineer's value rests on their ability to remain diversified, and to not become too specialized in an area thatcould one day be extinct. While it may be difficult to extract oneself from the path to specialization, it's what engineersought to do best.

– Matt Staben

The middleware and the OS make it possible for the application specialists to concentrate on applicationdevelopment in a backward compatible fashion while providing new fuctionality.

However, it is fact of the day that newer HW interconnect standards, CPUs & Instruction sets, newer platfors are stillevolving. Cases in point: PCI Express, Hypertransport, Bluetooth, Zigbee, UWB, USB2.0, PowerQuicc, ARM & ARC & many DSPprocessors. Add various protocol acceleration engines with their specialities. As the HW/Silicon developers pile onmorefeatures, functionality & efficiencies, it takes the embedded generalists to bring those features into the realm of theapplication developers: the folks who do audio/ video/ networking & browsers – which is a lot of development & debug work. Inmy experience, one is constantly on the run for better tools in the embedded world. Imperfection & coexistence with one'stool vendor's development/debug cycle are hard realities of embedded development. 60% of embedded development is gettingone's tools to perform as promised. All these job fuctions require generalists who had “seen it all” in their previous years.

With these observations, I disagree with Jack.

IMO, employers tend to have a split personality about specialization, depending on the time of hiring & time of employment.At the time of hiring, they want to hire the guy “who knows it all”. In other words, specialization is a disqualification.When it is time to start working, managers & employers to pigeonhole their people into specialization. When the time comes toreinvent the company & layoff, those very specilzations encouraged by the company will be slashed off.

In these turbulent tech employment times, specialiation is a suicide pill. My view: Start with a broad general experience &specialize afterwards.

– Hari Tadepalli

Don't put all your eggs in one basket; else your ability to communicate with other engineering groups within yourcompany will become ineffective. Frustration mounts and you risk having your engineering groups alienate each other. This canlead to a high turnover rate. Hire Multi-talented engineers. Cross Communication is important.

– Steve King

This explains why I get those blank stares when I ask that we observe the timing relationship on an o'scope of anI/O bit that we toggle in the firmware. I traditionally have used similar techniques to verify interrupt timing and latency. I never realized that only EEs thought like this. Oh well.

– Michael Weisner

1 thought on “The demise of the embedded generalist

  1. “At the end of the day, engineering basically means building something. Due to advances in technologies and power of abstraction, we can reuse pre-built components into our solution. That is what the business mostly wants to save time and money. So, as eng

    Log in to Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.