Informal observations -

Informal observations


For this, my last column of 2012, I want to share some observations of the state of the industry from my peripatetic wandering this year over four continents.I've presented my “Better Firmware Faster” seminar on-site to many hundreds of companies and thousands of developers. It was inspired by a couple of decades of work in the embedded tool business that took me into developers' labs all over the world. What I found then, and what is still true today, is that so many of us are facing the same problems: projects are late and systems get shipped seeded with bugs. In the '80s and '90s, engineers tried to deal with these realities by working harder and by superhuman debugging efforts. Today, the workload has become unrealistic; most projects are late the day they start and never catch up. The seminar is designed to show ways to avoid these problems, and I get a kick out of talking to the attendees to learn about their unique challenges.A central observation of the quality movement that revolutionized manufacturing is that products are both cheaper and delivered faster when quality is designed in from the beginning. In the software world this has morphed into a critical aphorism: a focus on fixing bugs will lead to neither on-time delivery nor quality code .That last clause should cause most of us to rethink our approach to building embedded systems (which is the focus of my seminar). But it hasn't. Most teams embrace the same heroics used for decades.But some things have changed.Code is getting better. At least new code; a lot of people are in maintenance on old systems which is invariably based on awful software.Although there are a ton of exceptions, I'm seeing more care going into structural issues: architecture and design. And, again with plenty of counterexamples, code is generally better crafted than a few years ago. Names make more sense, and engineers are careful to follow at least some stylistic issues. The data shows that this care translates into better products. The entire IT world fixes about 85% of the bugs in their code pre-shipment. Embedded products average a 95% fix rate. That's still too low, but at least our industry is ahead of the pack.But I'm not convinced it's improving fast enough to keep up with the increasing demands being made on firmware.I encounter teams that have wholeheartedly embraced one or another of the agile methods. There's a lot of talk about agile; it's a subject that always comes up, and XP and Scrum both get a lot of mind share. The truth, though, is that it's still pretty unusual to find groups that practice these approaches with any sort of discipline or rigor.Test-Driven Development has an increasing number of adherents, but I find very few teamsusing it; instead there's the occasional lone wolf using TDD.EEs are less common. Increasingly teams are forged from CS/CE people or EEs who have never strayed outside of the domain of firmware. Again, exceptions abound; the last company I visited this month had a large group that was nearly all EEs. That is so unusual I was quite taken aback.An increasing number of companies use at least some metrics, the most common being cyclometric complexity. Essentially none, though, use that to qualify their tests.Testing is getting worse, driven by crazy schedules and the increasing size of code bases.Teams continue to get bigger and individual developers complain more about being compartmentalized into narrow portions of a product. In many groups no one has any overall sense of how things work. It's not unusual to find hundreds of engineers–often in many different time zones–working on a single product.Legacy code is both the best and worst problem faced by many. It's good simply because legacy code is the stuff that typically generates most of a company's revenue. It's a disaster because so much is so awful. Because code bases keep getting bigger, and since the embedded world (at 41 years since the first microprocessor) is well into middle-age, the landfill of old code is huge.Many teams today work only on the fringes of the old software, making minor enhancements or bolting on new features. Few of these people are happy with what they are doing and most feel a weight of impending doom, expecting the old stuff to eventually implode.Big companies that buy lots of microprocessors rely more than ever on the semiconductor vendors to supply substantial portions of the code base. They have plenty of leverage and as a result it's sometimes hard to draw a line between the vendor's and the customer's engineers. Some of this is also driven by poor documentation of the parts, as well as insanely complex devices, like the SoCs in mobile phones, that are all but undocumentable.What hasn't changed is that engineers at small-to midsized companies are almost uniformly happy in their work. Some big outfits do an amazing job, though, of disenfranchising their people. And that's a darn shame.Happy holidays, all, and let's hope for a great 2013.Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges, and works as an expert witness on embedded issues. Contact him at. His website is .

5 thoughts on “Informal observations

  1. Regarding CS vs. EE programmers, I wonder if the current crop of CS guys are as paranoid about prototype HW as the ones I used to run into? Time was you could easily tell the CS guys because they'd always swear their first cut at code had zero errors, the

    Log in to Reply
  2. I enjoy bringing up new HW, especially when I have had a hand in specifying what the HW was going to do. I always assume there will be problems, but I also recognize my code might have a problem. One of the first things I get going is a simple CLI, which a

    Log in to Reply
  3. My experience has been otherwise. If there is a stability issue, the general assumption is that it's a software problem, and unless proven otherwise, EE's won't look into it. Perhaps the reason for that is that it is perceived that it is easier to fix soft

    Log in to Reply
  4. Whoa! I hadn't intended to get into the general issue of “stability” which sometimes takes a REAL systems approach to resolve. For example if the system has an analog loop the loop might have inadequate gain or phase margin; if it has a digital loop there

    Log in to Reply
  5. The last four times someone said the software was broken, it was really a hardware problem. Wrong clock; wrong resistor; bad contacts; dead processor. My degrees say CS; I've spent 34 years trying to make metal work. If it's not a team effort, with reco

    Log in to Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.