Firmware basics for the boss, part 2As the boss, you can get the best out of your engineers by helping them hone their craftthe key to your mutual success.
Firmware engineers often tell me they want to do the right things, such as employing careful design practices but are defeated by their bosses' ignorance of software issues. Sound familiar? If so, cut out last month's and this month's column and slip them under your boss's door. It will give the boss the tools he or she needs to be more successful.
A poll on Embedded.com suggests 85% of companies won't spend more than $1,000 on any but the most essential tools. Considering the $100k+ loaded cost of a single engineer, it's nuts to not spend a few grand on a tool that offers even a small productivity boost.
Like what? Lint, for one. Lint is a program that examines the source code and identifies suspicious areas. It's like a compiler's syntax checker on steroids. Only Lint is smart enough to watch variable and function usage across multiple files. Compilers can't do that. For a fraction of the cost, aggressive Lint usage picks out many problems before debugging starts. Lint all source files before doing code inspections.
Gimpel (www.gimpel.com) sells one for $239. It's up to you to buy it and to ensure your engineers use it on all new code. Lint is annoying at first, often initially zeroing in on constructs that are indeed fine. Don't let that quirk turn your people off. Tame it, and then reap great reductions in debugging times.
Debugging eats 50% of most projects' schedules. The average developer has a 5 to 10% error rate. Anything that trims that rate even a smidgen saves big bucks.
Make sure the developers aren't cheating their tools. Warning levels on compilers, for instance, should be set to the lowest possible level so all warnings are displayed. And then insist the team write warning-free code. It's astonishing how we ship firmware that spews warnings when compiled. The compiler, which understands the language's syntax far better than any of your people, is in effect shouting "Look here. Here! This is scary!" How can anyone ignore such a compelling danger sign?
Write warning-free code so that maintenance people in months or decades won't be baffled by the messages. "Is it supposed to do this? Or did I reinstall the compiler incorrectly? Which of these is important?" This means changing the way they write C. Use explicit casting. Parentheses when there's any doubt. These are all good programming practices anyway, with zero cost in engineering, execution speed, or code size. What's the downside?
Editors, compilers, linkers, and debuggers are essential and non-negotiable tools as it's impossible to do any development without these. Consider others. Complexity analyzers can yield tremendous insight into functions, identifying "bad code" early before the team wastes their time and spirits trying to beat the cruddy code into submission. See www.chris-lott.org/resources/cmetrics for a list of freebies. Bug tracking software helps identify problem areassee a list of resources at www.aptest.com/resources.html.
Most firmware developers are desperate for better debugging tools. Unhappily, the grand old days of in-circuit emulators are over. These tools provided deep insight into the intrinsically hard-to-probe embedded system. Their replacement, the background debug mode (BDM), offers far less capability. Have mercy on your folks and insist the hardware team dedicate a couple of spare parallel output bits just to the software people. They'll use these along with instrumented code for a myriad of debugging tasks, especially for hard-to-measure performance issues.
Your developersnot tools, not widgets, not componentsare your prime resource. As one wag noted, "my inventory walks out the door each night."
I've recommended several books in these two articles. Please, though, read Peopleware by DeMarco and Lister.1 It's a slender volume that you'll plow through in just a couple of enjoyable hours. Pursuing the elusive underpinnings of software productivity, for 10 years the authors conducted a "coding war" between volunteering companies.
The results? Well, at first the data was a scrambled mess. Nothing correlated. Teams that excelled on the projects (by any measure: speed, bug count, matching specs) were neither more highly paid nor more experienced than the losers. Crunching every parameter revealed the answerdevelopers imprisoned in noisy cubicles, those who had no defense against frequent interruptions, did poorly.
How poorly? The numbers are breathtaking. The best quartile was 300% more productive than the lowest 25%. Yet privacy was the only difference between the groups.
Think about itwould you like three times faster development?
It takes your developers 15 minutes, on average, to move from active perception of the office busyness to being totally and productively engaged in the cyberworld of coding. Yet a mere 11 minutes passes between interruptions for the average developer. Ever wonder why firmware costs so much? E-mail, the phone, people looking for coffee filters, and sometimes you, boss, all clamor for attention.
Sadly, most developers live in cubicles today, which are, as Dilbert so astutely noted, "anti-productivity pods." Next time you hire someone, peer into his cube occasionally. At first he's anxious to work hard, focus, and crank out a great product. He'll try to tune out the poor bloke in the next cube who's jabbering on the phone with his lawyer about the divorce. But we're all human; after a week or so he's leaning back from the keyboard, ears raised to get the latest developments. A productive environment? Nope.
I advise you to put your developers in private offices, with doors and off-switches on the phones. You probably won't do that. Every time I've fought this battle with management I've lost, usually because the interior designers promise cubes offer more "flexibility." But even cubicles have options.
Encourage your people to identify their most productive hoursthat time of day when their brains are engaged and working at max efficiency. Me, I'm a morning person. Others have different habits. But find those productive hours and help them shield themselves from interruptions for about three hours a day. In that short time, with the three-times productivity boost, they'll get an entire day's work done. The other five hours can be used for meetings, e-mail, phone contacts, supporting other projects, and so on.
Give your folks a curtain to pull across the cube's opening. Obviously a curtain rod would decapitate employees, generally a bad idea despite the legions of unemployed engineers clamoring for work. Instead, use a Velcro strip to secure the curtain in place. Put a sign on the curtain labeled "enter and die"; the sign and curtain go up during the employee's three superprogramming hours per day. Train the team to respect their colleagues' privacy during these quiet hours. At first they'll be frantic: "but I've GOT to know the input parameters to this function or I'm stuck!" With time they'll learn when Joe, Mary, or Bob will be busy and plan ahead. Similarly, if you really need a project update and Shirley has her curtain up, back slowly and quietly away. Wait till their hours of silence are over.
Have them turn off their phone during this time. If Mary's spouse needs her to pick up milk on the way home, well, that's perfect voicemail fodder. If the kids are in the hospital, then the phone attendant can break in on her quiet time.
The study took place before e-mail was common. You know, that cute little bleep that alerts you to the same tired old joke that's been circulating around the 'net for the last three months, while diverting attention from the problem at hand. Every few seconds, it seems. Tell your people to disable e-mail while cloistered.
When I talk to developers about the interruption curse they complain that the boss is the worst offender. Resist the temptation to interrupt. Remember just how productive that person is at the moment and wait till the curtain comes down.
(If you're afraid the employee is hiding behind the curtain surfing the net or playing Doom, well, there are far more severe problems than just productivity issues. Without trustmutual trustany engineering department is in trouble).
Remember the Titanic
Where should you use your best people? It's natural to put the superprogrammers on the biggest and most complex projects. Resist that urgeit's wrong.
Capers Jones, in a private study produced for IBM, showed that the best people excel on small (one man-month) projects, typically being six times more productive than the worst members of the team.2 That advantage diminishes as the system grows. On an eight man-month effort the ratio shrinks to under 3 to 1. At 64 man-months it's about 1.5 to 1, and much beyond that the best do as badly as the worst. Or the worst as well as the best. Whatever.
That observation tells us something important about how we partition big projects. Find ways to break big systems down into many small, mostly independent parts. Or at least strip out as much as possible from the huge carcass of code you're planning to generate, putting the removed sections into their own tasks or even separate processors. Give these smaller sections to the superprogrammers. They'll crank out solutions fast.
For example, suppose an I/O device, say an optical encoder, is tied to your system. Remove it. Add a CPU, a cheap PIC, AVR, Z8, or similar sub-one-dollar part, just to manage that one device. Have it return its data in engineering units: "the shaft angle is 27 degrees." Even a slowly rotating encoder would generate thousands of interrupts a second, a burden to even the fastest CPU that's also tasked with many other activities. Yet even a tiny microcontroller can easily handle the data if there's nothing else going on. One smart developer can crank out perfect I/O code in little time.
(An important rule of thumb states that 90% loaded systems double development time, compared to one of 70% or less; 95% loading triples development time.)
While cleverly partitioning the project for the sake of accelerating the development schedule, think like the customer does, not as the firmware folks do. The customer only sees features; never objects, ISRs, or functions. Features are what sell the product.
That means break the development effort down into feature-chunks. The first feature of all, of course, is a simple skeleton that sets up the peripherals and gets to main(). That and a few critical ISRs, perhaps an RTOS, and the like form the backbone upon which everything else is built.
Beyond the backbone are the things the customer will see. In a digital camera there's a handler for the charged-couple device, a liquid crystal display subsystem, some sort of flash filesystem. Cool tricks like image enhancement, digital zoom, and much more will be the sizzle that excites marketing. None of those, of course, has much to do with the basic camera functionality.
Create a list of the features and prioritize. What's most important? Least? Thenand this is the trickimplement the most important features first.
Does that sound trite? It is, yet every time I look at a product in trouble no one has taken this step. Developers have virtually every feature half-implemented. The ship date arrives and nothing works. Worse, there's no clear recovery strategy since so much effort has been expended on things that are not terribly important.
So in a panic, management starts tossing out features. One 2002 study showed that 74% of projects wind up with 30% or more of the features being eliminated. Not only is that a terrible wastethese are partially implemented features but the product goes to market late, with a subset of its functionality. If the system were built as I'm recommending, even schedule slippages would, at worst, result in scrubbing a few requirements that had as yet not consumed engineering time. Failure, sure, but failure in a rather successful way.
Dollars and sense
Finally, did you know great codethe really good stuff, that which has the highest reliabilitycosts the same as cruddy software to develop? This goes against common sense. Of course, all things being equal, highly safety-critical code is always much more expensive than consumer-quality code.
But what if we don't hold all things equal? A study by Oddur Benediktsson showed that using higher and higher levels of disciplined software process lets one build higher-rel software at a constant cost.3 If your projects march from low reliability along an upwards line to truly safety-critical code, and if your outfit follows, in his study, increasing levels of the Capability Maturity Model, the cost remains constant.
Makes one think. And hopefully, it makes one rein in the hackers who are more focused on cranking code than specifying, designing, and carefully implementing a world-class product.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at email@example.com.
- DeMarco, Tom and Timothy Lister. PeoplewareProductive Projects and Teams, 2nd Edition. New York, NY: Dorset House Publishing, 1999.
- Jones, Capers. Assessment and Control of Software Risks. Englewood Cliffs, NJ: Yourdon Press, 1993.
- Benediktsson, Oddur. Safety Critical Software and Development Productivity, conference proceedings, Second World Conference on Software Quality, September 2000.
After reading the first two parts of your series, I don't know whether to laugh or cry! The whole issue of interruptions has been something that I have tried to deal with for many years, and so far have not been able to resolve. Here, I am the only person who does electronic product design. That means concept, circuit design, PC board design, software, packaging, occasionally instructions, production support, customer support (occasionally)and, when I have nothing to do, R&D!
I have tried to get additional design personnel, tech support people, production engineers, etc. over the years, but we have been hampered by lack of money and difficulty in finding people with the right combination of skills & personality. Part of this is related to the lack of money issue!
At any rate - I am a poster child for the problems that occur from rapid context switching. I have sometimes reached the point where I sit at my desk (and I do enjoy the luxury of my own office) and literally do nothing because I can't figure out which of a dozen things to do needs my attention the most. I ask for prioroties so that I can focus on projects that are most necessary, but the priority can change in a few minutes, depending upon what strikes the boss' fancy at the time.
My advice to managers is to allow your people to choose the tools they need to get the work done, and make sure that YOU have your priorities in order & back up your people when they tell some other manager or department, "No, I can't do that right now."
OK, venting over! Back to the grind!
- Dave Telling
Firmware costs. The issue with delays in schedule and over commitment, in my mind, are mainly due to having inexperienced software engineers and/or a poor specification or scope. Trying to prepare Sr. managers for "gold-digger" engineers just is not feasible. If I spent $40,000,000 on a project that was scrapped, I would hope someone would fire me. Somewhere around 2 years late, I hope I would have replaced my staff. If time and money aren't the problem, the logical conclusion is scope. The guy that tells me he can "code that puppy over the weekend" and does, is the one I give a raise to.
In general, my problem with you article is that is simply talks about control. In my humble opinion, bosses should not have to control the process. Software engineers need to realize that the work do is just a little bit of magic. The magic they produce is vital to the survival of their company. If they don't get it done, the project, and even the company could fail. Excuses, delays, not using VCSs, bug tracking, and skipping reviews should not be tolerated. There are lots of very good programmers without jobs today because of those things. The countless hours of planning, lobbying for money, and creating the project scope are indeed the jobs of the bosses; execution is in the hand of the engineer.
- Keith Hansen