Metrics for Developers - Embedded.com

Metrics for Developers

Metrics can be used for good or ill, but they work best as a window into the development process. They're not very effective for evaluating people.

I bet you don't really know-quantitatively-what your productivity is, measured in lines of code per hour or any other units. Most teams average a few hundred lines of code per month per programmer, measured from program outset to shipping.

I bet you don't really know-quantitatively-your effectiveness at meeting schedules. (Eighty percent of products are delivered late.) Or the cost of a new project-generally $15 to $30 per line of code. Or how ship dates are exponentially related to code size. (Barry Boehm showed that schedules are proportional to KLOC1.5, where KLOC is the code size in thousands of lines. Later work suggests that the exponent is often larger.)

I bet you can't anticipate-quantitatively-bug rates in a new development. Yet bugs and poor quality contribute to late projects more than any other factor. (Once a module compiles cleanly, you can expect about a 5% error rate.)

Some developers subscribe to the belief that we're practitioners of a complex creative process that implicitly defies measurement. Like artist, we invent products from the void, building something from nothing more than our own cleverness. Did Da Vinci work to a schedule? It'll be done when it's done!

Balderdash.

MBA textbooks claim, “If you can't measure it, you can't manage it.”

More balderdash.

If you're mass producing widgets, dozens of different measurements will indeed yield the information needed to optimize both productivity and quality. Likewise, financial controls are nothing more than extremely detailed measures of a company's performance. In both cases, managers have a fiduciary responsibility to employ every possible tool to help the business thrive.

Wise engineering managers, too, must employ every reasonable tool to optimize development. Abdicate management and chaos will reign. For all but the smallest projects, it's impossible to coordinate the efforts of multiple teams, and to keep individuals on-track, without using a reasonable set of metrics.

Metric abuse

Unfortunately, most of us run in terror when the new management team announces yet another round of metrics. Experience shows that MBA's love to collect data. They seem much less adept at interpreting it, and at effecting useful change from the measures.

My favorite dysfunctional measurement is the time card. In most organizations, it's that dreary Friday-afternoon routine we remember at the last minute, when the paycheck police withhold the check until we turn in a completely filled out timecard.

The motivation is getting paid; the “entry criteria” (as the process people would say) is filling in the time card. There's neither feedback nor closure. We could enter codes indicating vacation time or parties in Las Vegas (if only there were an appropriate charge number); both would suffice for getting the paycheck.

We're all smart people; most of us went to college. Those lab classes taught us a lot of skills, none more useful than the fabrication of data. So we exercise our creative powers in recreating more or less what happened during the week. The data is meaningless, of course. Few honest folks expect much accuracy on the timecards. Yet the ritual is repeated week after week in companies scattered across the globe.

Why do we persist in this parody of precision, when better approaches exist? Lawyers, whose careers rest on accurate billing have tools that make gathering such data both easy and accurate. It seems to me that either it's time to stop taking the data, or to use an alternative that's meaningful.

As a teenager, I worked as a technician for an outfit that, in a desperate attempt to fix massive quality problems, mandated that every part we replaced in the production units go out for analysis by a lab to find the root causes of the failures. What if we erroneously replaced something that was, in fact, fine? No problem, management patronizingly assured us, we know you won't make such mistakes. We did. Since every replaced part must by definition be bad, it behooved us to ensure that even these mistakenly changed devices had failed. I remember putting 110 VAC across a metal-can op amp, hoping to fry the silicon. Imagine my shock when the power blew a hole in the package! I never found out what root cause the lab assigned that failure to.

Silly measurements create contempt in the development team for metrics. If your organization is so broken, consider some easy measurements that can yield interesting data. For instance, try the Aspirin Metric: place a big bottle of aspirin in a centrally located area and log its rate of consumption. Or the Friday Beer Tab Metric: create a spreadsheet that logs the bar tab for the team's weekly gathering. Both approaches can tell you a lot about burnout.

Managing strictly by measurement corrupts the data we're trying to gather. Heisenberg showed us that we can not know both the position and momentum of a particle; measuring one effects the other. A similar principle applies to organizational metrics. A wonderful Dilbert strip shows the pointy-haired boss telling his tech writer that she'll be rewarded based on the number of words she writes, with points off for those deleted. “Don't want to reward the wrong behavior,” the boss struts. In the next scene, Tina maxes her pay, while minimizing productivity, by writing tons of jibberish.

Years ago, when running a tool business, I noticed that production was slipping. In my infinite wisdom, I imposed a set of piecework rules. Shipments quickly increased, but quality fell. The imposition of more measures boosted quality. Inventory levels skyrocketed. The techs gave up repairing bad boards and simply stacked defective ones.

I learned that management by measurement is like squeezing a balloon smaller. For a while it works, but then an unexpected aneurysm pops out. We're simply not smart enough to anticipate all possible problems.

Why measure?

Engineering managers take data for one of three reasons. The first is to reward or punish the participants. As a young engineer working on a big job with an impossible schedule, my boss told me that if the unit was “done” (whatever that meant) by the show date, I could accompany it to Europe. That sounded pretty good to me. The machine and I got our overseas vacation but, of course, the unit barely worked. My goals were satisfied while the company's were not. Rewards have to be structured carefully, so that both parties get exactly what they want.

Working for that company made the most dramatic theme park ride look sedate. Its fortunes rose and fell at a dizzying pace. When we went public, I overheard an outside auditor say, “I've seen every trick this outfit has pulled, but never all by one company.” So it wasn't unusual for the boss to tell me that if we were late or over budget on a new project, I'd have to lay off a few people. That's a horrible way to measure one's performance. The threat of such punishment hanging over our heads drove the best people to headhunters and humiliated the rest.

Measures are a poor way to motivate people. Deming, the quality guru, found there are two kinds of motivators. The first, called “intrinsic,” are those that come from within, from deep in our souls. Wanting to do a good job. Feeling like part of a great team. Taking pride in our work.

He also studied “extrinsic” motivators: when the boss makes a silly demand, or imposes strict rules we don't understand or support. Deming found, to no one's surprise, that extrinsic motivators drive out the intrinsic ones. We comply unwillingly and drag our feet.

Or, we resist. In the '60s, we learned that passive resistance is an effective way to toss sand in the machinery. Go limp when the cops bust you; they have to drag you away. The same thing happens in engineering. We know those managers will be gone in two years, promoted or fired. Do nothing for a while and the rules will change.

Measures are ineffective motivators. Take data for one reason only: to get insight into a product and process, to learn what is going on now, so we can improve it tomorrow.

Good measures

Every time someone proposes a metric, ensure that it will lead to greater insight and understanding. Ruthlessly abandon metrics that don't.

There are four components to any measurement: collecting, interpreting, presenting, and acting on the data. Typically, we think in terms of data collection and forget about the other three, equally important steps.

Raw data is pretty, well, raw. Counting source code lines might make sense, but until it's normalized somehow (are you counting whitespace and comments?), it's a nonsensical jumble of unrelated numbers. If counting bugs, what constitutes a bug? A violation of the firmware standard? We've got to interpret the data to make sense of it, and to remove the randomness that comes with the collection process.

This is the communications age. We are the people who built it, who constructed the cellular and Internet equipment that facilitates an unprecedented amount of person-to-person interaction. Yet we're often the least capable users of this fantastic infrastructure. Communication is the key to career growth and to effective teamwork. Once the data makes sense we must present it to ourselves, our colleagues, our bosses, and our customers. Presentation is more than a dreary recitation of facts and figures. It's a sales job. Breathe life into the sterile numbers; make the insight they offer clear.

Many of us are reluctant or even terrified to stand up in front of a crowd. I was in my thirties when I realized that this kind of fright was limiting my career. So I decided to take some action and learn to give talks. The first few times were horrible, sweat-filled hours that left me ready to delve back into solo programming forever. But with practice it became easier and eventually very enjoyable. If the thought of talking to a group makes you fearful, or if you leave the audience with glazed, bored eyes, take some action. Fix the problem. Dale Carnegie and Toastmasters both have effective programs.

Finally, act on the data. Numbers are useless unless we use them to effect change. Improve something. Run experiments. Tune the process and see what happens to the numbers. Collect, interpret, present, and act. Neglect any of these and the measurement is useless.

Other requirements

Measurements must be easy. Onerous data collection always fails. It's abandoned as soon as oversight lags. Measurements must make sense today. Unfortunately, metrics can be like government bureaucracies. Once imposed, even when used to understand a particular problem, they never go away. One case in point is a company that faced huge turnover. Various measurements indicated that the problem stemmed from bizarre program management procedures. Process changes all but eliminated defections, which plummeted and then plateaued within months, yet years later they still took the same measures.

Maybe it makes sense to re-measure at much less frequent intervals, but wise management will impose a sunset law on every metric, holding each to review at frequent intervals

Finally, measurements must be visible, so that people see what is going on. They need to see the value of action versus avoidance and sometimes even outright sabotage. It all comes down to people and their buy-in. Lose that, and everything collapses.

The meaning of measurement

I was sailing on a friend's boat recently, going great guns in a decent breeze. In a quest for speed, he put up the spinnaker. Foam flew, the wake grew, and Lee was elated.

Having looked at the knotmeter both before and after flying the bigger sail, I knew the facts: we weren't going any faster. Sure felt like it, but the boat was at hull speed, and no amount of sail would accelerate her.

What we feel is tempered by what we expect and want. Only quantitative data tells the truth.

But desiring to be a polite guest, I kept my observations to myself. The knotmeter told the truth, but, being unwilling to act on the information, I needn't have bothered to look at it.

Jack G. Ganssle writes and gives seminars about ways to build better firmware faster. His latest book is The Art of Designing Embedded Systems. Contact him at .

Return to June 2002 Table of Contents

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.