Advertisement

Scheduling Momisms, or lessons your mom should have taught you

August 01, 2007

Jack Ganssle-August 01, 2007

The IEEE Code of Ethics requires us to be honest and realistic in stating claims or estimates.

Momism ( môm iz' em) n. 1. A brief statement of a principle passed maternally. 2. A tersely worded statement of an observation of truth: APHORISM

I recently saw a commercial in which a middle-aged couple are sitting in their car, in the dark, apparently in front of their son's apartment. Neither speaks a word, but at first we hear dad's thoughts: "What's wrong with that kid? He's 22 and it's time he grew up and took care of his own problems." Then mom's mental musings fade in: "What do we have money for, anyway, if not to help him?"

Those words could have come from my wife and I. Or from so many other parents we know who have teenagers and 20-somethings. Fathers let go; mothers keep an empathic connection to their kids that's truly remarkable, admirable, and, for dads, sometimes frustrating.

Biographies often paint a harsh picture of dad as distant or the disciplinarian. Mom is always a saint. All of us remember mom's words of wisdom and advice, usually gently given in counterpoint to dad's imperative directives. My mother offered lots of advice, some of which still guides my work. Here are a few of her thoughts about creating schedules for embedded projects.

Be honest in scheduling--Mom thought that the Devil drives scheduling in most engineering teams, which is why most projects are late. She often quoted the IEEE Code of Ethics, which, among other things, requires us to "be honest and realistic in stating claims or estimates."1

Few of us do this, of course.

One of the problems with modern politics is that it's entirely focused on the practical rather than the ethical. I wish we started all policy debates with a search for the ethical, deriving a sensible strategy from that. Similarly, the phrase: "The boss is gonna cut whatever schedule we give him in half, so let's double our estimate" is, while both practical and logical, mendacious. If the boss will cut the schedule in half, that's a different problem unrelated to that of creating a realistic schedule. Deal with it separately from scheduling.

There will always be a fundamental disconnect between the very real needs of management and those of engineering. Capricious deadlines aside, a Mars mission, for instance, cannot slip. There's a launch window only every two years. You can't negotiate with planetary geometry. The schedules on more earthbound projects may appear more fluid, but very real business drivers, such as limited engineering funding, market windows, and customer needs, legitimately compromise the also very real needs of the development team. Strategies promoted by the agile community do help bridge this gap.

Manage complexity--Mom always warned us to manage complexity because code complexity grows faster than code size. Robert Glass claims that for every 25% increase in a software problem's requirements, the complexity of the solution doubles.2 Perhaps this explains why firmware sizes are growing 46% per year.3 Couple that with the typical $20- to $40-per-line cost of firmware and it's clear that a little feature creep can cost astronomical amounts of time and money.

Complexity is the Devil's workshop--Complex code is buggy code. Data I've collected, to be shared in a future article, unsurprisingly shows that complex functions are buggy functions. So, measure complexity using the McCabe metrics and recode those with high scores.4

Never create a calendar schedule--Scheduling is the process of estimating how much work is required to complete a project. Your calendar represents a snapshot of time allocated to numerous disparate and always changing projects, meetings, support, and needs. The two are measures of very different things and are not equivalent.

Don't overload the CPU--Ah, this is one my mother whispered to us nightly, just before sending all five of us off to our beds in our Dr. Dentons. Her words still echo in my mind: "Dear, a 90% loaded system doubles development time; at 95% loading, the schedule triples."5 Though hardly surprising, this fact is too often neglected by many of us. Finding a few extra CPU cycles is just as difficult as squeezing one little feature into a system with little spare ROM or flash.

Never provide a single-point estimate--One of mom's pet peeves. She'd tell us: "An estimate is a probability distribution function. Only a fool believes any distribution given with zero variance!" Provide min/max estimates specified to some standard deviation. I like to use 2s, giving a probability of being right 95% of the time. Karl Wiegers has good advice here.6

Use your best people on the small projects--Superprogrammers, the best developers on the team, are six times more productive than the worst team member, but only when both are measured on small projects. As project size grows the best and the worst become equally nonproductive. Big projects have vast amounts of overhead--meetings, e-mail, and the like. A superprogrammer can attend a meeting no faster than a sluggard.

Your boss is right about the show--Engineers have always been infuriated when reasonable estimates get tossed because the show--there's always a show--is three months away. That's a rotten way to schedule a project. But it may be the correct way. The show is important. It's a chance to see customers, demo to the press, frighten the competition. Engineering's role is to serve business needs, including important events like the show.

If a real deadline and the show deadline aren't congruent, create enough features to make a fantastic demo. Attendees don't dig through every menu and option; they rarely look at a product for more than 10 or 20 seconds. How much needs to work? Again, the agile precepts express much that's wise when it comes to building a system to meet an early deadline.

Where's your $%#W$ report card?--Mom is no dummy now and was just as sharp when I was in high school. The Jesuits foolishly mailed our report cards home. For some reason, the report cards always turned up on a Friday afternoon. Till we were caught, my siblings and I would pluck our reports from the mailbox and reinsert them Monday, so the weekend wouldn't be ruined.

Engineering teams sometimes do the same thing when scheduling. Hunched over Microsoft Project, everyone jiggles triangles to create something that's, well, believable if not accurate. We're really just trying to postpone the inevitable day of reckoning. Maybe something will change for the better. Maybe the project will be cancelled. But the instinct to defer pain is a strong one. As mom often said: "Fess up now!"

Separate your R from your D--There's no such thing as R&D. I have no idea why we routinely lump "research" with "development." That's about as accurate as linking "science" and "astrology."

Development is the process of solving a known problem with known technology. Research is all about discovering new things. If research were schedulable like development, we could expect scientists to find a cure for cancer by April 22, 2008. If that's too aggressive, add resources: more scientists.

A lot of embedded applications rely on some underlying science, such as the response of a sample to various wavelengths of light. If the science isn't right or completely understood, the schedule is toast.

But mom had plenty to say about resources. In fact, I discovered that Fred Brooks secreted microphones in our house to pick up mom's homespun wisdom, which he later plagiarized into his The Mythical Man-Month.7, 8 He ripped off mom's dinner-table throwaway phrase: "Children, remember that adding people to a late project makes it later." More people means more meetings, more reports, and less productive work.

Don't take the Lord's name in vain--One of her prime values as a churchgoing lady, this prohibition was something mom never let us forget. It took us a while to discover, however, that this rule covered an entire host of words in addition to the Lord's name, most of which were composed of four letters. Since she doesn't follow Embedded Systems Design magazine (she prefers IEEE Transactions on Software Engineering), I can get away with using one of those here: hack. Oh, how she detests that curse! I remember getting whacked for hacking some Perl code, till I explained that no one knows how to write clean Perl. (She's an old-school kind of girl and mostly works in assembly and SNOBOL.)

But mom was wrong. Hack is a perfectly acceptable word even in mixed company (OO developers and procedural programmers, that is). Hacking is a perfect tool for doing research. When the hardware team misdocuments I/O control registers, the firmware folks must use either bribes or focused hacking to elicit the devices' actual operation. But once the research is done, switch to a more disciplined model and create awesome code.

Overtime takes engineers away from their mothers. Mom sure nailed this! A 2005 study by Circadian Technologies documented what we all know: consistent overtime is a Bad Idea.9 Productivity can decrease by as much as 25% when people work 60 or more hours a week for weeks on end. It also increases turnover by as much as 300%, while nearly doubling absenteeism.

Once, I remember, she said: "Honey, someday folks will invent something called eXtreme Programming. Those XPers will say some funny things; some of those fellows will need their mouths washed with soap. And where was their mother when they learned to capitalize? But they will be right when they say 'never work overtime two weeks in a row.'"

Concentrate on one thing at a time--Mom was quite clairvoyant, which is probably why she was so good at scheduling. Why, all five of us shipped in about nine months! When I was a small lad hiding under a desk during the Cuban Missile Crisis, she warned: "Someday you'll have a wireless phone so you can call me anytime those godless commies threaten you. Just don't yak on the phone while driving, unless you need to call me." I think she was trying to warn me of the perils of multitasking. Researchers have found a 20% efficiency hit when multitasking between two activities.10 That jumps to 65% when managing four. So balancing multiple activities, although often unavoidable, destroys schedules.

Finally, here's a bit of wisdom for all of us: Call your mother.

Jack Ganssle (jack@ganssle.com) is a lecturer and consultant specializing in embedded systems' development issues. For more information about Jack click here .

Mom's sources:

1. IEEE Code of Ethics, www.ieee.org/portal/pages/about/whatis/code.html
Back

2. Glass, Robert. Facts and Fallacies of Software Engineering. Boston, MA: Addison-Wesley, 2003.
Back

3. Rich Nass quotes VDC in the webinar "Solving the Embedded Code Crisis." Tuesday, June 19, 2007. http://www.embedded.com/showArticle.jhtml?articleID=199903515)
Back

4. McCabe metrics: http://en.wikipedia.org/wiki/Cyclomatic_complexity
Back

5. Davis, Alan. 201 Principles of Software Development. New York, NY: McGraw-Hill, 1995.
Back

6. Karl Wiegers has good advice in his paper "Stop Promising Miracles,"originally published in Software Development, February 2000: www.processimpact.com/ articles/delphi.html
Back

7. Fred Brooks secreted microphones in our house to pick up mom's homespun wisdom--see National Inquirier, May 20, 1978, page 28, right next to the story about the 600-pound cat.
Back

8. Brooks, Frederick P. The Mythical Man-Month, second edition. Reading, MA: Addison-Wesley, 1995.
Back

9. Circadian Technologies, Inc. Shiftwork Practices 2005. www.circadian.com/publications/swp2005.html
Back

10. Clark, Kim and Steven Wheelwright. Managing New Product and Process Development: Text and Cases. New York: The Free Press, 1994.
Back


Excellent article. Probably could be made into a book, but each point is summed up well enough that a bood would probably only serve to clutter the concise message of each item. Maybe a pamphlet or at least licensed to technology teachers so that they can include in Classroom instuctional materials. Keep up the good work!

- Jeff Kohut


I just read your article on Momisms. Since we just talked, I'm revved for this exact discussion. So here's your article, with my Tensilica-centric comments:

Manage complexity--Mom always warned us to manage complexity because code complexity grows faster than code size. Robert Glass claims that for every 25% increase in a software problem's requirements, the complexity of the solution doubles.2 Perhaps this explains why firmware sizes are growing 46% per year.3 Couple that with the typical $20- to $40-per-line cost of firmware and it's clear that a little feature creep can cost astronomical amounts of time and money.

/One of the chief contributors to code complexity is multitasking. We have adopted RTOSes (well, some of us have) in an effort to separate tasks, but tasks cannot be truly separate if they're running on the same processor. All operating systems are created to give the illusion that each task has a whole processor. Why do we multitask? To save the cost of another processor. That was a real concern in the days when processors were individual chips. It's much less of a concern when a processor is a half millimeter squared of silicon. Saving processors on SOCs is often false economy. We all believe hardware is cheap thanks to Moore's Law, but somehow processors don't seem to qualify as hardware. Go figure./

Complexity is the Devil's workshop--Complex code is buggy code. Data I've collected, to be shared in a future article, unsurprisingly shows that complex functions are buggy functions. So, measure complexity using the McCabe metrics and recode those with high scores.

/How does code get complex? By adding functions. No one tries to create monolithic blocks of 500,000 transistors. We know we can't handle the complexity. Yet we create programs as though they were immune to the sin of complexity. Got more functions to implement, get another 100MHz of processor speed and write more code. That's old school and it's not a good way to design modern systems based on nanometer silicon./

Don't overload the CPU--Ah, this is one my mother whispered to us nightly, just before sending all five of us off to our beds in our Dr. Dentons. Her words still echo in my mind: "Dear, a 90% loaded system doubles development time; at 95% loading, the schedule triples."5 Though hardly surprising, this fact is too often neglected by many of us. Finding a few extra CPU cycles is just as difficult as squeezing one little feature into a system with little spare ROM or flash.

/Loading a processor to 90% is like loading a bridge to 90%. You're inviting failure. Why? Again, it's to save a processor. Why? Because everyone knows that processors are expensive. This maxim is not true any longer. Moore's Law has made transistors plentiful. It's clock rate and design time that are now in short supply, making them expensive. Technology evolves and changes design assumptions. The I35W bridge that fell down in Minneapolis was designed during a time when steel was relatively expensive and welding labor was cheap. Consequently, the steel used was thinner, to save on raw material cost, and the number of welds was increased to add stiffening. Now we know that every weld is a potential failure point. In addition, labor is now more expensive that raw materials. Bridge design and the maxims used to engineer bridges have changed accordingly. Bridges are now designed to be beefier with fewer connectors. Electronic system design must also change for 21st-century reality, not 1980s reality. /

Development is the process of solving a known problem with known technology. Research is all about discovering new things. If research were schedulable like development, we could expect scientists to find a cure for cancer by April 22, 2008. If that's too aggressive, add resources: more scientists.

/I think too much development is based on outdated assumptions. Question all of your assumptions. Of course, this means you need to introspectively root out all of those subconscious assumptions you've collected since you started as an engineer./

But mom had plenty to say about resources. In fact, I discovered that Fred Brooks secreted microphones in our house to pick up mom's homespun wisdom, which he later plagiarized into his The Mythical Man-Month.7, 8 He ripped off mom's dinner-table throwaway phrase: "Children, remember that adding people to a late project makes it later." More people means more meetings, more reports, and less productive work.

/There's a make-versus-buy corollary here. A lot of engineers like to say "I can do it better" when looking at a purchased bit of IP (software, hardware, tools, it's all the same). In the short term, the local team may well be able to design a better widget (software code, hardware block, tool, whatever). In the long term, that local team most likely will not maintain or improve whatever's developed because they will move on to other projects, become managers, leave the company, or whatever. Testing's yet another story. How many engineers do you know who like to break their own stuff? I don't know that many. Testing of internal design too often gets a lick and a prayer. Too many design teams have to work with old, unsupported but "better" bits of technology that may as well have been coded in Harry Potter's magic spells for all the good they do. These technology bits have to be incorporated as whole cloth because the people that understood the underlying design are no longer available and every reuse often digs up a new failure mode in the design. Companies that sell technology for incorporation into your design need to test more thoroughly and you get the benefit of all the companies that have used that technology before you. Now some will say that they've purchased technology before and it was of poor quality (they may use a coarser term). Indeed, some technology purveyors are "Freds in the shed" and they don't test. They don't document. Don't buy from them. Buy quality. It pays. Ask for references. See who has already become successful using any potential new, purchased technology./

But mom was wrong. Hack is a perfectly acceptable word even in mixed company (OO developers and procedural programmers, that is). Hacking is a perfect tool for doing research. When the hardware team misdocuments I/O control registers, the firmware folks must use either bribes or focused hacking to elicit the devices' actual operation. But once the research is done, switch to a more disciplined model and create awesome code.

/Make versus buy applies here too. A company that sells something as a product must do a good job of documenting the product or the company will quickly go out of business. Use technology that's been documented as though someone's job depended on the documentation quality./

Concentrate on one thing at a time--Mom was quite clairvoyant, which is probably why she was so good at scheduling. Why, all five of us shipped in about nine months! When I was a small lad hiding under a desk during the Cuban Missile Crisis, she warned: "Someday you'll have a wireless phone so you can call me anytime those godless commies threaten you. Just don't yak on the phone while driving, unless you need to call me." I think she was trying to warn me of the perils of multitasking. Researchers have found a 20% efficiency hit when multitasking between two activities.10 That jumps to 65% when managing four. So balancing multiple activities, although often unavoidable, destroys schedules.

/That advice applies to processors too. There's an overhead for multitasking. Ignore it at your peril./

- Steve Leibson

Loading comments...