The abbreviations from the world of quality engineering can be mystifying. Take ISO9000, QS14000, and CMM for example.
And the last of these is no less baffling when you spell it out. The Capability Maturity Model. Even so, CMM is a useful tool for embedded systems engineers.
Thought of principally as a quality tool, CMM improves the entire software process. It stops feature creep, eliminates arbitrary deadlines, and kills bugs dead! Okay, I'm exaggerating slightly, but CMM does significant good in all these areas. More importantly, it tackles two critical problems in software development: waste and liability.
The waste associated with software is well documented. A study of 17 major Department of Defense (DoD) software contracts found that the average 28-month schedule was missed by 20 months.1 This means that any improvements in software productivity will help you in the market.
But getting to market isn't everything. Engineers have to think about what happens after shipment, and liability is one of the issues they need to consider. Lawyers would love to have an engineer on the stand who couldn't explain how his company designed their product. CMM provides a way of articulating the process.
And, more important than legal hassles, you don't want to hurt anyone. If you've read the report on Therac-25,2,3 you might remember that one of the faults was a variable that was not bounded, allowing it to overflow. Would a code review have saved people's lives? Maybe, but the report is careful to say that the problems of Therac-25 lay not in any particular hardware or software bug, but in the software engineering process-which is exactly what CMM addresses.
CMM was created by the Software Engineering Institute (SEI). The SEI is a federally funded research and development center sponsored by the Department of Defense. The SEI's mission is to aid engineers in improving the quality of software by helping organizations to predict and control quality, schedule, cost, cycle time, and productivity. They also aim to help engineers analyze, predict, and control selected properties of software systems.
“All models are wrong, some models are useful.”
These words, from The Capability Maturity Model: Guideline for Improving the Software Process by George Box, explain why the SEI-refreshingly-does not regard the CMM standard as some kind of Holy Grail. This probably has to do with the fact that CMM was developed from assessing actual software companies and including their feedback. As a result, CMM is not a standard in the typical, more rigid sense. It evolves as new data become available. This flexibility makes CMM more useful to engineers who need to accommodate up-to-date software practices, which evolve constantly themselves.
The CMM consists of five levels, through which your company gradually ascends, similar to the levels of skill signified by belts of different color in many martial arts. Each level breaks down further into Key Process Areas (KPAs; these guys will never have enough acronyms). For the most part, each level builds on, and requires mastery of, the preceding level. This is not a process undertaken lightly. Each level requires roughly two years to achieve, and you will not succeed without strong management support.
Level One: The Initial Level. This level includes companies whose successful projects depend on individual effort, can't be easily repeated, and likely exceed budget and schedule. Most companies, big and small, fall into this category. This covers everyone from the lone, run-and-gun cowboy coder to the company that has most of the formal tools in place, but doesn't implement them consistently.
Level Two: The Repeatable Level. Processes in this level enable successes to be repeated. Software projects are planned, time estimates are realistic, and necessary resources are determined ahead of time. Software projects are tracked, the project schedule is verified through milestones and audits, schedule slippages are caught early, and the problems corrected or the schedule redefined.
Level Three: The Defined Level. The software process and the management process are integrated. (At Level One, they seem to be mortal enemies.) Training programs are put in place as needed. Software engineering groups work with other groups concurrently on projects to communicate and address system and group issues. Projects follow a well-defined engineering process.
Level Four: The Managed Level. This level is where Statistical Quality Control begins. Pretty heavy stuff, but at this level the software is of a predictably high quality. The purpose of Level Four is to develop a quantitative understanding of the software process. The software process is then evaluated to identify variations and bring the performance of the process within defined limits. A Software project's quantitative quality goals are defined, monitored, and revised throughout the software life cycle.
Level 5: The Optimizing Level. Level 5 is software engineer Nirvana. (Interestingly, most of the handful of companies that have attained this level are located in India.4 ) At Level 5, the entire organization focuses on continuous process improvement. The people working on the Space Shuttle achieved this level. I'll let you draw your own conclusion from a comparison of the Space Shuttle's Level 5 CMM software design philosophy to that of the Mars Polar Explorers, whose mantra was “better, faster, cheaper.” 5 Only one software failure (a benign annunciation fault in 1989) has occurred during flight on the Space Shuttle since 1985 (the Challenger exploded due to a faulty o-ring). The Mars Exploration mission has a perfect 2-for-2 record of destroying or nearly destroying their multimillion-dollar spacecraft due to software “glitches.” At Level Five, a Software project is searched for defects encountered in the past, and actions are taken to prevent the defects from occurring in the future. New technologies are researched to aid in current and future projects.
Key Process Areas
The following KPA's are from Level 2 of the CMM:
Requirements Management. Requirements management is the ultimate in CYA. Requirements Management means writing down what the customer wants and getting him to sign-off on it. It may not be obvious, but faults at the requirements level are the most expensive to fix. For example, let's say there was an engineer named Dan (all names have been changed to protect the innocent) who was working for a company that had a customer who wanted a product that would display what gear a truck was in. They wanted arrows to indicate whether the driver should shift up or down. Sales had even come up with a clever name, the Slick Shiftr. Dan determined he could use hardware from another project, change a decal on the front, and write new software to create the device. Sales wanted it in a week, Dan told them three. Since sales wanted the product so soon, no one did a formal spec; the engineers worked strictly from what the salesmen told them. Dan got the software done in two weeks, and convinced a mechanical engineer to get the decals he needed. “The product's great,” said the salesman. “Now we just get their Human Factors people to sign off on it and ship 600 of 'em!” Unfortunately, as you might have guessed, the Human Factors people weren't happy. They thought the LCD display was too small. The mechanical engineer, who had gotten his supplier to work weekends, gave Dan all 600 decals. He still has them in a cabinet. Too bad they're too small to work as coasters.
Requirements Management also combats feature creep. If it isn't in the specification, your customer agreed to, they didn't pay for it. One of the biggest problems with the embedded systems community is the idea that software is free. A Requirements Management document brings the cost of software into the light, changing it from a burden, to a product.
Configuration Management – If you only implement one part of the CMM, make sure it's Configuration Management (CM). While CM used to be a gigantic, unwieldy paper trail, you will quickly find CM tools to be worth their weight in rework. Check your software into the tool. Poke around. Make some changes. If some of the change you made cause the software to crash, you can use the CM tool to look at the differences in your code. You can even revert to an earlier version. Some tools will let you compare any two files, not just the ones you have checked in.
Software Project Planning – Write down what you are going to do and when you are going to do it. Software Project Planning takes the requirements established in Requirements Management and generates plans for performing the engineering and management of the software project. The Software Project Planning process estimates the size of the project and resources needed, and develops a timeline for the delivery of “work products” based on the available resources. The work products are agreed-upon items that identify milestones in a project (e.g. compiler chosen, emulator up and running, delivery of first prototype, and so on).
Software Project Tracking and Oversight – In this is the Key Process Area, you make sure you do what you said you would in Software Project Planning. Software Project Tracking and Oversight also gives management visibility into the progress of a project. I once worked with a sales manager who liked to check our progress constantly. When I was working on one of his pet projects he would come by my desk every couple of hours and ask, “How's it going, Don?” When I got the LCD working, I had it display “HI JIM”, so it would show my progress before he had a chance to bother me. This is one way of giving management visibility into a project. With the CMM, you use the documents developed in Software Project Planning to develop the milestones that will show how a project is progressing, as well as any deviations from the schedule. If done correctly, deviations from the schedule will be noticed in time to take action, either by making the changes necessary to keep to schedule, or by revising the schedule to reflect reality.
Software Subcontract Management -Software Subcontract Management means selecting a qualified software subcontractor and managing their work successfully. This once again points out the need for good Requirements Management. How do you tell someone else how to do a job unless you have a clear idea what needs to be done? You are now the customer. Generate requirements for the subcontractor, a plan on when the requirements are to be accomplished, and track the subcontractor's work based on delivery of work products.
Similar to the Levels, the KPAs each expand on the previous KPA. Each KPA is further defined by the CMM with Key Practices. Requirements Management has fourteen Key Practices, such as the following: “For each project, responsibility is established for analyzing the system requirements and allocating them to hardware, software, and other system components.”
A company has successfully reached a level when they can show that they are consistently implementing the Key Practices of each KPA of that level, and all previous levels.
Gunfighter or Lawman?
In The Soul of a New Machine by Tracy Kidder, 6 one of the Design Engineers Kidder interviewed “remarked that what had happened near the end of the project reminded him of the typical conclusion of one sort of Western. A town hires a gunslinger to clean it up, but when he's taken care of their problem, he's still a gunslinger and sooner or later the respectable citizens are going to run him out of town.” Contrast this to a town's sheriff, who basically does the same job, but he's highly respected, and people feel comfortable with him. What's the difference? The gunfighter's methods are chaotic and unknown to any but himself. He might save a town, but he could also destroy it. A sheriff is a known quantity. He can tell you, step by step, what action he will take in any situation. While a gunfighter is run out of town even if he is a success, a successful sheriff is often elected to higher office. The sheriff has a repeatable process that allows the townsfolk visibility into his actions. CMM is a sheriff of software development.
So you want to hang up the guns, quit traveling from shootout to shootout, and become a respectable citizen by adopting CMM. Where do you start? Plenty of consultants out there are willing to help you (for a fee, of course). Before you call one, you should check the Internet, especially the SEI's website. They're kind of like the County Extension; you just need to get familiar with their services. Since you are starting out, you'll be shooting for Level Two. My suggestion would be to work on Requirements Management and Configuration Management first. These are simple, useful tools, and pay dividends quickly. While the tools used at the higher levels evolve from the lower levels, the CMM recommends using some higher-level tools early on to help jumpstart the process. One of these tools is a Software Engineering Process Group (a Level Three KPA). This group could initially pick the Configuration Management software and write up the format for Requirements Management. They could gather the data needed to prove CMM's usefulness to management. This group would also familiarize project members with the CMM.
One other tool I recommend for when you're getting started is peer review of documents generated for Requirements Management, of code, and of test procedures and reports (Level Three KPA). Done properly, peer review is an effective timesaver. 7
Attaining each level takes two years of commitment from the engineering staff and from management. You may think management will be a hard sell, but they're not stupid. They know as well as anyone the burden of software. They may just be skeptical about CMM's promises. Keeping them involved and informed through accurate metrics will help you retain their support. You can always whisper the words “potential lawsuit” and “liability” to them. If lawyers can sue ladder companies into nonexistence, they sure won't have problems with more complex systems.
How far do you need to take the CMM? Maybe not all that far. Author and former editor-in-chief of IEEE Software Magazine , Dr Alan Davis, believes that companies at level 4 and 5 have a disadvantage in getting products to market rapidly. 8 And if you decide to attain ISO9000 certification, it is generally agreed that a company that is Level Two and well on the way to Level Three can easily be certificated. 9,10 Unless required by a customer, your company may decide to stop at Level Three, and possibly integrate some of the Key Processes in level 4 and 5 in some other manner.
Will you and I reach Level 5? Well, maybe not in this lifetime.
Don Warbritton has worked for 13 years in industry, writing embedded code for everything from ice cream machines to hay balers, in several programming languages, including Forth. He has a BSME from the University of Michigan, and a MSSE from Oakland University. He enjoys rock climbing, hiking, and mountain biking near his home in western Colorado. He currently is a Software Engineer for Ametek/Dixson, manufacturers of instrumentation for the Heavy Truck market.
1. The Capability Maturity Model: Guidelines for Improving the Software Process , Software Engineering Institute. Publisher: Addison-Wesley.
2. Safeware: System Safety And Computers , by Nancy G. Leveson, University of Washington. Publisher: Addison-Wesley.
3. IEEE Computer , Vol. 26, No. 7, July 1993, pp. 18-41 “An Investigation of the Therac-25 Accidents” http://courses.cs.vt.edu/~cs3604/lib/Therac_25/Therac_1.html.
4. Denver Post , page 1, March 29, 2000 “Glitch doomed Mars landing” by Ann Schrader, Denver Post Science Writer.
6. The Soul of a New Machine, by Tracy Kidder, Publisher Little, Brown.
7. “Benefits of CMM-Based Software Process Improvement: Initial Results,” http://www.sei.cmu.edu/publications/documents/94.reports/94tr013/94tr013chap04.html.
8. “Indian software firms need to deliver faster,” zdnetindia,
9. The TickIT Guide: A guide to Software Quality System Construction and Certification using ISO9001 : 1994, Publisher: British Standards Institution.
10. “A Comparison of ISO 9001 and the Capability Maturity Model for Software,” by Mark Paulk, http://www.sei.cmu.edu/publications/documents/94.reports/94.tr.012.html.