eXtreme Programming is all the rage these days. But what is it and how does it apply to embedded software development? Jack offers his two cents.
Engineering: n. The application of mathematical and scientific principles to practical ends, especially the design, construction, and operation of economical and efficient structures, equipment, and systems.
Does the preceding definition suggest “software engineering” is an oxymoron? In the space of a single generation, we've created a society intrinsically dependent on software, both of the embedded and non-embedded varieties. No one knows just how much software gets built each year, but it's obvious that every electronic product now has a software component. I often complain about the buggy code that surrounds us, but, in fact, most software works remarkably well. Despite plenty of software disasters, large and small, the world hums along.
But I'd wager that much of what we build is fragile, held together by heroics and Band-Aids. Worse, software ages badly. Entropy sets in, maintenance all too often turns a rickety structure into a house of cards swaying in the breeze. I'm often reminded of an old joke: if architects constructed buildings the way software people make code, the first woodpecker that came along would destroy civilization.
We all know how hard it is to create high-quality software, and we're also familiar with the even more difficult process of keeping it clean, despite enhancements and maintenance. Like supermodels pursuing the latest fad diet, many of us flail about, trying all sorts of new development methodologies and ideas.
Most of these attempts fail. I think the methodologies are usually too big, too complex, too time consuming, and too mired in paperwork. For instance, if we decide to use the Capability Maturity Model (CMM), for example, we're guaranteed to fail without major buy-in from the whole company, from the president on down. Since software is generally perceived as a necessary evil, any attempt to rally the organization in a novel, but expensive, manner is just a high-tech form of tilting at windmills.
So I look for small-scale approaches, ideas that developers can adopt individually, or perhaps as small teams. I'd rather fly under the radar screen of the big bosses and make changes now that give immediate benefits. One discipline I adopted years ago, one that is so personal no one even needs to know what you're up to, is the Personal Software Process, a product of the mind of Watts Humphrey. Humphrey scaled many of the grand CMM ideas down to individual levels.
But in recent years, a revolution of sorts has been brewing in the ranks of programmers. Many feel the idea of Big Up-Front Design is flawed. They worry that we're not really smart enough to anticipate all possible problems. A new philosophy emphasizing coding over design has spawned several interesting approaches to building products. The most popular, or, at least, most talked about, is eXtreme Programming (XP), a brainchild of Kent Beck.
Frankly, I'm a bit on the fence. I believe a huge part of the design problem stems from shoddy work, not impossible systems. Most of us detest the tedious effort involved in thinking through the entire design before committing to code. Design is hard. Too many of us shy away from the very tough job of doing it right. Management can be even more dysfunctional. When they demand a schedule by Friday, when there's no chance to complete a detailed design, they are, in effect, telling us to lie to them. The meaningless schedule then dominates the project, shortcutting proper requirements analysis and all the rest.
Yet even the best-run project is a work-in-progress, since requirements always change over the course of development. The design is rooted in quicksand. We are wrong to rail against change requests; these are inherently part of a complex effort. (However, when management expects changes without cost, they are in the wrong).
Kent Beck envisioned a new software development process where programmers do all of the “good” things in excess. If code reviews are good, we'll do them all of the time. If code that grows convoluted is bad, we'll clean it up constantly. If testing is important, testing will be as important as coding. If customer feedback helps, the customer will be an integral member of the team.
Beck believes that software projects work best when guided by many small course corrections rather than a few big ones. The philosophy of XP is that everything changes all of the time. The business model changes. Technology evolves. Team members come and go. Requirements change. As Beck puts it “The problem isn't change, per se, because change is going to happen; the problem, rather, is the inability to cope with change when it comes.”
The XP programming model is derived from four core values, rather than an abstract view of how we should make projects. This approach greatly appeals to me, as I see so many projects head towards disaster when there aren't any well-intentioned, courageous developers and leaders to keep things on track.
The first value is communication. Problems often come from lack of talk. We don't listen well to the customer, or to Joe down the hall working on the comm interface, or to the boss who is working under constraints we haven't considered.
The second value is simplicity. Keep the code clean. Make it do nothing more than is absolutely required to get the current task done. Gulp. That means don't plan too much for the future. XP believes it's best to do something simple now, and change it later to meet our future, enhanced view of the overall system's need. A fascinating idea, though I waver a bit on this one.
Third, employ massive feedback at each stage of development. As he says, “Optimism is the occupational hazard of programming. Feedback is the treatment.” Brilliant! After all, we know that feedback stabilizes systems. This includes program development. Feedback means testing constantly to prove (or at least show) correctness. It means working intimately with the customer.
Finally, we're doomed to fail without courage, the fourth and, perhaps, hardest of XP's core values. Courage means being willing to toss out crummy code and start over. Maybe someone has a crazy idea; courage means spending time investigating it, and being willing to back up and refocus if the idea pans out. Courage means telling the customer-and your boss-the truth about project status.
So how do we translate these values into action? XP defines a dozen practices.
Practice 1: The planning game
Business and technical people work together to make basic and critical project decisions. What are the important release dates? How much time will a feature need? What's the scope of the project? And which features are truly the most important?
The work is divided into what Beck calls “stories.” A story is a fully-implemented feature, described in a jargon-free way the customer can understand. Negotiations between business and technical people result in a set of stories that forms a release, though each release is much smaller than the entire project.
One difficult part of XP is the necessity of accepting a foggy future. The planning game does not anticipate a complete project definition, let alone design, since change is so dynamic. This gives me nightmares, though it's interesting and not to be lightly disregarded.
Practice 2: Small releases
Get something out to the customer frequently. Aim for releases every few weeks. This practice often necessitates an on-site customer who can evaluate the release, suggest or demand changes, or perhaps even accept it.
Many modern programming approaches emphasize frequent releases of subsets of the project. All recognize the difficulty of meeting customer expectations when we dump one huge completed system in their laps. The spiral development model (code, release, re-estimate the next iteration, and repeat) is another method that recognizes the difficulty of up-front estimation, particularly for risky projects.
Practice 3: Metaphor
Just as important as a dreary list of features or stories is an overall high-level description of the project, in terms that the techies, business people, and customer can understand. The words chosen to express the metaphor form the start of a common language used by all of the stakeholders.
Practice 4: Simple design
XP pushes our quest for simplicity to rather frightening heights. Adherents tell us to write each class or function to do only what the current story demands. Sure, you're thinking that the next story, which you'll start working on in a couple of weeks, might need a more generalized handler. Or maybe we need to abstract this class a little more in case another developer has a specific need. These ideas are anathema to XP, the philosophy of which is to do what you need now, and change it later as needed.
This is one area where XP departs from conventional development models. We're taught that code is expensive. XP preaches that coding can be relatively cheap. Clean, simple code can be reworked for less than we usually figure. This is a fascinating, though scary, idea.
Practice 5: Testing
Every function has its test suite. The tests are as important as the code; shortchanging tests is strictly forbidden. As we exercise each class or function, we build a library of tests that are preserved with the code itself.
Practice 5 neatly complements Practice 4. Fear of changing code diminishes when an extensive test suite is ready to quickly prove or disprove functionality of the modified software.
Though there's certainly nothing novel about stressing the importance of test, to me this is one of XP's most powerful and exciting ideas. (Okay, so I need to get a life.) I've got files of embedded disasters, almost all of which stemmed from lousy testing. Even highly disciplined organizations have difficulty generating adequate tests because these are always constructed based on an external view of what the function is supposed to do. We have a hard time anticipating all of the myriad paths a real function may take to implement a desired capability.
In XP, we write the tests as we write the code. This is the time to write a comprehensive test that irons out every possible combination in a five level deep IF-ELSE statement. It's a chance to ensure that we test every exception possibility, as well as the boundary condition issues that come with passed parameters.
When we change the code later, we change the tests as needed. Neither tests nor production code are allowed on their own; both form a synergistic whole.
Testing might be hard to implement with many embedded applications. The XP notion of a test is something rather automated, which may defy systems that require someone to press buttons and interact with other devices. Yet how do we dare not create such exhaustive and effective automatic tests?
Practice 6: Refactoring
Refactoring is a fancy word for chucking the cruddy code and starting anew. Whenever you see a function that could be simpler, rewrite it. Period. This is a pretty interesting idea, especially when it comes to maintenance, since we know how code bloats.
Practice 7: Pair programming
In an XP environment, two fully qualified programmers work at a single computer. One implements and types, the other audits. Each takes turns on the hot seat.
I can hear the howls of protest. This holds little attraction for an anti-social reprobate like myself, but there's plenty to justify the idea. Code inspections use teams of four or five people to review the code, and are proven to be up to 20 times more efficient than traditional debugging for finding problems.
XP goes against the mainstream of software engineering thought by coding and reviewing-by means of pair programming-all at once. There are no meetings, no slowdowns awaiting an inspection team. It's an active and dynamic process. I'll admit some reservations here, but do enthusiastically support pair programming over the more usual “Joe retires to the corner to wrestle alone with the function” approach.
Practice 8: Collective ownership
No one has exclusive ownership of any portion of the code base. If you see a chance to add value to any part of the project, you are required to do so, even if someone else wrote that section. This is another scary idea, since, generally, no one understands a function like the original author. But the test suite removes much of the risk, since any change is immediately verifiable.
Practice 9: Continuous integration
Integration is generally a bad thing, since it defers testing until other components are available. In XP, and other development approaches, we integrate daily, even hourly. It's the only real way to ensure changes to one section don't break something else. We code and test, test and integrate, relentlessly.
Continuous integration is a superset of the testing practice. Its role is to ensure that, in addition to unit tests, we're evaluating the entire system as it exists at the moment.
Practice 10: 40-hour work week
Superheroes in the movies never tire. But in real life, heroics generally lead to exhausted, error-prone engineers. Yet overtime is not always unavoidable, so the XP rule is that you can't work two successive weeks of OT. Cool.
Practice 11: On-site customer
The customer doesn't show up just once in a while for a demo. He's an integral member of the development team who works all week long with these engineers. When questions arise, he's there with an answer. At each small release, a real live human, one who will use the system, evaluates the project. This is an interactive role, since customers need the developer, not as guides, but as companions. For only when the project actually does something will the customer know what he really wants. The crucible is deployment: real use by real people.
Practice 12: Coding standards
All code meets an agreed-upon standard. This critical step makes collective ownership possible, since we're using a common style. Standards reduce obfuscated software and tricky, confusing constructs.
In surveys I've done, few companies use XP in embedded work. But among the handful I've talked to who are using it, all are wildly enthusiastic.
Is XP the answer? Of course not. It does, however, embody some very powerful ideas. A wise developer always studies software engineering, stealing the best ideas from each new approach. Stasis is death.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Jack founded two companies specializing in embedded systems. Contact him at .