Have you ever been lost? And I don't mean just turned around, but really lost , and not the pleasant sort of “wander around and enjoy the scenery” lost. Software development projects can get lost, too. Whether your project starts out as a routine run up a familiar highway or an intrepid voyage into the heart of darkness, there will be days (sometimes many days) where you will be lost, or will think that you are lost.
There are many ways to become lost, but the visible manifestation of “lost” in a software system includes bugs, performance problems, unexpected and not-to-spec behavior. While lost, you are vulnerable because you are busy dealing with navigation problems, and aren't ready to respond to positive or negative new developments. It is a rare project that doesn't go through at least one stage of disorientation.
The speed with which you find your bearings and get the project un- lost is what separates successful projects from the rest.
So how do you do that?
Knowing that there's a good chance that one day you'll walk into the office and find your project adrift, you should prepare a set of tools.
How fancy those tools are will depend on a lot of things, but the most important factor will be how precious your time is to you. Is it merely inconvenient to be lost, like someone whining in the back seat for you to pull over? Or could it mean a missed delivery or demo, and your project being mugged by internal politics?
Here are ten things to look for when putting together your toolset.
1. Usable daily. If you don't know how to use them, then they can't help you. Whether you get your daily navigational fix from a simple serial port and a couple of LEDs, or via the best IDE and system analysis tools available, you need to get it daily.
If you and your team aren't practicing with your toolset every day to solve small problems, then you're not developing the expertise that you will need when a more difficult problem comes along. (And if you don't have daily small problems, then you must not be doing much development, which is a whole different problem altogether.)
2. YOU ARE HERE. Before you can figure out where to go, you must find out where you are. So whether break out your astrolabe or your GPS unit to do it, you need to know a few fundamental facts. Traditionally, for C/C++ code this has meant a stack trace and basic variable visibility — and if your system has little memory and a simple processor core, this may still be sufficient.
However, in recent years many applications have moved on from simple bare-metal, single-thread applications. If your current project has multiple cores, multiple threads, multiple applications, or is merely very large and complex, make sure that your tools can keep up.
3. Beyond the simple trip to the grocery store. So you have a simple, effective system for troubleshooting problems, and you use it regularly. But big, intractable problems are out there, and they can derail your project for weeks or even months.
Before one of these hits, think about whether your toolset can scale from regular daily use to the big, difficult problems. Don't mistake what I'm saying — if you really have a tough bug, no one tool is going to make the diagnosis obvious except in hindsight.
You need engineering talent and good instincts, but you also need tools which are capable, extensible, and reliable enough so that you can sharpen them into the domain-specific analysis tool that is going to help you hunt down your problem. Look for an API — scripting or otherwise — and ways to link your software tools into other specialized hardware and software analysis systems.
4. Not just big freeways, but dirt roads, too. Once you know where you are (see item 2), you will often find that you're at a dead end, far down a winding road. The problem which put you on the wrong path is, in the words of a famous software commentator, that left turn you missed back at Albuquerque.
In the world of embedded software, these kinds of problems can be due to stack overflows, deadlocks and other improper use of synchronization, to name just a few. To save yourself a lot of time retracing your steps, look for tools that can tell you at a glance if you have strayed from the beaten path, and how to get back.
5. How did I get here? Similarly, the best tools can not only tell you where you are, but also give you a full rundown of what happened for the previous milliseconds (and seconds, or minutes).
There are several common techniques to do this, from simple application-level or kernel-level instrumentation recorded in a buffer, to slightly more sophisticated streaming of that information over a network or serial port.
Finally, sophisticated trace analysis tools can turn the millions of lines of instruction-by-instruction system trace from a trace port analyzer into a high-level overview of exactly what happened in your system.
6. Why aren't we there yet? Performance problems can be one of the most pernicious schedule killers, and if you don't have a good performance analysis system, you will be limited to the slow and error-prone process of eyeballing the system and guessing about what's eating up processing power.
Better still, if you have a performance analysis system that is simple and easy to use during daily development, then you can make suitable performance tradeoffs early, rather than dealing with them late in the development process when they are typically more expensive to fix.
7. Have we been this way before? Any sane development effort has a test suite for assuring that the system continues to work as daily changes are integrated. And unless you and your customer are both crazy, there should be substantial testing of your system before it ships.
Once you get serious about the process, a key question you will have is how thoroughly your test suite covers the code that will ship. It turns out that most capable debug systems can answer this for you, either through instrumentation or via information captured from hardware trace ports.
8. Okay, I took a wrong turn. What are my options? Once you've isolated a problem and you think you understand the mechanism, it's time to consider how to fix it. If you're lucky, you have a well-maintained codebase and you know it like the back of your hand.
Most developers are not so lucky. They have code that has been passed back and forth, no one fully understands why or how it works, and certain parts have suffered that benign neglect of being just a little bit understaffed over a long time.
Navigating and fixing such code safely can be a time-consuming, error-prone challenge unless you have capable source analysis tools. These tools can cross-reference the codebase for you, allowing you to easily navigate between your targeted fix area and related or affected sites.
9. Reporting home from distant lands. The development process rarely ends neatly at the point where units begin shipping to customers. Nearly all projects have a beta period of some sort, and certain types of projects spend most of their lives in a “delivered but remotely maintained” existence where the original developers (or their descendants) could be recalled to fix the system at almost any time.
It's in these situations that a field debug and update system shows its value. Field debug, typically implemented with a small logging, instrumentation and reporting module, helps reduce the cost of this expensive class of problem.
Field update systems, in concert with system design which can accommodate it, allow you to safely and securely update a system, sometimes even without taking the system out of service for a measurable period of time.
10. “Please execute a legal U-turn” Like a GPS navigation system, really good debug setups can also tell you when you've made a mistake, before you spend hours dealing with the consequences. The best examples of this are static and dynamic error analysis tools.
Static analysis tools integrate with your compiler, finding problems ranging from uninitialized variables through array bounds overrun, and even including multi-threaded deadlock checks.
Dynamic analysis code is linked into your system, and looks for array bounds violations, stack overflows, and other memory corruption occurring at runtime. Both tools can alert you to a potential navigation problem before you get too far afield.
Anderson McKay is engineering manager at Green Hills Software.