TRENDS IN DEBUGGING

Alas, but for bugs our systems would ship on time and on budget. But for bugs we’d be heroes, producing quality products each and every time.

More often than not, some 50% of a project’s development time is spent chasing bugs. This appalling number surely indicates a problem with the software design process. But even the best methodologies will never eliminate all defects. Debugging is the dark side of development, where schedules fall apart and management ages prematurely.

Since the dawn of the microprocessor era, vendors have offered a range of tools for exterminating bugs. Surely the biggest change in the history of debuggers for embedded systems was the ’80s-era switch from a low-level hex/disassembled view of the code to today’s ubiquitous and essential source-level view. Those new to the field might find it hard to believe that all debugging once took place in raw hex. That was possible in part because assembly language was de rigeur ; C had not yet made inroads into the firmware arena. Compilers decompose each line of code, scattering instructions in memory in nonintuitive ways. Source debugging became essential.

The second biggest change in debuggers paralleled the change in the nature of the systems we build. Once the focus was on hardware, with a few thousand bytes of code tossed in to make it all work. Now, of course, in most cases hardware is secondary, merely an enabler for huge chunks of firmware. So too for debuggers, which were once boxes with minimal user interfaces; most of the cost and engineering went into the circuits in the tool. Applied Microsystems, one of the first firms in this business, first offered in-circuit emulators (ICEs) that sported a one-line LED display with a hex keypad for data entry. Now their products, and those of other ICE vendors, often look like cigarette packages, and are just as devoid of buttons and displays. The interface lives in the user’s workstation. The software debugger contains most of the system’s intelligence and functionality, with the ICE itself largely just a port into your target system.

So what is a debugger? Is it the software that drives an ICE? Is it the ICE and software together? What about BDMs and other devices that interface to target systems?

Current usage suggests that a debugger is the workstation-based software that provides a GUI-based environment for finding bugs in your code. The debugger itself is almost always distinct from the hardware or software that interfaces to your target hardware (or to a simulation engine for those working without hardware).

While debuggers were once produced by vendors selling emulators and other target-interface tools, they are now largely created by companies specializing in the software side of the problem, usually by compiler companies. There’s logic to this: compilers have become so complex that only the vendor truly understands the relationships between generated code and the original source. Variable typing and the extra layers of abstraction posed by C++ exacerbates the problem.

An almost incestuous and mind-boggling array of strategic alliances has sprung up between companies that sell compilers, ICEs, BDMs, RTOSes, and every other tool we use. Lines of specialization are vague at best, though it isn’t unusual to see alliances based on processor usage. For instance, Diab-SDS (itself the result of a recent merger) provides compilers and matching debuggers mostly for Motorola processors. Virtually every company that sells a target interface device like an ICE offers the SDSI debugger as one option.

From a vendor perspective, this separation of software debugger from target system interface makes a lot of sense. Most embedded tool vendors are relatively small operations with only dozens of employees. You can count on your fingers the number of vendors with more than a hundred employees. The astonishing complexity of processors, compilers, and other tools suggests that small vendors simply cannot be experts at every aspect of firmware troubleshooting. It makes good business sense to specialize in one or two aspects of the bug-fixing, and to partner with other experts who address other parts of the problem.

From a user standpoint, the benefits are profound but mixed. We want the best, most reliable, and feature-packed tools our engineering dollars can buy. Get a killer software component from vendor A and an awesome target interface from vendor B and surely we’ll have the best possible development environment. The trouble, of course, lies in ensuring that these tools work seamlessly together, and getting quality support rather than high-tech finger pointing.

Talking about software debuggers is impossible without some sort of context. The wonderful features we expect come from the software, but the limitations in the tools’ capabilities come largely from the target interface. One debugger might drive a half dozen different sorts of target probes, but exhibit vastly different performance based on which tool it drives.

ICEs

The earliest debugging tools were emulators; old-timers reminisce about Intel’s famous ’70s-era “Blue Box,” the quintessential prototype of all modern ICEs.

An ICE electrically (and sometimes mechanically) replaces the CPU in your target system. It becomes the world’s most expensive processor, one that metaphorically lifts the hood on the CPU, giving you a clear view into internal system operation. The ICE is essentially a connection to the target’s processor, with a comm link back to the workstation and software debugger. The debugger can query the ICE about register contents, memory values, and the like, as well as set breakpoints and implement other debugging features.

Typical troubleshooting features in an ICE include breakpoints, the backbone of all traditional debugging. An ICE can uniquely set hardware breakpoints that do not affect your code, and complex “if-then” breakpoints that trigger activity based on what your system does in real time. Emulation RAM replaces target ROM to allow you to download and test code quickly.

Probably the most important feature an ICE offers—one that is almost unique to this tool—is real-time trace, which captures your program’s execution without stealing cycles. Coupled with the emulator’s triggering logic, trace can find tough bugs without ever slowing the code down. In today’s real-time systems, breakpoints are often deadly, so trace may be the only useful debugging feature.

Theoretically (and traditionally), an ICE provides every possible debugging resource. In the last decade, though, we’ve seen some feature backpedaling in an effort to keep prices down. A baffling fact of this industry is that companies have a low tolerance for pricey firmware development tools (despite the astronomical cost of writing code), so ICE vendors have struggled to provide a reasonable mix of troubleshooting features at low costs. This means hardware breakpoints are often replaced with rough software equivalents (often not a problem), trace depths and widths may be compromised, and exotic features like performance analysis are relegated to extra, external tools.

BDMs

Twenty years ago, pundits predicted the death of the ICE, back when a 10MHz clock rate seemed breathtaking. Vendors have defied all such expectations, today providing tools that run reliably at rates an order of magnitude higher. But high speeds mean high prices, which are intolerable for many companies, especially those with large development teams.

Chip vendors have long suffered from the (quite reasonable) demands of developers who will not design poorly supported CPUs into their systems. This led to the chip folks underwriting development costs for emulators and other tools, and in some cases, buying complete tool chains for major customers.

Clearly this situation doesn’t delight the accounting types looking at profit and loss figures. Costs, high processor speeds, and modern complex processors that are difficult to emulate led several vendors to create an on-board debug interface, originally called Background Debug Mode (BDM) by Motorola. Variants include JTAG interfaces, BDM+, and a dozen others. All describe similar technologies: the processor itself includes some level of basic debug capability, along with a serial interface dedicated to debugging.

The serial interface provides an independent channel into the processor’s brain. Ideally, no matter what the CPU is up to, a remote debugger can send debug commands to the device.

Essentially the vendors separated “run-control” features from more complex trace and the like. Run-control means the BDM-like devices can do all of the basic low-level troubleshooting things, like examine and alter memory and registers, set software breakpoints, and start or stop program execution.

These run-control debuggers are a boon to developers. All you need is a software debugger and a simple interface between the CPU’s serial debug port and your PC to drive the beast. Hardware costs approach zero. Unlike emulators, which require access to a hundred or more processor signal pins, the BDM’s simple three to five wire connection is mechanically and electrically reliable.

You do give up real-time trace and emulation RAM capabilities, as well as complex breakpoints, timers, and other real-time features. Some vendors (Hewlett-Packard, Applied Micro-systems Corp.) supplement the BDM with other tools (like a logic analyzer) to restore trace. Costs go up, of course, as do features and debug capabilities.

The trend in BDMs is toward more on-chip debug features. The holy grail is extracting trace info from the execution stream, but bandwidth issues preclude ever getting all of that data over a simple serial link. One option offered by some processors is branch monitoring: whenever the code jumps to another context, some amount of address information goes to the debug tools. The future will probably include a mix of tools, from simple low-cost BDMs, to those with some limited trace monitoring, to more expensive and more complete environments that include a trace collection device with the BDM run-control product. You’ll decide just which features you want—and can afford—and will buy an appropriate mix of tools.

ROM monitors

Perhaps the oldest of debugging tools is the ROM monitor, which is nothing more than a hunk of debugging code linked into your firmware. You devote one of your target system’s serial ports to the debugger; this link communicates with the software debugger. A minimal system without an extra port can’t use a ROM monitor.

The ROM monitor is much like a BDM. It offers similar run-control features (but never trace), and, as it uses no hardware at all, can be the cheapest of all tools. A ROM monitor does eat resources, including the mentioned serial port and extra ROM and RAM. However, the memory footprint is usually minor.

A ROM monitor is especially well suited to poorly supported processors, those for which debugging options are few.

Software debuggers connect to other tools as well, such as simulators (which may experience a resurgence since systems-on-a-chip mean that no target hardware is available until late in the project) and ROM emulators. A ROM emulator is a hardware device that plugs into a memory socket and offers varying levels of debug features, from nothing more than a place to download code to surprisingly sophisticated run-control and even trace.

Buying a debugger

In a perfect world we’d select our product’s CPU based on the usual factors, but with a heavily weighted nod to tool support. Unhappily, this approach is rather unusual, and too many projects get into massive trouble because of lousy tools support. It sure would be nice if CPUs that come with poor toolchains died a death of neglect, a Darwinian natural selection process.

Instead we tend to choose the CPU first, and then pick what we hope will be an adequate toolset afterwards. As I mentioned earlier, many vendors specialize in tools for particular processors or family of processors. Use the annual ESP Buyer’s Guide to find a match between CPU and vendors. But be aware that you cannot select a software debugger in isolation; this tool must be carefully matched to all of the others.

The first thing to consider when selecting a software debugger, oddly, is what sort of target interface you’d like to have. We looked at the various options earlier, but this decision comes down to one of desired features vs. price.

Be ready for a bit of sticker shock. That BDM interface might consist of no more than two cheap ICs on a tiny circuit board, but add in a decent source-level debugger and you may spend several thousand dollars. Though software carries no recurring cost, the market for embedded systems tools is astonishingly small. Vendors amortize millions of dollars of software development—writing the debugger—over low volumes. The full time support staff we demand adds more costs.

A business sage suggests that wise people work hard to preserve their options. Try to avoid getting locked into a single solution. If you plan to buy an ICE, select a debugger vendor that supports the ICE and a BDM or ROM monitor, so that if the emulator doesn’t live up to the salesperson’s promises you can quickly swap in the BDM solution without changing the rest of the toolchain.

Reliability—coupled with a strong support staff—is more important than any particular technical feature, yet these are the hardest items to measure. Get references of happy customers. Contact those people, and ask them for further references in an attempt to spread the net beyond what the vendor expects. Check Usenet postings (comp.arch.embedded) for problems and recommendations. If the company has a user’s group, query that organization. Most debuggers connect to a wide range of other things, from compilers to target interfaces to RTOSes. Call these vendors and ask them which tools work best together. Call the support staff with difficult questions. Measure their response time and technical depth.

All reputable vendors offer at least a 30-day money back guarantee. Exploit this. Get the tools early, when there’s still time in the schedule to make mistakes. And then try them! Don’t let them sit on the shelf until the 30-day period expires.

Since the final toolchain will consist of products from a number of vendors, see how well each product works with the others. Some debuggers offer little more than a DOS-like screen for issuing raw target-interface commands. That is, the source-level control may be breathtaking, but a command-line only link to the target interface hardware is endlessly frustrating. Worse, such a crude link often means you won’t be able to use symbolic references when sending debugging commands to the hardware.

If you’re using an RTOS, make sure the debugger is truly and deeply aware of the particular OS’s operation. Today most debuggers provide some level of RTOS-awareness, but often at a relatively minimal level. To combat this potential problem, some RTOS vendors offer their own RTOS-only debugger. Others include a sophisticated debug API that some debugger vendors exploit deeply. Fact is, it’s tough to manage tasks and their contexts unless you’re doing it entirely at the source level, which is almost impossible to obtain unless it’s within the context of the source-level debugger. Pick a debugger/RTOS combination that is seamlessly integrated, and that gives you symbolic/source-level support for tasks and task contexts.

A few years ago, I’d recommend ensuring that the debugger supported enums, structs, and other C language constructs. That support is now universal. The new frontier is C++, which demands adjuncts like class browsers and the like.

Big projects make for unmanageable code. And if you’re working with real object-oriented programs (vendors report that only 15% of firmware developers buying C++ compilers use them as anything more than super-Cs), then class relationships get complex very quickly. Thus vendors offer various sorts of source code engineering tools and features. Class browsers are now common. Other features that help us understand relationships in the code include maps of which function is called from what point, where variables are read or written, as well as include file cross referencers.

Some debuggers offer all or some of these source code engineering tools; at least one company (TakeFive Software, www.takefive.com ) sells such a tool that runs as an adjunct to your debugging environment. Regardless, on a big project we tend to spend too much time digging through listings or blindly searching files; it makes a lot of sense to invest in a tool that shows relationships graphically and easily.

If we’re spending some 50% of a project debugging, we interact with no tool more than the software debugger and associated target interface. You could spend months—even years—staring at its windows. Just as minor bad habits bloom into marriage-killers, small glitches and awkward features will drive you crazy over the course of months using the debugger. When you buy a car, you check the feel of the driver’s position and ensure the controls are conveniently located. Figure on spending some time checking out the debugger’s look and feel, and don’t hesitate to evaluate products from several vendors.

Advice for debugger users

Every tool represents some sort of compromise. Understand the strengths and weaknesses of your toolchain to avoid the worst frustrations.

First assess your target interface strategy. If you’re using an emulator, look deeply into how the unit connects to the CPU. Modern surface mount processors have whisker-thin leads that frustrate reliable connections. Nothing is more annoying than the unreliable ICE-to-target interface, the one that shifts due to the slightest air current in the lab. Consider creating a prototype target system with soldered wires to the ICE connection.

Make sure critical signals are totally clean, such as clock, address-latch (on multiplexed buses), interrupts, and all CPU timing ins and outs. Try to design your target so it can run with the emulator’s internal clock. Leave some slop in timing margins because the ICE will require a few nanoseconds of each cycle.

If you’re using a BDM connection, leave yourself an out for dealing with real-time issues. Find a way to connect a logic analyzer or more sophisticated tool to the system. Best of all, select a debugger that drives both the BDM and a trace-like tool, so if you find yourself chasing real-time problems you’re doing so from within the source-aware debugger.

If the processor does have a BDM mode but you’ve decided to use other sorts of tools (such as an ICE or a ROM monitor) then by all means put a BDM connector on your design. The cost approaches zero (especially if you just put the pads on the circuit board without actually loading the connector), yet may save the project if the other tools don’t live up to expectations. Again, if you select a software debugger that drives both the other tool and a BDM, you’ll have no learning curve when the world starts to fall apart.

Remember that the software debugger reloads data from your target whenever you hit a breakpoint or otherwise change the firmware’s context. The data it loads depends on which windows you’ve opened. Displaying, for example, a UART’s data register may clear the data-ready flag when a new byte appears. The act of reading from the target in effect can change the system’s operation, a sort of uncertainty principle of embedded systems.

Some debuggers will use a small set of target resources. RTOS-awareness might mean another monitoring task runs. A ROM monitor requires stack, ROM, and RAM space, as well as a serial port. Some ICEs demand a bit of stack space or other resources. Query your vendor deeply to see what resources you must provide to the tools, and then be quite sure your environment can sacrifice whatever is required.

Consolidate and integrate

The debugger industry is consolidating, a trend bound to continue. Making a complete C/C++ debugger/compiler package is incredibly expensive, so we can expect a few companies to become the preeminent suppliers, with each one focused on particular market niches, probably defined by CPU.

No doubt prices will rise to some extent, though this will be mitigated by increasing functionality as the products include more source code engineering tools.

The BDM-like target interface seems poised to dominate the market, especially for higher-end CPUs. We’ll see clever extensions to the BDM standards to support more trace collection, as well as other external tools that do the same.

Software debuggers will offer new and innovative ways of helping us manage and understand our code. They will also integrate with other packages (like communications), thereby providing us with high-level views of the operation of each chunk of commercial code.

Jack Ganssle is a consulting technical editor of ESP . He conducts seminars on embedded systems and helps companies with their embedded challenges. His “Break Points” column appears monthly in this magazine. Contact him at jack@ganssle.com .