New debug tools give unprecedented continuous real-time access to on-chip data.

In the days of large scale integration (LSI) and discrete components, designing a system required the developer to specify and interconnect many devices. Debugging was simpler because developers had access to almost every signal at every point in time. Today, very large scale integration (VLSI), mixed analog/digital technology, and the pervasive system-on-a-chip make the design task easier, but debug has become difficult thanks to the invisibility of signals buried deep in these chips.

Until recently, designers have had to settle for partial snapshots of system activity as revealed with in-system emulators or JTAG scans. These debugging approaches, however, prove less than adequate in real-time systems, especially when an error-causing event happens between these periodic snapshots. To address this issue, chip manufacturers have developed new trace-based debugging schemes that continually provide high visibility into critical system elements and pathways. These tools are useful for both debugging and system optimization.

Tools widely available today make it possible to catch 95% of system bugs. An in-circuit emulator is effective for solving basic software issues, and a software simulator gives a good idea of what's happening inside a chip. The emulator, though, is invasive and impacts how the system performs, while a simulator can't predict or respond to real-time events. Thus, these tools aren't effective at helping uncover complex, intermittent, and context-sensitive real-time bugs. It's a slow process to detect scheduling issues or race conditions between events, intermittent glitches, and problems due to a stack overflow, runaway code, false interrupts, or other unpredictable system crashes.

For some types of system problems, these tools are adequate, although debugging can be tedious. As a sort of system sleuth, the developer can take a guess as to what caused the problem, go to critical sections of code, and set a group of emulation breakpoints that hopefully will isolate the problematic event. When the code reaches the breakpoint condition, it stops the CPU, and the designer can store system information so it can be examined.

After evaluating this snapshot, the designer will have to take an educated guess as to what could have caused the problem and will have to do so without the help of any historical data that shows the system's status leading up to the event. Further, each breakpoint might be associated with several thousands of lines of code, and it's that one particular problematic line that needs to be identified.

After evaluating the situation when the error occurred, another set of breakpoints needs to be set to repeat the process. By trial and error, the issue might eventually emerge after tedious hours of unearthing the possible problems and trying to reconstruct the events that led to the actual error. Only then can exhaustive testing ensure that the real source of the problem has been uncovered.

In some cases, this approach is simply inadequate. As noted earlier, debug tools are invasive; that is, their existence affects system performance. When debugging aids are removed, whether in hardware or software, the system may perform differently and the problem may no longer exist--or other new ones might arise.

Further, if the CPU is stopped during a debug session, it completely changes the way a real-time system operates. With breakpoints, the designer may be able to examine the effect of the error condition, but the cause of the error might have happened long ago and what caused it may be masked by several other system events that have already occurred. It may even be an unusual combination of circumstances that leads to that particular failure.

Trace into the past
All developers face these frustrating situations when there's insufficient information to quickly find the source of a system bug. However, now chip makers and debug-tool suppliers have worked together to provide a new type of debug aid known generically as a trace tool. It provides two major benefits. First, it stores a historical record of key system events continuously in real time without having to stop the system CPU. Second, it is so non-intrusive and thus doesn't have any effect on system performance.

A trace tool, such as the one shown in Figure 1, consists of a CPU with on-board triggering logic and data collection along with a high-speed interface to move the data to emulation hardware. That hardware continues to serve in the role it does in a conventional debug session, controlling the processor and monitoring its status--except in trace mode it controls the trigger logic, not the processor core itself. A benefit of this architecture is that the emulator doesn't steal any processing cycles from the device. Besides controlling the device, the emulator can be used for profiling purposes, such as counting the number of occurrences of an event or for synchronizing the operation of multiple devices.

In a trace tool, the emulation hardware also retrieves the stored program information and can be viewed from a software-debugging environment running on a PC. The on-chip debug unit enables high visibility into key system data pathways and its sophisticated triggering, which is controlled from the PC environment, and enables any condition of interest to be isolated. This thereby locates where and why complex, intermittent problems occur.