Non-intrusive debug -

Non-intrusive debug


Debugging represents a very significant part of an embedded software development project. All developers have their own favorite approaches and each one has its strengths and weaknesses. A key issue is how intrusive the debug tools are – i.e. the extent to which debugging software affects the functionality of the code. This is not a black and white issue, as a number of factors and priorities need to be considered. This article outlines different approaches to debugging, from the perspective of intrusion, and also considers the implications with respect to code optimization.

What is intrusive debug?
There are two parameters that characterize any piece of code: size and speed. In broad terms, intrusive debug may be anything that affects either of these factors.

A number common debug techniques involve the incorporation of additional code into the application, the sole function of which is to facilitate debug. This affects the overall memory footprint and may have an effect on the execution time of the application code. There may also be unexpected side-effects caused by the use of this code.

If a system has plenty of memory – over and above the needs of the application code – the inclusion of extra debug software should not be a problem. However, many embedded systems are designed to have enough memory, but only just enough, and there is no provision for adding more just during debug time. Under these circumstances, the additional memory hit would be very problematic.

An example of an intrusive debug technique, that is very widely used and exhibits this issue, is the temporary addition of printf() calls to to track the path through some code. Such instrumented code might look something like this:

...av = 99;while (av != 0){	av = fun();	printf("av = %dn", av);  // debug}printf("loop exitn");  // debug...

This approach works well enough, but has some drawbacks:

  • The entire application needs to be rebuilt every time you want to change the debug setup.
  • The printf() function needs to send its output somewhere. Figuring out where to send the text to and setting up the library function to do this may be challenging.
  • This function is quite large, as it has way more functionality than required for debugging. Unless the application code uses printf() (which is unlikely), inclusion of the library function will have a drastic effect on the memory footprint.
  • Many library functions are non-reentrant and their use in a multithreaded system may be problematic.

Some of these problems may be addressed by using custom functions, written specifically for doing debug and designed to be small and simple. A possible set might be:

  • debug_print_text() – outputs a text string
  • debug_print_number() – outputs an integer in a suitable base
  • debug_print_newline() – aids display formatting

This is for C. For C++, the functions might be overloaded.

The code example might then look like this:

...av = 99;while (av != 0){	av = fun();	debug_print_text("av = "); 	debug_print_number(av);	debug_print_newline();}debug_print_text("loop exit");debug_print_newline();...

The intrusion into embedded code’s execution performance – i.e. how fast it runs – can be even more of a problem than memory footprint, as its effect may be much subtler. Outputting “trace” data, as shown in the above example, takes time and that execution time may be significant.

Another situation when execution time may be adversely affected is when using an RTOS. Ironically, although such an operating system is intended to help the developer build real time applications, the built-in (normally optional) debug facilities, whilst very useful in some situations, may affect the real time execution profile. In other words, the code might run differently (from a time perspective) depending on whether the debug facility is activated or not. This is, by definition, intrusion. 

When does intrusion not matter?
Given that sufficient memory is available, a debug technology that increases the image size is unlikely to affect the execution of the application code by that fact alone. The only intrusion will be on the execution performance. So, time-critical (i.e. real time) code is the most sensitive. Code with no particular time constraints – pure logic/algorithms – will be unaffected. Hence, the intrusion level of debug does not matter at all when checking out code logic.

Next page >>

Low intrusion debug environments
There are a number of debug technologies which offer low or very low levels of intrusion:

In-Circuit Emulation (ICE) – This technology was, some years ago, the de facto standard for any embedded development. It has limited availability nowadays, as CPU clock speeds and chip packaging technology have made it decreasingly viable. The idea was simple enough. The CPU was replaced by a probe, which was controlled by a debugger. To the rest of the circuit, the probe was indistinguishable from the actual microprocessor. This enabled debugging of unmodified code at close to full speed.

JTAG – As ICE became less viable, it was gradually replaced by debuggers connected, by means of a suitable adaptor, to the JTAG port on a target board. This technology allows code to be run at full speed, until it hits a breakpoint, when the CPU is paused. Similar interruptions occur if a variable is monitored.

Hardware instrumentation – A step on from JTAG is for dedicated debug hardware to be incorporated into the chip (or on the board). This opens up possibilities for real time (potentially non-intrusive) trace.

Debug monitor/agent – Although this technology introduces a memory footprint overhead, it can be a very cost-effective debug strategy. Typically, it uses a pre-existing communications channel to connect to the debugger. A key benefit may be task-/thread-aware debug, where, for example, a breakpoint may just stop a particular task, leaving the rest of the system uninterrupted

Simulation – There are a number of simulation technologies of interest to embedded software developers. Each one offers a combination of execution speed and time precision. The big benefit of a simulator is that debug functionality is, in effect, added to the CPU transparently and has no impact on the code execution. A cycle/time accurate simulator will execute code much more slowly than real time, but, because its internal timing is accurate, very time critical software can be debugged/verified.

Software instrumentation – The idea of adding extra debug code was discussed earlier (using printf() etc.). There are, however, more sophisticated implementations offering much lower intrusion in high performance systems. LTTng is an open source tracing framework for Linux, for example.

Apart from debugging during development it is prudent to add additional self-test code to the application. This need not be intrusive at all. This topic will be covered in detail in a future article.

True non-intrusive debug
Sophisticated debugging with absolutely no intrusion is almost impossible, but there are a few possibilities:

Logic analyzers and oscilloscopes – These instruments are generally thought of as being tools for hardware development, but they can be an aid to finding software bugs. Essentially they enable the state of a number of logic signals to be captured – a snapshot – according to one or more specific criteria.

Hardware instrumentation – As mentioned above, the addition of specific hardware for debug tends to be very non-intrusive. It is possible for the intrusion to be eliminated entirely in some cases.

FPGA – A variation on the hardware instrumentation theme is to add the debug functionality to an FPGA design. Actually, even if the final design will not be realized in an FPGA, this might be viable for prototyping and, hence, for debug.

Emulation – Hardware designers may use an emulation system to verify their designs. This is like “simulation on steroids” and, hence, offers a potentially non-intrusive debug possibility with an execution speed close to real time.

Optimized code debug
An extreme example of debug intrusion is the need to debug some code, that is, by necessity, different from the code that will actually be shipped. This can occur as a result of code optimization. All modern compilers optimize code and most do a very good job. Embedded tools tend to offer a higher degree of control, enabling the code to tuned for speed or size (at least). A problem is that some debuggers are confused by optimized code and, in certain cases, just refuse to allow debugging to take place. This sounds unreasonable, but makes sense because the correspondence between the source code and the executable may be fuzzy or lost entirely.

For example, this code:

char a[4];for (i=1; i<4; i++)	a[i] = 0;

should yield a loop, which could be stepped though during debug. However, an optimizing compiler might generate a single 32-bit move or clear instruction. Debugging would be confusing, even though the code is doing its job perfectly.

There are two ways to address this problem:

  • Just ship non-optimized code – this is unlikely to be acceptable.
  • Use tools that allow debug of optimized code. The initial stages of debugging should be done without (too much) optimization. It can be increased for some final checks before shipping.

Intrusive debugging is not a precise term. Different kinds of intrusion may occur and some may matter more than others in any given situation. Intrusion is not necessarily bad. The key skill for a developer is to understand how their chosen debug technology (or technologies – more than one may be used through the life of a project) works so that its validity, in a specific context, may be assessed.

Colin Walls has over thirty years experience in the electronics industry, largely dedicated to embedded software. A frequent presenter at conferences and seminars and author of numerous technical articles and two books on embedded software, Colin is an embedded software technologist with Mentor Embedded [the Mentor Graphics Embedded Software Division], and is based in the UK. His regular blog is located at: He may be reached by email at

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.