Troubleshooting real-time software issues using a logic analyzer

David B. Stewart, PhD, InHand Electronics, Inc.

February 27, 2012

David B. Stewart, PhD, InHand Electronics, Inc.

This logic analyzer technique is a power-tool for the embedded software engineer's toolbox.

Click here for more content from ESD March 2012.

While many embedded software issues can be debugged using either print statements or symbolic debuggers (such as JTAG tools), there are many instances where neither of those methods can help. For example, suppose the problem is in an interrupt handler that is executing ten thousand times per second. It's not practical to put print statements as that will ruin the real-time execution. Adding breakpoints to monitor the interrupt will likely break functionality.

Another example is synchronization and timing errors in real-time systems. Developers regularly add mutexes, semaphores, condition variables, or other mechanisms to guard shared resources. But how do you test and debug an issue with these mechanisms? Can you tell whether or not a race condition or priority inversion is actually occurring? How do you track down a glitch that only happens randomly? Can you identify the root cause of a particular real-time constraint not being met?

To troubleshoot these types of real-time issues, the logic analyzer, which traditionally is used by hardware engineers to debug their digital circuits, can be the software engineer's best aide. It provides a new power-tool for your software debugging toolbox. It enables tracing through a program with microsecond resolution and can actively be used to monitor drivers, interrupt handlers, operating system, or real-time application code that has critical time constraints.

The logic analyzer is not the holy grail of debug tools by a long shot. In fact, this method is significantly more challenging than using either print statements or symbolic debugging. I'm still waiting to have a system with a button that says "Debug Me"--click it, and your problem is found and fixed. But until that tool exists, you may need to go deep into the trenches, namely the lowest levels of code where print statements and symbolic debuggers cannot do a good job, to find the root cause of a problem. The logic analyzer allows the software engineer to do that. The methods I describe in this article are not the easiest to learn and are extremely difficult to master. However, if you do learn them and effectively use them, it could make the difference between you forever being an average embedded systems engineer and becoming an expert.

The techniques I present here should supplement your current debugging tool box. If you spend an unsuccessful day or more trying to track down a problem with other tools, don't spend more time using the same methods. Instead, try the techniques in this article.

The methods can be used on almost any embedded architecture. I have used these techniques on small 8-bit and 16-bit microcontrollers with single-thread execution as slow as 8MHz and using only 2K RAM. The same techniques have been used equally effectively on multi-threaded complex ARM architectures such as the XScale, OMAP, and DaVinci processors with 512MB RAM and 1GHz clock speed. Although the setup is different in each case, the fundamentals of the logic analyzer methods remain the same.

The key to understanding the benefit of this logic-analyzer approach is to recognize that a software troubleshooter is a detective. A debug tool only provides clues. The better the clues, the faster the problem can be found, and thus the quicker it can be fixed. The logic analyzer takes more setup time than print statements or a symbolic debugger, but the quality and quantity of the clues can be significantly better.