Troubleshooting real-time software issues using a logic analyzer
This logic analyzer technique is a power-tool for the embedded software engineer's toolbox.
|Click here for more content from ESD March 2012.|
Another example is synchronization and timing errors in real-time systems. Developers regularly add mutexes, semaphores, condition variables, or other mechanisms to guard shared resources. But how do you test and debug an issue with these mechanisms? Can you tell whether or not a race condition or priority inversion is actually occurring? How do you track down a glitch that only happens randomly? Can you identify the root cause of a particular real-time constraint not being met?
To troubleshoot these types of real-time issues, the logic analyzer, which traditionally is used by hardware engineers to debug their digital circuits, can be the software engineer's best aide. It provides a new power-tool for your software debugging toolbox. It enables tracing through a program with microsecond resolution and can actively be used to monitor drivers, interrupt handlers, operating system, or real-time application code that has critical time constraints.
The logic analyzer is not the holy grail of debug tools by a long shot. In fact, this method is significantly more challenging than using either print statements or symbolic debugging. I'm still waiting to have a system with a button that says "Debug Me"--click it, and your problem is found and fixed. But until that tool exists, you may need to go deep into the trenches, namely the lowest levels of code where print statements and symbolic debuggers cannot do a good job, to find the root cause of a problem. The logic analyzer allows the software engineer to do that. The methods I describe in this article are not the easiest to learn and are extremely difficult to master. However, if you do learn them and effectively use them, it could make the difference between you forever being an average embedded systems engineer and becoming an expert.
Dave Stewart is one of the nation's leading experts in architectures and implementation for embedded real-time systems. In his many years of experience, previously as a consultant and now as director of systems integration at InHand Electronics, Dave has regularly been involved in the final stages of system delivery. He credits the techniques described in this article as "the primary difference between being an average engineer who has difficulty finding closure because there is always a handful of nasty lingering bugs to fix, and the master troubleshooter who finds the root cause of the final issues and resolves them to be able to say 'Done.'" This article is based on a class he taught at ESC Boston 2011.
Dave Stewart will teach four classes at ESC/DESIGN West in March, 2012.
ESC-210: Solving Real Problems that Required Hiring a Consultant
2:00 PM–3:00 PM March 27
ESC-223: Costly Mistakes of Real-Time Software Development
3:15 PM–4:15 PM, March 27
ESC-313: Remotely Troubleshooting Embedded Systems using High Quality Log Files
2:00 PM–3:00 PM, March 28
ESC-316: Achieving Real-Time Performance with Linux/Android or Windows CE
3:15 PM–4:15 PM March 28
Use promo code UBMDESIGN for 20% off upcoming ESC classes. For more information about these and other sessions, go to www.ubmdesign.com/sessions/esc
The techniques I present here should supplement your current debugging tool box. If you spend an unsuccessful day or more trying to track down a problem with other tools, don't spend more time using the same methods. Instead, try the techniques in this article.
The methods can be used on almost any embedded architecture. I have used these techniques on small 8-bit and 16-bit microcontrollers with single-thread execution as slow as 8MHz and using only 2K RAM. The same techniques have been used equally effectively on multi-threaded complex ARM architectures such as the XScale, OMAP, and DaVinci processors with 512MB RAM and 1GHz clock speed. Although the setup is different in each case, the fundamentals of the logic analyzer methods remain the same.
The key to understanding the benefit of this logic-analyzer approach is to recognize that a software troubleshooter is a detective. A debug tool only provides clues. The better the clues, the faster the problem can be found, and thus the quicker it can be fixed. The logic analyzer takes more setup time than print statements or a symbolic debugger, but the quality and quantity of the clues can be significantly better.
Page 1 of 6Next >