Troubleshooting real-time software issues using a logic analyzer

This logic analyzer technique is a power-tool for the embedded software engineer's toolbox.

Click here for more content from ESD March 2012.

While many embedded software issues can be debugged using either print statements or symbolic debuggers (such as JTAG tools), there are many instances where neither of those methods can help. For example, suppose the problem is in an interrupt handler that is executing ten thousand times per second. It's not practical to put print statements as that will ruin the real-time execution. Adding breakpoints to monitor the interrupt will likely break functionality.

Another example is synchronization and timing errors in real-time systems. Developers regularly add mutexes, semaphores, condition variables, or other mechanisms to guard shared resources. But how do you test and debug an issue with these mechanisms? Can you tell whether or not a race condition or priority inversion is actually occurring? How do you track down a glitch that only happens randomly? Can you identify the root cause of a particular real-time constraint not being met?

To troubleshoot these types of real-time issues, the logic analyzer, which traditionally is used by hardware engineers to debug their digital circuits, can be the software engineer's best aide. It provides a new power-tool for your software debugging toolbox. It enables tracing through a program with microsecond resolution and can actively be used to monitor drivers, interrupt handlers, operating system, or real-time application code that has critical time constraints.

Dave Stewart
ESC DESIGN West 2012 speaker logo Dave Stewart is one of the nation's leading experts in architectures and implementation for embedded real-time systems. In his many years of experience, previously as a consultant and now as director of systems integration at InHand Electronics, Dave has regularly been involved in the final stages of system delivery. He credits the techniques described in this article as “the primary difference between being an average engineer who has difficulty finding closure because there is always a handful of nasty lingering bugs to fix, and the master troubleshooter who finds the root cause of the final issues and resolves them to be able to say 'Done.'” This article is based on a class he taught at ESC Boston 2011.

ESC classes:
Dave Stewart will teach four classes at ESC/DESIGN West in March, 2012.

ESC-210: Solving Real Problems that Required Hiring a Consultant
2:00 PM–3:00 PM March 27

ESC-223: Costly Mistakes of Real-Time Software Development
3:15 PM–4:15 PM, March 27

ESC-313: Remotely Troubleshooting Embedded Systems using High Quality Log Files
2:00 PM–3:00 PM, March 28

ESC-316: Achieving Real-Time Performance with Linux/Android or Windows CE
3:15 PM–4:15 PM March 28

Use promo code UBMDESIGN for 20% off upcoming ESC classes. For more information about these and other sessions, go to www.ubmdesign.com/sessions/esc

The logic analyzer is not the holy grail of debug tools by a long shot. In fact, this method is significantly more challenging than using either print statements or symbolic debugging. I'm still waiting to have a system with a button that says “Debug Me”–click it, and your problem is found and fixed. But until that tool exists, you may need to go deep into the trenches, namely the lowest levels of code where print statements and symbolic debuggers cannot do a good job, to find the root cause of a problem. The logic analyzer allows the software engineer to do that. The methods I describe in this article are not the easiest to learn and are extremely difficult to master. However, if you do learn them and effectively use them, it could make the difference between you forever being an average embedded systems engineer and becoming an expert.

The techniques I present here should supplement your current debugging tool box. If you spend an unsuccessful day or more trying to track down a problem with other tools, don't spend more time using the same methods. Instead, try the techniques in this article.

The methods can be used on almost any embedded architecture. I have used these techniques on small 8-bit and 16-bit microcontrollers with single-thread execution as slow as 8MHz and using only 2K RAM. The same techniques have been used equally effectively on multi-threaded complex ARM architectures such as the XScale, OMAP, and DaVinci processors with 512MB RAM and 1GHz clock speed. Although the setup is different in each case, the fundamentals of the logic analyzer methods remain the same.

The key to understanding the benefit of this logic-analyzer approach is to recognize that a software troubleshooter is a detective. A debug tool only provides clues. The better the clues, the faster the problem can be found, and thus the quicker it can be fixed. The logic analyzer takes more setup time than print statements or a symbolic debugger, but the quality and quantity of the clues can be significantly better.
Debug macros for printing
The use of macros to standardize print statement debugging is both useful to know and represents the basis for logic analyzer debugging macros.

Listing 1: Typical 'print statement' debugging.

void myfunc(void) {    int x,y;    printf("Start of myfuncn");    x = read();    printf("x = %dn",x);    y = f(x);    printf("f(x) = %dn",y);    etc.    printf("End of myfuncn");}

Consider the procedure when using print statements to debug code. A make-shift code segment is shown in Listing 1 , with print statements added to trace the flow of the code. This would produce the following output (with assumed values for x and y ):

Start of myfuncx = 10f(x) = 24End of myfunc

While such code will help trace the internals of myfunc() , it's rather inefficient to type in all those print statements, and the amount of information provided in each statement is minimal. What if these lines of code were just a handful out of thousands of debug statements? Tracing back to the original source code could also be time consuming.

Listing 2: Macros to standardize print statement debugging

#define DEBUG_WHERE() printf("[%s/%u] %s()n",__FILE__,__LINE__,__FUNCTION__)#define DEBUG_INT(_var) printf("[%s/%u] %s = %dn",__FILE__,__LINE__,#_var,_var)

A simple solution is to abstract print statement debugging into macros. Listing 2 shows two macros that we can use. With these macros, the debug code can be changed to that shown in Listing 3 .

Listing 3: Using debug macros.

void myfunc(void) {    int x,y;    DEBUG_WHERE();    x = read();    DEBUG_INT(x);    y = f(x);    DEBUG_INT(y);    etc.    DEBUG_WHERE();}

The resulting output would be:

[file.c/3] myfunc()[file.c/5] x = 10[file.c/7] y = 24[file.c/9] myfunc()

When such code is intermixed with thousands of debug statements, it's still possible to easily follow the code, as filenames, function names, line numbers, and values of variables are each clearly marked. Because macros are used, it's also easier to use shortcuts or cut-and-paste to add the debug statements to your code, thus also making it more efficient.

Suppose we want to print the variables in hex instead of decimal. We can create alternate macros, as shown in Listing 4 .

Listing 4: Defining additional print statement debug macros.

#define DEBUG_HEX8(_var) printf("[%s/%u] %s = 0x%02Xn",__FILE__,__LINE__,#_var,_var)#define DEBUG_HEX16(_var) printf("[%s/%u] %s = 0x%04Xn",__FILE__,__LINE__,#_var,_var)

If we replace lines 5 and 7 with DEBUG_HEX8() and DEBUG_HEX16() macros respectively, it results in this change to the outputs:

[file.c/5] x = 0x0A[file.c/7] y = 0x0018

There are many different ways that macros can be used to standardize debug print statements, but I won't cover them here, since our focus is on debugging with the logic analyzer.

The constraint with the print-statement method is that you might be able to add maybe 50 to 100 debug statements per second to your code before it breaks functionality, or maybe just a handful per second before it affects real-time performance. Using the logic analyzer, it's possible to get thousands of debug statements per second with negligible impact on the system's real-time performance. Furthermore, the data is time-stamped and can be visualized as timing diagrams, enabling rapidly identifying anomalies in the code.The concept of logic-analyzer debugging is similar to using these DEBUG macros, except that the macros will be defined to send codes to a logic analyzer instead of print statements to a console.

Before going into detail on logic-analyzer macros, I'll describe the logic analyzer and hardware setup needed to support this type of debugging.

Logic analyzer
A logic analyzer is a hardware test instrument that is used to monitor digital signals. Typical logic analyzers have anywhere from 32 to hundreds of digital channels. For software debugging, we generally only need 16 to 24 channels, and speeds under 100 MHz. So there is no need to get high speed or maximum number of channels.

The lower-cost analyzers can be obtained for under $500; they're simply a pod that plugs into the USB port of a PC, and software on the PC is used to visualize the captures. At the other extreme are some very powerful analyzers that can cost $20,000 or more. While the high-end analyzers have many attractive features that can speed up troubleshooting, a low-cost one is sufficient, at least for starters.

The one so-called luxury of a logic analyzer that is highly desirable is the memory depth. Memory depth represents how many samples can be stored in the logic analyzer before its memory fills up. For example, if sampling at 1 million samples per second, an analyzer with 32-K depth will fill up in just 32 ms, while an analyzer with 4-M depth can run for four seconds. The latter is extremely beneficial, because it allows you to watch the system, have a couple of seconds to manually trigger after you observe the failure, and know that the signals have been captured when the problem occurred. If you only have 32 ms, it's not possible to manually observe a failure, and thus it's more difficult to get the problem captured.

Other features that can be helpful but aren't necessary are good search filters, state machine triggers, and ability to compress captured data so that samples are only stored when the signals actually change, and thus longer traces can be obtained.

Every logic analyzer has at least two views for captured data, a timing view and a state view, as Figure 1 illustrates. The timing view is used to quickly visualize patterns, identify anomalies, and zoom in to points of interest in a data capture. The state view is primarily useful with more advanced troubleshooting, in which the data is downloaded to a PC and software is written to automatically parse the data and extra information such as execution time, checking for real-time deadline errors, and validating that no race conditions occurred. How to automatically parse a logic analyzer's data is a more advanced topic that I won't cover in this article.


Click on image to enlarge.

Hardware setup
To use a logic analyzer to troubleshoot software, you must connect it to the embedded system under test. Unfortunately, at least for existing hardware platforms, this is sometimes the biggest challenge.

The goal is to identify eight unused digital outputs from the embedded system that can be used to send codes from software to the logic analyzer. Sometimes this is easy. There might be an unused 8-bit digital I/O port that can be used for this function. In other cases, there might be eight spare pins, but because they're scattered on various I/O ports, they're not as easy to write to. Nevertheless, as long as they're free, that is good enough.

If there aren't at least eight available pins, you may need to scrounge for them. For example, suppose a system has an SD card, but the SD card is not used when the failure to be debugged happens. The command, clock, four data lines, card detect, and write protect lines can potentially all be used to provide 8 bits; they just need to be reconfigured in software.

If the target system has an FPGA (or CPLD), it may be possible to use eight pins from that device and send a command to the FPGA using whatever communication means already exists between those two devices, then allow the FGPA to output the digital signals.

The next challenge is to hook up these eight outputs to one of the ports on the logic analyzer. Most logic analyzer probes can connect directly to 0.1-inch pins and include clips that allow attaching onto smaller pins. If necessary, a hardware technician can add wires to the embedded system under test to bring out these eight lines in such a way that the logic analyzer can more easily be connected to it.

If the problem that needs to be debugged includes some kind of I/O, such as push-button keys, a controlled digital output, or a serial link, each of these signals should also be connected to other channels on the logic analyzer.

Figure 2 shows a block diagram of the setup. The I/O connections on channel 2 are optional.



When designing hardware for a new real-time embedded system, a design-for-test requirement should be to have eight dedicated digital output pins available specifically for software's use, and to have these brought out to a connector for easy connection to a logic analyzer. At InHand Electronics, we typically place a high-density connector on the board with all the signals that we may want to hook up to the logic analyzer for software debugging, so that board real estate used for software debugging is minimized. We have a separate break-out board populated with 0.1-inch pins for each signal that makes it easy to connect a logic analyzer. An example is shown in Figure 3 ; a cable is used to make the connection from the system under test to the breakout board.


Click on image to enlarge.

Once the hardware is setup, the next step is to define macros to output codes to the logic analyzer.

Defining LDEBUG macros
The first step is to create the LDEBUG_HEX8() macro (or alternately, an inline function) to send an 8-bit code to the logic analyzer via the digital outputs identified above. The actual definition of the macro depends on the processor, hardware setup, and which outputs were selected. An additional macro or function, LDEBUG_INIT() , might need to be defined to initialize the selected pins as digital outputs.

Listing 5 shows four examples of defining the LDEBUG macros, extracted from real target platforms that I've used in the past. In Listing 5a , a TI MSP430 had Port 4 reserved for this LDEBUG function. The port is initialized to output, and writing to the output register sends the 8-bit value to the processor's port four pins.

Listing 5 Examples of defining macros for logic analyzer debugging.
(a) LDEBUG macros for TI MSP430 Port 4

 uint8_t *LDEBUG_dout8      = (uint8_t *) 0x001D;  // reg addr  uint8_t *LDEBUG_dir8       = (uint8_t *) 0x001E;  // reg addr  #define LDEBUG_INIT ()      *LDEBUG_dir8 = 0xFF    // output  #define LDEBUG_HEX8 (_h8)   *LDEBUG_out = (_h8) 

(b) LDEBUG macros for PXA270 using GPIOs 64 thru 71.

 uint32_t *LDEBUG_set8 = (uint32_t *) 0x40E00020; // GPIO2 set uint32_t *LDEBUG_clr8 = (uint32_t *) 0x40E0002C; // GPIO2 clr  uint32_t *LDEBUG_dir8 = (uint32_t *) 0x40E00014; // GPIO2 dir  #define LDEBUG_INIT () *LDEBUG_dir8 |= 0x000000FF  #define LDEBUG_HEX8 (_h8) {                    uint32_t h32 = (uint32_t) hex8;           uint32_t sg2,cg2;                         sg2= (  h32  & 0x000000FF);               cg2= ((~h32) & 0x000000FF);               *_gpsr2 = sg2;                            *_gpcr2 = cg2;                      } 

(c) LDEBUG macros for PXA270 using 84, 85, 93, 94, 116, 106, 107, and 108 as D0 thru D7 respectively.

 static volatile uint32_t *_gpdr2 = (uint32_t *)0x40E00014;// GPIO2 dir  static volatile uint32_t *_gpsr2 = (uint32_t *)0x40E00020;// GPIO2 set  static volatile uint32_t *_gpcr2 = (uint32_t *)0x40E0002C;// GPIO2 clr  static volatile uint32_t *_gpdr3 = (uint32_t *)0x40E0010C;// GPIO3 dir  static volatile uint32_t *_gpsr3 = (uint32_t *)0x40E00118;// GPIO3 set  static volatile uint32_t *_gpcr3 = (uint32_t *)0x40E00124;// GPIO3 clr     // set directions  #define _MASK2 0x60300000  #define _MASK3 0x00101C00  #define LDEBUG_INIT () {*_gpdr2 |= _MASK2; *_gpdr3 |= _MASK3;}  #define LDEBUG_HEX8 (_h8) LDEBUG_Hex8(_h8)  void LDEBUG_Hex8(uint8_t h8) {     uint32_t h32 = (uint32_t) hex8;     uint32_t gr2,gr3,ngr2,ngr3;         // don't write to registers yet; we want to write to registers         // as quickly as possible, to minimize race conditions.     gr2=(((h32&0x0C)<<(29-2)|((h32&0x03)<<(20-0)))); // D3.D2|D1.D0     gr3=(((h32&0xE0)<<(10-5)|((h32&0x10)<<(20-4)))); // D7.D6.D5|D4     h32 = ~h32;    ngr2=(((h32&0x0C)<<(29-2)|((h32&0x03)<<(20-0)))); // D3.D2|D1.D0     ngr3=(((h32&0xE0)<<(10-5)|((h32&0x10)<<(20-4)))); // D7.D6.D5|D4     *_gpcr2 = ngr2;     *_gpsr2 = gr2;     *_gpcr3 = ngr3;     *_gpsr3 = gr3;  }

(d) LDEBUG macros on a system with an adequately-programmed FPGA connected to memory bus.

 #define LDEBUG_HEX8(_h8)  fpga_Write(3,_h8)

A slightly more complex macro is shown in Listing 5b . This example shows the macros for a Marvell PXA270, which has separate set and clear registers. In this example, GPIOs 64 thru 71, which are bits 0 through 7 on GPIO Port 2 of the processor, are used.

A third, much more complex scenario happens when scrounging bits, as shown in Listing 5c . This example is also for a PXA270, but in this case, the specific GPIOs that were available are GPIOs 84, 85, 93, 94 on GPIO Port 2, and GPIOs 116, 106, 107, and 108 on GPIO Port 3.

The fourth example (which is what is used on Fury in Figure 3 ) is shown in Listing 5d . This definition is clearly the simplest but requires that hardware was designed specifically to accommodate this feature. An FPGA is connected to the processor's memory bus, and the FGPA's application programming interface is used to abstract writes to the register. This example assumes the FPGA is configured to output any value written to register 3 to the 8-bit digital output.

Regardless of the complexity of the macro, once the macro is defined, all remaining usage is simple by calling LDEBUG_HEX8() , and the code passed as an argument can be captured by the logic analyzer.

Verifying with the setup
Before troubleshooting a real issue, test the macros and verify the setup using the test function provided in Listing 6 . This is the equivalent of getting "Hello World" program working as a first step on a new platform.

Listing 6: Function to test logic analyzer debug macros

#include "LDEBUG.h"void LDEBUG_TEST (void) {    LDEBUG_HEX8(0xFF);    LDEBUG_HEX8(0x01);    LDEBUG_HEX8(0x02);    LDEBUG_HEX8(0x04);    LDEBUG_HEX8(0x08);    LDEBUG_HEX8(0x10);    LDEBUG_HEX8(0x20);    LDEBUG_HEX8(0x40);    LDEBUG_HEX8(0x80);    LDEBUG_HEX8(0x55);    LDEBUG_HEX8(0xAA);};

Execute the function on the target platform. It doesn't matter if this function is executed as a standalone application, or is integrated into the initialization code of the full system. As long as it gets executed and the test pattern it generates can be captured on the logic analyzer, it's good enough.

Setup the logic analyzer to trigger on Bit 7 (of the 8-bit digital output port) transitioning from a 1 to a 0. Run the logic analyzer and wait for trigger. Then execute the code. Consult your logic analyzer's technical documentation for help in setting up such a trigger.

If all goes well, the logic analyzer will get triggered when the code is executed, and the timing view on the logic analyzer screen will be the step pattern as shown in Figure 4 . If this doesn't happen, review all steps above and troubleshoot the macros or hardware setup. If you don't capture the pattern at all, it could be an issue with initialization of the debug I/O ports as output, a problem with writing to these I/O ports, or a channel or trigger setup issue on the logic analyzer. If you get some kind of pattern but it doesn't match what is shown, review each individual signal connection.


Click on image to enlarge.

Once you're able to capture this test pattern, you're ready to use this technique for debugging.
Debugging basics
What makes logic analyzer debugging more difficult than print statements or symbolic debugging is that the data captured is very cryptic.

Listing 7: Example of using print debug macros.

void get_vector() {    pos = get_position();    vel = get_velocity();    acc = get_acceleration();    tim = get_timestamp();    DEBUG_HEX16(pos);    DEBUG_HEX16(vel);    DEBUG_HEX16(acc);    DEBUG_HEX16(tim);};

Listing 7 shows a code segment using the print DEBUG macros. Output from executing this code would be:

[file.c/9] pos = 0x0063[file.c/10] vel = 0x1D18[file.c/11] acc = 0xFD4E[file.c/12] tim = 0x0DE5

Listing 8 shows the same code segment instead using the LDEBUG_HEX16() macro, which is built upon the LDEBUG_HEX8() macro created previously.

Listing 8: Example of using logic analyzer debug macros

#define LDEBUG_HEX16 (code,value)   {    LDEBUG_HEX8(code);                  LDEBUG_HEX8(value >> 8);            LDEBUG_HEX8(value & 0xFF);      }uint16_t pos, vel, acc, tim; void get_vector(vector_t &data)     {    pos = get_position();    vel = get_velocity();    acc = get_acceleration();    tim = get_timestamp();    LDEBUG_HEX16(0x40,pos);     LDEBUG_HEX16(0x41,vel);    LDEBUG_HEX16(0x42,acc);    LDEBUG_HEX16(0x43,tim);};

The logic analyzer capture for the first LDEBUG operation is shown in Figure 5 . The first code seen is the 0x40, followed by the 16-bit data, which was split as two separate 8-bit values, first the MSB, then the LSB.



Zooming out on the logic analyzer, as shown in Figure 6 , we see all four of the LDEBUG_HEX16 commands. The codes (i.e. 0x40 through 0x43) are used to produce an easily recognizable pattern on the logic analyzer. Once this pattern is identified, the numbers in between represent the data variables. By correlating the codes and data values back with the source code, it's possible to determine the value of each variable.


Click on image to enlarge.

Suppose this code was executing a thousand times per second, and the logic analyzer captured one second of data. This particular sequence would then be repeated a thousand times during the one capture. It's obviously not possible to look at all thousand items manually.

You can use two approaches to wade through thousands of data elements. One approach is to switch to state view and save the data to a file. You can then write a program or script that parses through the file and looks for specific issues. This technique is more advanced; I don't describe it here. The other method is much less formal and often just as effective, and thus will likely be used much more often. I call this technique "visualizing performance," as described below.

Tracing execution
You can use the logic analyzer to view precisely what portion of code was executed during any interaction of a thread or driver or during any execution of an interrupt handler. At the same time, the logic analyzer captures timing measurements of each code segment within that function.

Consider the make-shift function shown in Listing 9 and assume that it's part of an interrupt handler in which print statements are not viable and inserting breakpoints would break the functionality. How do you know which branches are taken when?

Listing 9: Tracing code using logic analyzer debug macros.

void myfunc(int x) {    int y,z;    LDEBUG_HEX8(0xB1);    y = f(x);    if (y < 0) {        LDEBUG_HEX8(0xB2);        Do stuff;    } else {        LDEBUG_HEX8(0xB3);    }    z = g(y);    if (z > y) {        LDEBUG_HEX8(0xB4);        Do stuff        LDEBUG_HEX8(0xB5);        Do more stuff        LDEBUG_HEX8(0xB6);    } else {        LDEBUG_HEX8(0xB7);        if (z > 0) {          // ERROR!          LDEBUG_HEX8(0xBE);        }        LDEBUG_HEX8(0xB9);        do stuff        LDEBUG_HEX8(0xBA);        Do stuff        LDEBUG_HEX8(0xBB);    }    do_calc(x,y,z);    LDEBUG_HEX8(0xBF);}

By instrumenting the code with LDEBUG_HEX8 statements scattered, it's possible to determine both the precise execution trace and execution time measurement, as illustrated by the corresponding logic analyzer capture shown in Figure 7 . For example, during this specific iteration, we know that y was not less than zero, because the code 0xB3 showed up, not the code 0xB2. The error of z > 0 also did not occur, because the code 0xBE was not in the capture.


Click on image to enlarge.

From the log, we could also tell that the function do_calc(x,y,z) took approximately 23 µs to execute, based on the difference between 0xBB and 0xBF. A more precise measurement would be possible by either zooming into the time view, or switching to the state view and looking at the relative time stamp.

As a personal convention, I find it helpful to separate the 2-digit hex codes as "0xYZ", where Y =unique number for the function, and Z is a location within the function. In Figure 7 , Y=B. Z is 1 for entry, F for a normal exit, E for an error exit, and 2 thru D for 12 other trace points within the function. Thus 0xB1 represents entry into the function, and 0xBF represents normal exit. Z =0 is reserved to use as a specific trigger point when necessary. As you gain experience, however, you can choose any personal conventions that you find helpful. Having conventions makes it easier to parse the rather cryptic output of the logic analyzer.
Visualizing performance
A very hard--yet rather common--real-time issue that needs to be solved is when one or more threads aren't meeting their timing requirements. Quite often, the thread that is failing is not the culprit. Rather, it may be some other thread that has higher priority or using resources needed by the failing thread that is at the root of the problem, but knowing which thread to troubleshoot can be difficult to find.

The logic analyzer can be used to visually spot performance anomalies. Very often, real-time issues show themselves first at the very low level of execution. By instrumenting code with LDEBUG tracing statements and simply monitoring execution from a bird's eye view, you can acquire a trained eye to zero in on parts of the code where there is a potential anomaly. Then, look at the logic analyzer captures more closely to determine whether or not any of those anomalies are real problems.

An example is provided in Figure 8a . The top diagram is the bird's eye view. This is a 2-second logic analyzer capture of a system that's been instrumented with LDEBUG macros in various key parts of the code.


Click on image to enlarge.

In this sequence of diagrams, the green is I2 C communication, the white is the 8-bit debug codes, while the yellow are the per-bit timing diagrams. While the per-bit diagrams are not necessary when looking closely at the 8-bit codes, they're helpful in these birds-eye views to show patterns. (Note that to conserve space in this article, only four of the bits are shown; that is generally sufficient to see the patterns.)

Quite simply, most real-time errors show up as pattern anomalies when looking at a logic analyzer capture. So let's focus on some of the anomalies we can see.

To the left of the top image, the red ellipse is circled around an anomaly. Note how much more space there is between adjacent pulses. There is a performance issue here that should be investigated. By zooming in on this area, we can look at the individual hex codes to see what part of the code was executing, and how long each code segment was taking. However, for this example, let's instead focus on a different part.

The green represents I2 C communication, which seems to come in bursts of five messages at a time. However, near the middle, they stop. Is this normal? Or is this indicative of the problem?

To better understand this problem, we can zoom in to the rectangular portion, which is the last burst of communication before they stop.

At the new zoom level (Figure 8b ), we see more granularity, both with the I2 C communication as well as with the patterns generated by the LDEBUG statements. For the I2 C, we could see now that in the burst of five I2 C messages, the fifth one takes longer than the others. This is probably normal, if there are more bytes transferred. Nevertheless, when trying to find a glitch or performance anomaly, it's always worthwhile to double check that your observations match expectations.

At this level, we begin to see details of a potential performance anomaly. Remember it is always important to realize that any anomaly may in fact be normal; calling it an anomaly simply means it merits a closer look.

The specific anomaly to note is that the code 0x46 is always captured before an extended break. Is this normal? If the convention is followed that the second digit is an F at the end of any function, this means we have something blocking or taking really long right after code 0x46 is printed. This might not be a problem; for example, it might be a call to a function in another module which is not instrumented with LDEBUG statements. Or it might point to a real issue; perhaps the code is calling a function taking much longer than expected, or unexpectedly blocking when trying to acquire a resource.

Suppose the performance anomaly with the 0x46 is normal. We can zoom in again. In this case, we select the part of view in which the bit-patterns is different from everywhere else. It is surprising how often using that simple heuristic--a difference in the bit pattern--is enough to zero in quickly on real problems.

In the zoom of that area (Figure 8(c) ) we see two more potential anomalies, namely extended periods following each of codes 0xAF and 0x9F. The question then, is what happens immediately after those codes are sent to the logic analyzer? Repeating this approach, taking a few minutes per anomaly, most performance issues in the system can be identified within a few hours.

Figure 8(d) shows on additional zoom, which is the I2 C communication. Some logic analyzers will also show the hex values as well as protocol start and stop bits for serial communication, as shown in Figure 9 .


Click on image to enlarge.

Simply logical
The logic analyzer techniques described in this article provide you one additional, powerful tool to use to debug real-time software issues.

Consider this metaphor: if you needed to cut a branch from a tree, you would use a branch cutter that takes a second to pull out of the shed, "snip", and it is done. You would not go get a chainsaw for such a simple job. However, if you needed to cut down a large tree, the branch cutter is useless, and a chainsaw is much more powerful than an axe or saw. The logic analyzer is like that chain saw. It's powerful, but more difficult to set up and more awkward to use. Use it when necessary, but still hang on to all your other tools and use those when they can get the job done more efficiently.

Dave Stewart is the director of Research and Systems Integration at InHand Electronics, a developer of small, low-power electronics solutions for handheld and wireless device designers. He has significant experience in numerous market segments, including mobile computing, wireless sensor networks, aerospace, industrial control, consumer electronics, transportation, physiological monitoring, and environmental monitoring.

This content is provided courtesy of Embedded.com and Embedded Systems Design magazine.
See more content from Embedded Systems Design and Embedded Systems Programming magazines in the magazine archive.
This material was first printed in March 2012 Embedded Systems Design magazine.
Sign up for subscriptions and newsletters.
Copyright © 2012
UBM--All rights reserved.

1 thought on “Troubleshooting real-time software issues using a logic analyzer

  1. We implemented this logic analyzer idea into software. Your application is writing a log file that is visualized using tool called TimeToPic. Naturally it has more overhead since writing trace to file takes more time than accessing I/O but as general timin

    Log in to Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.