Hunting for Bugs in Real-World Applications…More Than a Child’s Game

With debugging features such as complex breakpoints, trigger sequencerand processor state storage now available in smaller 16-bitmicrocontroller architectures, the hunt for the pesky bugs inreal-world applications, under real-world operating conditions, becomesmore than just a child's game.

Debugging is an essential part of an embedded developer's dailylife. As applications become more complex, the debug process becomesmore challenging and time-consuming. The drawback of using simulatorsolutions for debugging has is an inaccurate reproduction ofapplication operation conditions.

Many simulators only simulate the CPU core and a few peripherals,but provide features such as code coverage, profiling and conditionalbreakpoints. While this may suffice for testing complex software-onlyalgorithms, embedded applications normally involve a fair amount ofhard-software interaction.

Real-time events occur, caused by externally connected circuitry andmicrocontroller (MCU) integrated peripherals such as analog-to-digitalconverters (ADC) and timer modules.

Also, the MCU can be part of a more complex system, and beinterfaced with external control and data storage ICs or used incombination with a digital signal processor (DSP) to provide supportfunctions. Software for communicating with these external ICs cannot beeasily debugged and verified in the “clean room” environment of a CPUcore simulator.

One very popular way to address these drawbacks is through the useof in-circuit emulators (ICEs). These combine the benefits of asimulator with the ability to run software on actual hardware inreal-time.

This is achieved by replacing the MCU on the target board with achip-package adapter that is “hooked up” to a box of complex electroniccircuitry that mimics the MCU's behavior and provides debugfunctionality.

Disadvantages of off-chip ICE
While usually very feature-rich, however, these emulators areexpensive, cumbersome and do not provide a 100 percent accurateenvironment. This is especially true when it comes to sensitive analogperipherals such as A/D and D/A converters, comparators, oscillatorsand voltage references.

Given these disadvantages, embedded emulation, which adds debugfunctionality to the MCU, has become very popular. Through embeddedemulation, the host PC used for debugging can directly interface to theon-chip emulation logic using a serial interface like JTAG.

The application code runs on the MCU as it would without a connecteddebug interface. The advantage lies in that because the softwarealready is developed under real-world operating conditions, noadditional hardware testing is necessary after software development,

Today embedded debug functionality is common even in smaller 16-bitand 8-bit controller architectures. But, most implementations onlyoffer a basic set of functions like memory access, CPU executioncontrol and hardware breakpoints.

This leaves much to be desired when compared to a full-blownin-circuit emulator (ICE). The latest 16-bit modern MCU models go onestep by providing a greater set of features to bringing functionalitycloser to what one is used to getting from an ICE.

In many cases, this can save developers the expense of an ICE whiletackling even the toughest debug scenarios.

Using an on-chip emulation module
To illustrate the capabilities and advantages of such built-incapabilities, use the enhanced emulation module (EEM) of the MSP430 MCUas an example. As shown in Figure 1,below , this module incorporates the following functional blocks:basic trigger inputs, trigger combination logic, trigger sequencer,trigger action, state storage and clock control.

Figure1. MSP430 Enhanced Embedded Emulation Module

For debugging with the such an on-chip module, at least one of theeight available basic trigger inputs must be configured. Commonly,on-chip embedded emulation implementations only allow setting triggersthat monitor the CPU address bus (MAB) and stop the program executionat a given memory location.

Triggers found in some modern-day MCUs such as those in the aboveexample, however, allow significantly more complex setups. In additionto the address bus, any triggers can be configured to monitor the CPUdata bus (MDB), the internal CPU registers and some of the processorcontrol signals.

Furthermore, it is possible to apply bit masks to address and databus triggers to isolate certain values of interest if desired. Using anadditional constant, comparison options such as “equal to,” “not equalto,” “less than,” and “greater than” can also be applied. Thecombination of these features enables a sophisticated breakpoint setupversus the standard “break-at-address” debug functionality.

The trigger combination logic forms complex triggers out of thebasic trigger inputs that are available. Trigger events output by thisblock are user-definable, logical AND-combinations of the basic triggerinputs. For example, by combining an address with a data bus trigger, amemory location can be monitored for certain values that are written orread there.

This complex trigger event can then be used to directly stop aprogram execution or generate a state storage event. Prior to thatfunction, trigger events can be processed by the trigger sequencer,which is a built-in state machine that has four states.

Programmable transition conditions made out of incoming triggers areused to generate transitions between the states. When the sequencerreaches the final event state, the MCU can be configured to stopprogram execution and/or generate a state storage event.

As mentioned earlier, any of the complex triggers can also be usedby the state storage unit. This is a circular buffer that holds up toeight entries, each of which is a snapshot of the 16-bit address bus,16-bit data bus and some important CPU control signals from the timethat the trigger occurred.

It can be seen as a simple trace buffer, which is capable ofcapturing status information without affecting the real-time behaviorof software running on the MCU. System snapshots can be taken on abasic trigger event, either by a combination of triggers, by thesequencer output or simply on every CPU clock cycle.

A useful add-on that comes with the EEM on some modern-day MCUs isthe clock control unit. Various peripherals such as an A/D converter,LCD driver, timer, and serial communication module can be driven by oneof the three available internal clock tree signals.

When stopping program execution, the clock control unit allows aper-module basis configuration of which peripheral continues to getclocked, and for which the clock will be stopped when the processor ishalted during debugging. If the emulation module simply stops allclocks (e.g., when a breakpoint is reached), unwanted side effectscould occur such as lost communication characters or erroneous A/Dconversion results.

Another possible implementation approach is to continue clockingperipheral modules on emulation stop. A possible problem with thissolution, however, is that certain modules such as a timer couldpermanently set interrupt flags even with the CPU halted, which couldmaking single-stepping through the source code quite challenging. Usingthe clock control block, the developer can limit the clock distributionto only the modules that are vital to the application.

EEM Triggers forming complexbreakpoints
Popular on-chip emulation modules offer the ability to set basichardware program breakpoints. However, debug scenarios can besimplified in many cases if there is a way to add a condition to thebreakpoint. Let's look at an example: an embedded application writtenin C implements a complex state machine. The current state is stored ina global variable, and gets updated in various places throughout thesource code. The problem is that the application exhibits unexpectedbehavior; state '3' is entered under the wrong conditions.

Now, the challenge is to find the section of source code that causedthis unexpected transition. A complex EEM module trigger can now beused to stop program execution whenever the value '3' is written intothe state machine variable 'StateVa r'. Thiscomplex breakpoint is a logical AND-combination of two basic EEMtriggers, generating a CPU stop event. Figure2 below shows a simplified block diagram of how this complextrigger can be implemented.

Figure2. Combining Basic Triggers

One basic trigger is configured to monitor the device's internaladdress bus (MAB) for the state machine variable 'StateVar 'address and the CPU write access control signal. The other trigger isused to monitor the internal data bus (MDB) for the value of 3.

Configuring the EEM logic with this set-up allow the programexecution to stop exactly at every instruction that writes the value'3' into 'StateVar '.This way, the code section that caused this write access can beidentified easily. The same mechanism can be applied not only for RAMaccess, but also for Flash and peripheral module access as well.

This complex breakpoint can be further configured by using the bitmask feature, which is implemented into every basic EEM trigger. Thisoption can be described best by taking a look at a real-world example.A customer using an ultra-low power MCU for a portable sports watchapplication reports a problem: the general purpose port pin 3 of port 1gets set unexpectedly. In the application, port 1 was used to controlvarious external circuitries and, therefore, was accessed multipletimes during the code execution.

The task here is to find out which CPU instruction and correspondingline of source code caused this unexpected port pin modification. Acomplex EEM breakpoint similar to the one just described can be used tomonitor write accesses to the port 1 output register.

Additionally, by using the bit-mask feature, bit 3 could beisolated. This isolation is achieved by programming 0x0008 as the maskAND-value, and 0x0008 as the trigger compare value. The MDB EEM triggerhardware now performs a bit-wise AND-operation prior to everycomparison.

This way, the CPU is left running and the program execution stopsexactly at every line of code that set bit 3 in the port 1 outputregister. After executing the program a couple of times, a buggyC-expression is identified as the cause of the unexpected behavior.

An awful bug – caught!
A common mistake in embedded applications is stack overflow. Most MCUarchitectures allocate space for stack in RAM. However, RAM is alimited and shared resource also used by other variables and programelements.

A common MCU practice is to set the stack pointer to the top of theRAM space during program initialization. For C programs, depending onthe development tool used, the linker allocates a section of RAM with adefault size of 0x50 byte for stack use.

When developing software and adding an increasing amount of globalvariables, the linker will eventually report there is insufficientmemory available at compile time. Now, the problem here is that if thestack size has not been carefully specified, the reserved space may notbe sufficient for the application.

When using dynamic memory allocation at run-time and recursiveprogramming techniques, space can be easily consumed. Also non-ideal,real-world events such as a bouncing button or other input signalgenerating nested interrupts can push the stack pointer to the edge,and ultimately over the edge. With no stack pointer run-time checking,a developer runs the risk that the stack will grow into the range wherevariables reside.

If this happens, vital application data can be corrupted withhard-to-explain effects ranging from strange program behavior to atotal software crash. A mechanism that immediately stops programexecution when the stack pointer (SP) leaves the designated RAM areacould easily identify the problem.

Figure 3 below shows anexample of how to set-up a complex-breakpoint that monitors an MSP430MCU SP. This and other screenshots are taken from the IAR EmbeddedWorkbench V3.20A IDE.

Figure3. Stack Overflow Detection

With this particular device, 0xA00 ” 0x50 = 0x9B0 was used as the lower limit. Should this breakpoint now stop programexecution, the stack contents can be examined to determine the rootcause of the overflow. If some amount of identical values can be foundthere, this might be an indication for a bouncing port interrupt issue.

Another form of complex breakpoints that can be formed are rangebreakpoints. A range breakpoint monitors the address or the data buswithin or outside of a specified range. The debugger uses two EEM basictriggers that are combined internally and are set to monitor the samebus.

One trigger is configured for a “less than” comparison mode whilethe other trigger is configured for a “greater than” comparison mode.The program execution will stop only when these two conditions arefulfilled. For instance, a range breakpoint can be used as monitor toensure that no CPU instruction fetch occurs outside of the programmemory.

This can help in debug scenarios in which the program counterbecomes corrupted due to an incorrectly calculated indirect jump. Thiscan also be combined with read/write modifiers, to protect memory areasfrom overwriting.

Making triggers smart
Another real-world example of the power of the EEM involved a digitalstill camera application, which used a MCU in conjunction with a DSP toperform support functions such as keypad scan, power management andreal-time clock. Both processors were connected through a serial UARTlink and the DSP sent multi-byte command sequences to the support MCUto request the current battery status. In this example, however, theMCU failed to react to the given command.

The developer faced this problem: how to introduce breakpoints intothe serial reception interrupt service function to determine theproblem root cause without disturbing the real-time behavior of theapplication. The DSP software timed out, the communication wasinterrupted and the data exchange never reached the interesting point.

The solution lay in feeding three complex triggers into the triggersequencer and using a setup like the one shown in Figure 4 below , to halt programexecution exactly after reception of the last byte of the 3-byte longcommand sequence.

Figure4. Sequencer Control

From now on, single-stepping showed path of the CPU through theprogram, and revealed why the command was never executed.

What is my program doing?
The state storage block is another powerful EEM component. Whendebugging code and having the state storage module configured tocapture on every CPU instruction fetch cycle, the storage buffercontains a history of the last eight assembler instructions executed(8-level deep history).

When stopping program execution manually or through a breakpoint,this list provides useful information as to what occurred prior tostopping execution. When configuring this feature to collect data on abasic or combined EEM trigger event, it is possible to log only certainop-codes such as “jump” and “branching” instructions. This gives apowerful instruction trace that allows inspection of the most recentprogram flow.

Because the state storage buffer is accessible through JTAG withoutinterfering with the CPU and target application operation, anotheruseful function is available: the implementation of a real-time watch.This watch can be helpful in many debug scenarios, such as a motorcontrol application, for example.

When breakpoints are inserted into the application to halt theprogram and read out variables through normal watch windows, thecontrol algorithm is disrupted, which could possibly cause a breakdownof the mechanical installation.

Combining one of the MCU's EEM triggers with the state storagefeature, it is possible to realize a real-time watch to monitorapplication variables without modifying the program code itself. With atrigger set to monitor a certain memory location containing thevariable of interest for write access, state storage events can begenerated.

The data-bus value transferred into the buffer now always containsthe last updated value of this variable. Figure 5 below shows a screencapture of the State Storage Window as implemented in the IAR EmbeddedWorkbench.

Figure5. State Storage Window

In this example, a global variable that contains the current motorspeed is located at address 0x200 and monitored for write accesses.With the motor control algorithm running in real-time, the statestorage window is refreshed automatically and displays the most currentmotor speed in the “Data bus” column, and all without affecting theapplication execution.

The screenshot shows an increase in motor speed. Even though thismechanism of using data-bus values as a real-time watch is limited to16-bit, it should be sufficient for most purposes.

Clocks under control
The EEM clock control block provided help during development of anotherembedded application. One task of the MCU was to drive a MOSFETswitching transistor that was used in a boost-converter circuit togenerate a high-voltage. The transistor was connected to a timer PWMoutput with the duty cycle controlled by a software algorithm.

With the application running, the engineer needed to modifyparameters that were located in RAM. Before the application was stoppedto modify the RAM contents, the engineer disabled the “Stop Timer_B clock on emulation stop “option in the clock control configuration dialog.

If manually stopping program execution the duty cycle of the outputtransistor was, for example, around 20%, this duty cycle would havebeen maintained because the timer was still running and generating theproper PWM waveform. If the timer would have been stopped, this couldhave caused the output to permanently drive the switching transistor,overloading and possibly damaging the circuitry.

With the EEM features presented here, debugging becomes more advancedand much easier when compared with some common basic embedded emulationimplementations. Another advantage is that there is no additionaltool-cost such as for an ICE, as all emulation functions are built intothe CPU core as a standard feature set.

With EEM, in-system debugging of complex real-world signalprocessing applications that handle sensitive analog signals becomespossible. If needed as an additional measure, galvanic insulation canbe easily added to the few signal lines that are used to communicatewith the CPU core (such as a JTAG interface) by using an isolated debuginterface.

This option would not be easily possible with the 64+ signal linesof a comparable ICE. However, certain premium debug features like codecoverage, profiling, and a deep trace buffer will still require an ICE.Future extensions such a profiling, improved real-time system accessand a deeper trace buffer are in development, further reducing the gapto an ICE and supporting the embedded developer's needs.

Andreas Dannenberg is an MSP430Applications Engineer for Texas Instruments in Dallas, Texas. During his three years at the company, he has workedon MSP430 software development and hardware design, as well as C2000and C6000 DSP application development.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.