Two relatively new developments in debugging technology are RTOS- or kernel-aware (KA) debugging, and Serial Wire Viewer (SWV). KA is generic and processor independent, while SWV applies specifically to ARM Cortex microcontrollers. This article is specific to the Cortex-M3 and -M4 processors and examines the complementary nature of these technologies. Traditional run/stop debugging has been greatly enhanced in the ARM Cortex-M architecture by the introduction of Serial Wire Viewer and other capabilities of the CoreSight debug architecture.
Kernel Aware debugging views have evolved and matured in the past decade and are now considered indispensable in satisfying the information requirements of software testers for general information on high level RTOS objects. However, a limitation of KA is the necessity to halt the processor to gain information. Without purchasing additional tools, basic evolution of certain important RTOS constructs in real time can be quickly and easily monitored using SWV. The combination of KA and SWV now provides insight into embedded applications that neither method alone was previously capable of providing.
Historical notes on JTAG debugging
In the late 1990s the JTAG (Joint Test Action Group) interface emerged as a replacement for the in-circuit emulator (ICE). The latter was running out of steam as controller clock frequencies increased, with only a grab bag of techniques ranging from blinking LEDs/printf() style debugging through ROM monitors, ROM emulators, and various background debug mode implementations (BDM). The JTAG interface, originally developed for boundary scanning of parts, quickly became the defacto method for hardware debugging in embedded design. It solved the limitations of ICE by incorporating three distinct building blocks:
- On-chip IP blocks that facilitate transmission of debugging information and control to the external debugging system on the host PC
- A debug probe that is connected between the host PC and the target that translates information from the on-chip IP to the communications channel (UART, USB or LAN) on the host PC
- An application on the PC to send/receive information from and control the device under test, and display it visually
Traditional JTAG debugging tools included mechanisms to run the application, run to a specific line of code, step, and hit breakpoints. In some cases, ancillary software extended the number of breakpoints useable for flash memory. Memory locations could be displayed and modified when the application halts. An extension of the on-chip IP called embedded trace macrocell (ETM) allowed instruction and data tracing. This was normally reserved for high-end parts.
Early in the last decade, ARM Ltd introduced a CoreSight specification for ARM Cortex microcontrollers that greatly extended the capabilities of traditional JTAG debugging. However, the focus of this article, the ARM Cortex-M3 and M4 microcontrollers, do not have the full implementation of the CoreSight architecture. This was done to preserve the basic low-cost framework of the Cortex-M family of parts. To retain essential debugging power in return for an attractive cost savings, ARM’s Coresight incorporates the Serial Wire Viewer (SWV).
The practical benefits of ARM CoreSight
There are a number of practical benefits conferred to ARM Cortex-M3 and M4 processors by this development, including:
- Fewer pins required on die than traditional JTAG
- Reporting data in real-time as the application is running without incurring penalties in run-time behavior
- Intrusive but inexpensive printf() style debugging via instruction trace macrocell (ITM) channel
- Uses inexpensive debug probes
- Works during low power modes
- Statistical profiling of applications possible
- Reporting of interrupts and events possible
- Easy timing measurement of code sections possible due to time stamping capability
However, there are a few drawbacks and limitations which must be borne in mind, namely:
- Data transmitted through SWO pin (which is a 1-bit interface) , some throughput limits on information (4-bit interfaces available on certain parts, but cost of debug probe increases sharply).
- Real-time data monitoring is limited by number of comparators on die, typically four
- Instruction trace possible in parts with 4-pin interface, but no data tracing
To compare and contrast a Coresight capable debugging environment with a traditional JTAG run/stop capability characteristic of say, ARM7TDMI parts, examine Figures 1 and 2 below.
Click on image to enlarge.
Figure 1 shows windows such as variables, breakpoints, memory, CPU core and special function registers.
Click on image to enlarge.
Figure 2 shows considerable additional information including:
- Real-time (without core overhead penalties or intrusion) plot of memory locations (X-Y-Z accelerometer variable values in this case)
- Statistical profile of where the application is spending its time
- A printf() output via ITM from the accelerometer servicing tasks
- Time indexed record of all system interrupts and exceptions with their descriptions
- Interrupt and exception statistics
An essential feature of this information is the way that it is displayed. At a glance, developers can quickly visualize the “health” of the application, and address questions such as:
Is the data from sensors evolving in real time as expected when stimulated?
Are interrupts firing on time, and in the correct sequence?
Is the application building up excessive amounts of processor time in certain functions or regions of the code?
These are extremely valuable parcels of information once all the various components of the application, e.g. peripheral drivers, application code, RTOS and possibly TCP/IP stack, embedded flash file system and USB stack, are bound together. Problems in one component of the code can easily be caused by problems in another, seemingly unrelated area. In this case, with raw data efficiently turned into information we see a variety of potential problems can be ruled out with simple and quick visual inspection.
Using an RTOS in embedded applications and its effect on debugging
Including an RTOS in an embedded application has a number of advantages and disadvantages that have been extensively discussed over the years in many articles, whitepapers, forums, and development groups. In recent years, the increasing cost/performance/resource ratios of processors like Cortex-M3/M4 has lowered the barrier to RTOS adoption.
Because the use of an RTOS was problematic in the days of in circuit emulators, in any quarters it is still regarded as an additional layer of difficulty and effort added to the debugging problem, even though contemporary debugging tools such as JTAG and Serial Wire Viewer work reliably with RTOS applications. The correct way to assess the impact of RTOS usage on the debugging problem is to consider what kind of information is required when debugging an RTOS based application. Every RTOS implementation is characterized by the following:
- Breaking the application into discrete tasks managed by the RTOS
- Use of an API for creating, managing and possibly deleting tasks
- Use of various mechanisms for inter-task communication and synchronization
- Use of RTOS provided time management capabilities
When viewed from this perspective, the debugging problem when using anRTOS shifts to a much higher level of abstraction. In traditionaldebugging, the developer typically wants to know precise low-leveldetails such as the value of a variable, contents of a register, bitpattern in an SFR, how the application behaves while stepping, etc.Although these considerations remain, debugging an RTOS requiresinformation such as:
When halted, what are the task states?
Which task is running, which tasks are ready, which are waiting?
If waiting, what resources are tasks waiting on?
How many synchronization objects such as semaphores and mutexes are being used and what is their status?
What events or messages exist, and what are the statuses of these objects?
Which timers have expired, which are still running?
Thisis where a demand for further and better visualization is created, andsatisfying that demand is particularly critical when debugging andconducting integration testing.
The notion of “kernel aware”debugging, that is, RTOS object rather than code object specificinformation, took hold around ten years ago. Initially regarded as agimmick, KA is now widely considered to be indispensable by embeddedsystems developers. The reasons are twofold. First, answers to the abovequestions collectively constitute a high level snapshot of the healthof the application from a perspective of the RTOS constructs whosecorrect interaction is essential to the success of the application.Second, the information desired is buried deeply within the datastructures used during construction of the tasks, services and controlmechanisms.
A developer does not want to have to search out thememory location where, for example, a mutex or a timer timeout flag isstored and display it if this can be avoided. In addition, if using anRTOS in binary-object library form, this is difficult or impossible.Well tested and optimally laid out information found in contemporaryintegrated development environments is usually easier and faster to readas well as being more informative at a glance.
Figure 3 shows a typical set of kernel aware debugging windows.
Click on image to enlarge.
Figure 3 – Kernel aware debugging windows for the embOS operating system
Uniting kernel aware debugging with serial wire viewer
Theprinciple drawback of kernel aware debugging is the same as traditionalJTAG. In order to visualize RTOS specific information such as taskstates, semaphores, messages, and events, the application must behalted. For certain devices under test such as motor controllers, thisis not an optimal strategy. Sometimes it is simply impossible withoutcompromising the hardware. There are two potential solutions to theproblems:
First, limited but often cruciallyimportant information on RTOS aware objects can be made availabledynamically using Serial Wire Viewer. Doing so requires backtrackingslightly from the previous question about whether or not the developerwants to inspect RTOS data structures at a low level and dig outessential elements for survey. When sufficient benefit is derived it isworthwhile for the developer to identify important and informativefields in RTOS specific data structures and configure SWV to reportthese dynamically.
The second alternative involves purchasing anadditional product. Micrium’s uC/Probe is a tool that reports KA debuginformation and other details about the application in real time. Thisrequires a communications channel such as JTAG probe or a USB, LAN, orUART channel, and is intrusive.
Focusing on the first technique of using Serial Wire Viewer, an example of this can be seen in Figure 3 .The window in the top right hand corner is the Serial Wire Data TraceTimeline Graph. This is time evolution of a field in a data structurethat holds the number of messages in a mailbox. Messages are normallyproduced by a task in order to provide information to another task. Thesource task inserts the message into the mailbox, and the destinationtask typically blocks until the message is available, and then extractsand uses that message. Figure 3 shows a regular pattern of productionand consumption of messages. Of course, this only has meaning in thecontext of the design and implementation of the application. Theappearance of such a waveform is no guarantee in and of itself that theapplication is behaving properly, however.
Having said that,suppose that the waveform depicted in the Serial Wire Data TraceTimeline Graph is in fact the desired and correct behavior. If a problemcropped up where there was a disruption in this process, it would beobvious immediately when inspecting the graph. Remember, this data isevolving in real time (and without processor overhead!) so the data canbe monitored while the application is running in its native environment,i.e. on real hardware, interacting with stimuli from the outside world.A disruption in the message traffic could then be correlated exactlywith other behavior. Further, the axis of this display is labeled intime units, so a software tester would know exactly when from testcommencement that the error occurred, which may be another valuable cluein finding the problem.
One of the goals in the design ofAtollic TrueSTUDIO to minimize the number of extra tools that developersneed to do their jobs, such as traditional Code Review tools,interfaces to version control, and bug tracking database servers, aswell as optional modules for code analysis and test automation.
Inan application using TCP/IP, the number of packets being transmitted asa function of time is often of interest. A tool like WireShark iscommonly used for this purpose. This information can tell the testerwhether or not the application is “still talking/listening” or whetheror not hiccups are present in the communications flow. As shown in Figure 4 TrueSTUDIO displays this information using the Data Trace Timeline Graph.
Click on image to enlarge.
Figure 4 – TCP/IP packets displayed as a function of time in realtime
Ata glance, communications rate can be roughly estimated. This is oftenan important metric in TCP/IP applications or services. As Figure 4shows, SWV is often capable of rendering essential information so thatthe tester does not have to shell out of the environment, consultadditional tools like WireShark, and then come back to the environmentto continue testing, debugging and data gathering.
The SerialWire Viewer’s capabilities for exact and accurate monitoring of memoryaccesses (variable updates) can be combined with real-time operatingsystems and other types of middleware to provide a uniquely powerfultool for advanced visualization and debugging of middleware-equippedembedded systems. Being able to visualize the time evolution of specificvariables and other events as the application executes can provide thedeveloper valuable insights into topics such as:
- Whether or not control algorithms are functioning properly
- Whether or not memory locations are being corrupted inadvertently
- Whether or not pointers are behaving as expected
- Locating sections of code that require optimization
- Locating specific lines of code that are causing memory corruption
- Determining whether or not interrupts are firing as expected
- The internal behavior of real-time operating systems and other middleware
Theproblems listed above are frequently the source of “million dollarbugs” so named because of the expense in time, money, and potential lossof company reputation associated with these bugs. The use of KA and SWVcan greatly mitigate these problems and avoid the necessity ofpurchasing additional tools.
Mark Moran is currently USGeneral Manager of Atollic Inc., a wholly owned subsidiary of AtollicAB. Mark began working with embedded systems in the late 1980s whileimplementing quality control and inspection equipment for the food andpharmaceutical industries. Since then, Mark has worked in a number ofcapacities within the embedded systems industry in addition toprogramming, as an FAE, technical support engineer, writer, andtechnical sales representative.