Using RTOS-aware debugging and Serial Wire Viewer to debug Cortex-M3/M4 designs
When viewed from this perspective, the debugging problem when using an RTOS shifts to a much higher level of abstraction. In traditional debugging, the developer typically wants to know precise low-level details such as the value of a variable, contents of a register, bit pattern in an SFR, how the application behaves while stepping, etc. Although these considerations remain, debugging an RTOS requires information such as:
When halted, what are the task states?
Which task is running, which tasks are ready, which are waiting?
If waiting, what resources are tasks waiting on?
How many synchronization objects such as semaphores and mutexes are being used and what is their status?
What events or messages exist, and what are the statuses of these objects?
Which timers have expired, which are still running?
This is where a demand for further and better visualization is created, and satisfying that demand is particularly critical when debugging and conducting integration testing.
The notion of “kernel aware” debugging, that is, RTOS object rather than code object specific information, took hold around ten years ago. Initially regarded as a gimmick, KA is now widely considered to be indispensable by embedded systems developers. The reasons are twofold. First, answers to the above questions collectively constitute a high level snapshot of the health of the application from a perspective of the RTOS constructs whose correct interaction is essential to the success of the application. Second, the information desired is buried deeply within the data structures used during construction of the tasks, services and control mechanisms.
A developer does not want to have to search out the memory location where, for example, a mutex or a timer timeout flag is stored and display it if this can be avoided. In addition, if using an RTOS in binary-object library form, this is difficult or impossible. Well tested and optimally laid out information found in contemporary integrated development environments is usually easier and faster to read as well as being more informative at a glance.
Figure 3 shows a typical set of kernel aware debugging windows.
Click on image to enlarge.
Figure 3 - Kernel aware debugging windows for the embOS operating system
Uniting kernel aware debugging with serial wire viewer
The principle drawback of kernel aware debugging is the same as traditional JTAG. In order to visualize RTOS specific information such as task states, semaphores, messages, and events, the application must be halted. For certain devices under test such as motor controllers, this is not an optimal strategy. Sometimes it is simply impossible without compromising the hardware. There are two potential solutions to the problems:
First, limited but often crucially important information on RTOS aware objects can be made available dynamically using Serial Wire Viewer. Doing so requires backtracking slightly from the previous question about whether or not the developer wants to inspect RTOS data structures at a low level and dig out essential elements for survey. When sufficient benefit is derived it is worthwhile for the developer to identify important and informative fields in RTOS specific data structures and configure SWV to report these dynamically.Focusing on the first technique of using Serial Wire Viewer, an example of this can be seen in Figure 3. The window in the top right hand corner is the Serial Wire Data Trace Timeline Graph. This is time evolution of a field in a data structure that holds the number of messages in a mailbox. Messages are normally produced by a task in order to provide information to another task. The source task inserts the message into the mailbox, and the destination task typically blocks until the message is available, and then extracts and uses that message. Figure 3 shows a regular pattern of production and consumption of messages. Of course, this only has meaning in the context of the design and implementation of the application. The appearance of such a waveform is no guarantee in and of itself that the application is behaving properly, however.
The second alternative involves purchasing an additional product. Micrium’s uC/Probe is a tool that reports KA debug information and other details about the application in real time. This requires a communications channel such as JTAG probe or a USB, LAN, or UART channel, and is intrusive.
Having said that, suppose that the waveform depicted in the Serial Wire Data Trace Timeline Graph is in fact the desired and correct behavior. If a problem cropped up where there was a disruption in this process, it would be obvious immediately when inspecting the graph. Remember, this data is evolving in real time (and without processor overhead!) so the data can be monitored while the application is running in its native environment, i.e. on real hardware, interacting with stimuli from the outside world. A disruption in the message traffic could then be correlated exactly with other behavior. Further, the axis of this display is labeled in time units, so a software tester would know exactly when from test commencement that the error occurred, which may be another valuable clue in finding the problem.
One of the goals in the design of Atollic TrueSTUDIO to minimize the number of extra tools that developers need to do their jobs, such as traditional Code Review tools, interfaces to version control, and bug tracking database servers, as well as optional modules for code analysis and test automation.
In an application using TCP/IP, the number of packets being transmitted as a function of time is often of interest. A tool like WireShark is commonly used for this purpose. This information can tell the tester whether or not the application is “still talking/listening” or whether or not hiccups are present in the communications flow. As shown in Figure 4 TrueSTUDIO displays this information using the Data Trace Timeline Graph.
Click on image to enlarge.
Figure 4 - TCP/IP packets displayed as a function of time in realtime
At a glance, communications rate can be roughly estimated. This is often an important metric in TCP/IP applications or services. As Figure 4 shows, SWV is often capable of rendering essential information so that the tester does not have to shell out of the environment, consult additional tools like WireShark, and then come back to the environment to continue testing, debugging and data gathering.
The Serial Wire Viewer’s capabilities for exact and accurate monitoring of memory accesses (variable updates) can be combined with real-time operating systems and other types of middleware to provide a uniquely powerful tool for advanced visualization and debugging of middleware-equipped embedded systems. Being able to visualize the time evolution of specific variables and other events as the application executes can provide the developer valuable insights into topics such as:
- Whether or not control algorithms are functioning properly
- Whether or not memory locations are being corrupted inadvertently
- Whether or not pointers are behaving as expected
- Locating sections of code that require optimization
- Locating specific lines of code that are causing memory corruption
- Determining whether or not interrupts are firing as expected
- The internal behavior of real-time operating systems and other middleware
The problems listed above are frequently the source of “million dollar bugs” so named because of the expense in time, money, and potential loss of company reputation associated with these bugs. The use of KA and SWV can greatly mitigate these problems and avoid the necessity of purchasing additional tools.
Mark Moran is currently US General Manager of Atollic Inc., a wholly owned subsidiary of Atollic AB. Mark began working with embedded systems in the late 1980s while implementing quality control and inspection equipment for the food and pharmaceutical industries. Since then, Mark has worked in a number of capacities within the embedded systems industry in addition to programming, as an FAE, technical support engineer, writer, and technical sales representative.
Loading comments... Write a comment