To read original PDF of the print article, click here.
Strategies for Debugging Embedded Systems
The best time to detect bugs is early in the development process. If you instrument your UML, you can even find them during analysis and design.
Integration and testing of software is difficult, and embedded systems provide the additional challenges of limited manipulation and visibility of the system through a small number of inputs and outputs. Abnormal system states, in particular, are difficult to test, because the system must be driven into the state before its behavior in that state can be determined.
This article introduces the idea of instrumentation code injected into the implementation of UML models for the purposes of increasing the controllability, observability, and testability of the system. The instrumentation is used in both the development and the target environments, and allows interactive system debugging at the model level. In batch mode, the instrumentation serves as the basis for data collection, initialization, and test automation. My goals are to:
A software fault is an erroneous instruction or computation within a program. Execution of that fault results in an error in the state of the software. When the error is propagated to the output, and becomes visible outside the system as an unexpected result, a failure has occurred. Controllability of a program is the ability of a suite of test cases to force the program under test to follow a particular execution path, possibly executing faults along the way. Observability of a program is the ability of the test suite to detect an error state, and thereby illuminate the existence of a fault.
The internal state of the system is important in determining the correctness of tests. The output of a system is dependent upon both the initial state of the system and the inputs applied to it. The same set of inputs applied to a different initial state will result in different outputs. The final state of the system must also be examined as part of evaluating the correctness of the test, as an incorrect internal state may evntually propagate to the system output, causing a failure. System complexity may also make it difficult to predict the correct outputs of the system.
Initial State + Inputs ---> Final State + Outputs
Using black box test techniques, only the external inputs and outputs of the system are available. A distinguishing sequence of test stimuli is required to propagate an error to the output so as to distinguish a faulty program from a correct one. The longer the required distinguishing sequence, the less testable the program.
Embedded systems are similar to black boxes in that controllability and observability are usually limited. Evaluating the final internal state of the system results reduces the distinguishing sequence of inputs required to detect an error, resulting in smaller, more manageable test cases. Instrumentation seeks to increase both controllability and observability in a software program to result in a more testable program.
The technique of using test support instrumentation in application code is a glass box approach to testing. In developing the UML models of the system, developers express an understanding of what the system is supposed to do. Instrumentation-based fault isolation strategies help leverage the knowledge in the UML models into integration testing. The operation and state of the system are more visible at the analysis level than at the code level, where it is obscured by implementation details.
Setting the initial system state for a test only from the external inputs requires some specific sequence of external stimulus. System operation under abnormal conditions is critical to verify in many embedded applications, but creating these initial conditions may not be simple. The techniques described here enable the creation of a test harness to greatly improve controllability and observability.
Phases of integration testing Integration testing is broken into two main phases, dynamic verification and target integration. Dynamic verification is the execution of UML models in the development environment. It focuses on determining the correctness of the models. Target integration involves software and hardware integration in the target environment. Both dynamic verification and target integration are done at the analysis level, with the same tools, using the test support instrumentation.
There are many reasons to do as much dynamic verification testing as possible: hardware availability, hardware/software isolation, shorter debugging cycle times, and access to tools. If you have high confidence in your models after running tests in dynamic verification, debugging in target integration can focus more heavily on the interfaces between system components and on platform-specific issues.
Modeling embedded systems with UML
To achieve these goals, model-based software engineering employs a translational approach, defined below. This article focuses on adding test support into code using a translational approach, but the techniques can also be applied to manually implemented UML models. Specific aspects of this type of the translational process are introduced in the following section.
DesignDesign is the creation of a strategy and mechanisms supporting the mapping of analysis constructs to a run-time environment. Design is conducted in a different concept space from analysis, and much of the preliminary design work can be completed independent of the analysis activities.
TranslationTranslation is the process in which the UML models for each analyzed domain are mapped to implementation through design strategies. Design is conducted at two levels:
Following the language and model compiler analogy further, a model compiler can add instrumentation into the generated code, just as a language compiler adds a symbol table and debug information into the executable. Instrumentation from both compilers allows the resulting application to be tested and debugged by the developer at the same level of abstraction as it was developed. Only in very rare cases would a high-level language developer want to look at the assembly or machine code when debugging an application. Similarly, for UML models, the developer will want to debug at the higher level of abstraction of the models, rather than with the implementation code.The translational approach uses information from the UML models to create code to support testing in addition to the application code. The instrumentation does not add any additional functionality to the software, other than enhanced testability. The test instrumentation is only available for test support and cannot be used during the normal operation of the software.
Because the instrumentation injected during model translation is based only on the UML model execution semantics, it provides a generic test harness that can be applied to any application. The instrumentation can be compiled out, or the code regenerated without the instrumentation, similar to the way debugging information is handled by the compiler.
For manually implemented systems, the level of instrumentation required is dependent on the complexity of the application, the test approach, the target environment, available memory, the support of other tools, and the time available. Thus, trade-offs must be made in order to deliver a quality product on time. In the UML models, one must identify important data values, attributes, inputs, and control points. For each of these items, add the appropriate instrumentation access. The remainder of this article will describe adding instrumentation using the translational approach, but the same principles can be applied to manual implementation.
Instrumented application architecture
The instrumentation agent acts as the interface between the application and the DVUI. The communication mechanism between the agent and DVUI can be any protocol-TCP/IP, RS-232, and so on. The agent supports information marshaling and communication via generated instrumentation code. It interfaces with the instrumentation to set and retrieve instance data and to support DVUI notification of break and trace points, and interfaces with the event processing to provide execution control, system stimulus, break points, and stepping.
Data instrumentation code is injected into the application during translation to monitor and update object instance population, attribute values, event queue population, event data item values, and service parameters.
Keeping a list of instances in the system is fairly straightforward. Each class must include some structure to hold the instances of that class, commonly some form of a linked list. Within the constructor, the new instance is added to the list, and in the destructor, it is removed.
To view the state of an instance, the instance can support ASCII serialization, similar to the Java toString() method. The toString() method would put the value of each of the instance's attributes into the string, as well as the instances with which it has associations. The string can then be transported from the application agent to the DVUI for display. To set the data of an instance, a fromString() method must be supported that can unpack the data and make the proper conversions and attribute assignments.
The state of an instance's state machine is also very interesting during debugging, because it represents an aspect of the control flow of the system. It is special because it affects the response of an instance to an event. Therefore, it's useful to separate the current state of the state machine from the rest of the class attributes.
The DVUI should be able to send to the agent the conditions on which to break. The break allows the DVUI to interactively browse the state of the system at that point in execution and adjust instances and attributes, if necessary. Trace points can also be set up by the DVUI. At a trace point, a message describing the triggering incident is passed to the DVUI and stored in a log file. Trace points are a great way to follow system execution at a high level. They can even be post-processed to generate a sequence diagram describing the scenario executed, or they can be used in regression testing.
Break and trace point controls can be implemented using the publish-subscribe pattern. The agent supports registration of each type of application execution incident. The DVUI can then subscribe to incidents, such as the creation of a particular instance or initiation of a state machine transition. Instrumentation within the application code notifies the agent when an incident occurs, and the agent then notifies the subscriber with the appropriate action, break, or trace.
One implementation of a system modeled with communicating state machines includes an event queue. Events sent by objects and from outside the domain are placed on the queue. An event loop executes continuously, pulling the next event off the queue and passing it to its destination instance, which executes the specified action. With this implementation, the event loop serves as a central place to monitor system execution.
Within the event loop, an instrumentation interface is added that watches the events going by and looks up the next state to be executed by the receiving instance. These conditions can be compared against the set of break and trace conditions set by the DVUI.
The instrumentation in the event loop could also support a single event step mode. In this mode, the DVUI would command the application to execute one action, but stop before the next.
If an analysis-level action language is used to describe the UML model actions and services, you can expand the idea of event stepping to stepping of action language by instrumenting the code in a similar way. Every line of action language would be preceded by an instrumentation instruction, keeping track of the line number and local variable values. Of course, this would be tedious and error prone if done manually, but is easily accomplished by translation.
A soft reset capability would allow the system to be cleared, reinitialized with another initial state, and run with another set of test cases.
One of the more difficult problems in embedded systems tends to be reproducibility of failures due to timing or sequencing of events. By controlling the sequencing of event transmission through the event queue, orderings of actions can be tested and reproduced. Of course, the interface to the event queue would need to allow the reordering of events, in addition to viewing the events.
Another side benefit of instrumentation is the ability to capture true target stimuli and replay it in the dynamic verification environment. If some part of the instrumentation were left in the finished product, it could record a subset of the system state and inputs, much like the “black box” on airplanes.
Figure 1: Emulation of domain's target environment
Figure 1 shows a test driver, either the DVUI or other program connected to the instrumentation agent, emulating the domain's target environment. The driver initializes the class instances that are part of the test. The test driver applies the test stimuli and captures the response. The test driver also emulates the responses of other domains by trapping the service calls and substituting the return values. The calls are a form of output from the point of view of the domain under test, and the responses provided by the test driver provide more input to the domain under test. The test driver uses the test instrumentation in the domain under test to trap and substitute messages.
Figure 2: Multi-domain testing
Single domain to system testing
As confidence is gained in lower level domains, the test support instrumentation can be disabled on those domains to reduce the number of checkpoints and increase throughput. If a problem is found, the test support can be re-enabled to collect more data on the specific test case.
Developers can write test cases, execute, and debug and the level of the models. Ideally, instrumentation should provide a full debug environment at the level of analysis. Full interactive manipulation of instances, events, and data should be available, as well as ways to actively stimulate the system, rather than just watch it. The instrumentation provides full access to the system under test at the level of UML modeling, allowing a glass box approach to integration testing with greater observability, controllability, and testability for embedded systems. Increased observability enabled by instrumentation also allows detection of internal error states without having to propagate the error to the output. This results in asier test case development and shorter debugging time, as the error is isolated much closer to the fault.
Once the capability of the full interactive analysis debugger exists, there is an even greater potential for improved quality and productivity by adding a batch capability to it. This batch capability would allow automated regression testing, automated data collection of failure test cases, or application of randomized test inputs and event sequences to obtain more analysis coverage.
Gregory Eakman is a principle consultant with Pathfinder Solutions and is a PhD candidate at Boston University in the area of automated testing of model-based software. He has successfully applied model-based software engineering to a varity of applications. Contact him at .
2. Pressman, Roger S. Software Engineering-A Practitioner's Approach, 3rd Edition. New York: McGraw-Hill, 1992.