One of the challenges in real”time systems, especially in multitaskingOS based implementations, is defect fixing. To resolve the defect onehas to be aware of the program flow during the defect or faultycondition. Normally, this is done by using in-circuit emulators (ICE)along with the break point feature available in the environment of theemulator.
However, the ICE support may not always be available for the systemunder consideration. Considering the case that ICE is available for thesystem under consideration, and the ICE hits a breakpoint, then thedynamics of the system is lost.
All the interactions the system was having with some networks orother systems/modules are stopped at once. Therefore it again may bedifficult to reproduce the defect or anomaly exactly. Though some makesof ICE provides a feature of real-time trace function, but that is alsolimited by the ICE memory.
At times it is required to stop the system in some logical trigger,so that the condition of the system can be analyzed at that point. Thisfeature is also not common in normal debugging systems.
Finally some development systems from WindRiver, tools like WindViewoffer powerful debugging mechanism. This mechanism allows user to trackcode flow and task/interrupt scheduler switching in a GUI basedinterface. However, the development system should have support fordeploying this debugging feature.
The method that we propose in this article overcomes numerous suchproblems with no extra investments. This requires some amount of freememory on board for both code/data and substantial amount of processingcapability.
The function trace re-useablecomponent
The method we have developed can be applied to debug thesoftware/firmware of any type of real-time system and is named as”Function Trace” (FT).
In this method, all the functions in the program are uniquely codedand as the function is executed the unique code is noted. These valuesare stored in a circular array in round-robin fashion. This is calledas function code logging. The array contains the program flowinformation which can be used for debugging purposes.
The basic requirement of employing this method is to have moderatespeed processor with some free memory. Thus 32-bit processors are thebest choice for applying this method of debugging. Depending on theprocessor speed this debugging method may be looked upon as an overhead for low speed / low end systems.
The tracking capability of the functions executed in the systemmakes it best suitable for both OS based multi-thread system and aswell as OS less systems. Function Trace can also be deployed inmultiple locations within a given function to figure out about theprogram flow within a function just as we might use it for program flowwithin a module / thread / task.
Function Trace can also be added on with time stamp provided thesystem considered has a counter or a Real Time Clock on board. Thisenables us to figure out not only the program flow but also the timetaken in various functions / modules within the system.
One of the important feature of this debugging method is that itallows the user to trace only the function(s) which they intended totrace. Therefore this mechanism eliminates tracing functions which arenot required, resulting into optimal usage of memory for storage oftracing data. Therefore this helps the developer to focus on therequired sections of the code.
Function Trace's basic operating principles
The Function Trace feature (Figure 1 below ) creates an executionlog of every function call defined in the application firmware. As anyapplication-defined function is called, an entry for the currenttask/interrupt context (for multi-tasking system) and function calledis generated in the log with the help of a unique 32-bit identifiercode.
|Figure1: The Concept of Function Trace|
The identifier can be generated by concatenating a function code andfile code, where the function is defined.
To control the logging of function trace better, the functions canbe divided into various levels thus allowing the control of executionflow log. The entire application might be, for example, divided intonumerous small applications. The smaller applications might be dividedinto modules and further into units.
Thus one might think of tracing at the module level functions, somemight think of module level or unit level functions for zooming intothe problem area. To implement this control, a function level flag isused. All the functions are categorized into some levels. Based on theflag (set/ reset) the function will be logged to the trace for a givenlevel.
The logged data shall be stored in RAM within the embedded system,therefore the user of this method has to plan a way to retrieve thefunction trace data through communication ports or ICE connectivity orany other method suitable for the system under debug. After the data isextracted, further analysis can be done upon the sequence of executionof the code during the faulty condition.
The architecture of function tracing
The basics of function trace architecture are divided into thefollowing: encoding of functions, task/interrupt encoding, tracelogging and log management.
Encoding of functions. The function encoding is a veryimportant part in this method. All the functions are to be identifiedby a code which should be unique. This encoding can be done in variousways. An example is discussed below:-
A 32-bit value is used for encoding all the functions. All the files/ functions / parts within a function within the entire program aregiven unique code. The first 16 bits are used to encode a file, andthen all the functions in the particular file are segregated by oneunique 8 bit no.
Combination of file identifier and function identifier will make thefunction code unique. The reserved section might be used to perhapsencode positions within the function or any other information that isuseful for debugging. The division of 16bit for file name and 8 bitsfor function are just examples. One might use any other combination tooas shown in Figure 2 below .
|Figure2: The Function Code|
For example, as shown in Table 1, below , let's suppose anembedded program contains 3 files File1, File2 and File3. File1contains Func_a and Func_b, File2 contains Func_c, File3 containsFunc_d, Func_e and Func_f.
|Table1: Coding of functions|
Task/interrupt encoding for rtos based systems. One might alsoencode the context of execution (Interrupt/Task) within the entries inthe Function Trace. Tasks or threads in RTOS based system areidentified with some identifier. This value (normally 32 bit) can beused to identify the task. Depending upon the RTOS, the task identifieris located in the Task Control Blocks. This value is used here toidentify the tasks.
Similarly interrupts may also be identified with their levels ornumbers. Thus the task identifiers and the interrupt numbers can beused here to segregate the context switching.
This entry can be limited to interrupts and normal context in theNon-RTOS systems since there is only one main loop in such systems.
This technique also helps in identifying and debugging functionswhich are being used in both the contexts like interrupt and main loop,tasks etc.
Logging of trace. Whenever a function is called, thefunction code will be stored in a circular array. The size of the arraywill depend on the resource availability and depth required for defectanalysis. A large array will increase the time depth of the loggedrecord.
Normally in non OS based system the function code will be logged. InOS based system, along with the function code the task-id or theinterrupt level from which the function is being called can alsologged.
The structure of the function trace variable will be as follows:
For example, a logging function (logFunctionCode) can be called at theentry point of all the functions to store the record, or a loggingfunction might be called at a function exit or even at a critical phaseor even at an exit. Therefore depends upon where the debug is required.<>The pseudo code for the function definition will be as follows:
Note that the entire logging function is protected from preemptionor interruption. This is noted as the “critical section” in the pseudocode.
Log Management Operations and Features
Log management involves xx critical operations and capabilities,indluding the indexing of the function trace, function levels, andtrace start/stop.
Index of function trace. The function trace is loggedin a circular array with a single index. At times, it may becomedifficult to identify the last logged entry of the array.
Therefore the index of the function trace array will have to besaved as a global entry. The index that we are discussing about has tobe protected from being corrupted in case the function trace loggingfunction is called by concurrent functions / tasks / interrupts.Therefore it is mandatory to protect the logging function usingcritical section. The logging function has to be compact and small.
Function levels. The time depth of the function tracedepends on the size of the trace log array. However, system may notrequire logging all the functions such as low level driver routines. Toimplement this selective logging, all the functions in the program canbe categorized into some levels as shown in Table 2 below . Thecall of the logging routine can be enclosed into the level macro.
|Table2: Coding of Functions with Levels|
As shown in Table 2 above all the functions in the programare associated with some level. Thus the function trace log call willbe as follows:
By enabling the macro (Eg: LEVEL1 or LEVEL2) globally, the loggingcapability of the functions can be modified as per the requirement.Therefore, one should ideally create functions along with the functiontrace call along with appropriate logging level macro. Compiling thecode as per the set macros would result in code generation as perlogging requirements.
Start/stop of function trace. While trying to fix ananomaly, one may require starting and stopping the function trace tofocus onto the defect / anomaly at hand.
Normally at the start of the program the function trace logging willbe started. The logging can be stopped in occurrence of an event in theoperation. This can be done by evaluating a Boolean variable at thestart of logFunctionCode() function asshown below :
By controlling the Boolean variable “stopFunctionTrace”, the loggingcan therefore be controlled.
The function trace can be stopped in a number of differentscenarios, including:
* In the predefined error state, which is common in most of theembedded firmware
* On occurrence of software or hardware exception
* Also in some relevant logical event, a trigger can be designed toforcefully stop the logging
The mechanism is however to be decided by the developers withrespect to an embedded system under consideration and the hardwareunder consideration.
Function trace enhancements
Other than the features discussed above, the functionality of thiscomponent also can be enhanced in numerous ways, however notexhaustive, including time stamping, trace positioning, and functiontrace analysis.
Time stamp functionality. In real-time systems dataanalysis of the timing of execution is equally important as the flow offunctionality. The time stamp of the function log may also be takenalong with the function code provided a real time clock is available inthe system.
A relative time-stamp can also be logged using a free runningcounter if available in the hardware. In this case the function tracearray will contain one more entry for the time stamp field. Whilelogging the function code the free running counter value would also bestored. The relative values between the logs will indicate the timerequired in execution of the functions traced.
Position trace. At times it is required to trace thepart of the function executed especially for OS based system or in caseof some functions where the entire function is divided into some cases,the actual case of execution is the point of interest. The positiontrace can trace different parts of the same function.
The last 8 bit of the function code (See Figure 2 earlier) was leftreserved. This part can be coded to identify different lines of thesame function. Therefore allowing the user to understand how thatparticular function is executed during the defect / anomaly condition.
Function trace data analysis. The data logged byfunction trace is stored in the Function Trace Array which is locatedin the system RAM. One has to design a mechanism by which the datamight be extracted out of the embedded system and be viewed in aformatted form.
The general options that might be considered are as follows:
* Most of the embedded systems generally have a communicationchannel to connect to the external world. Using this communicationchannel, the data can be taken to the external world for analysis.
* Some systems have non-volatile storage device with in the board.The function trace data can be stored into this device and can besuitably taken out at a later point in time after connecting an ICE.
The data once extracted from the system can now be interpreted asper the encoding done in Table 1earlier . The index shall provide the pointer for the last entryof data and therefore the interpretation shall be required to be doneaccordingly. However care should be taken about the endianness of thedata since it depends on the type of the target system. (LittleEndian or a Big Endian machine )
Thus the developer will realize the program flow which would provideleads for resolving the anomalies.
The data available from function trace is a couple of numbers. Someprocedure should be defined to parse the data into a readable form.This would provide easy interpretation of the captured data. One mightdevelop a macro or a lookup table for interpreting the results inMicrosoft Excel.
Microsoft Excel based. Assuming that the data for thefunction trace is available in form of two columns of data, where thefirst column represents the task id/interrupt level and the secondrepresents the function code. Now the parsing can be done using MSExcelas shown in Figure 3 below .
Column A contains the function code and column B contains thecorresponding function name. Similarly, column H contains the Interruptlevel/Task ID and column I contains the corresponding Task/Interruptname.
Now the available data for function code is copied on column E andtask ID/Interrupt level on column C. The table lookup formula is usedto decode the function name, task name and interrupt name. Column D andF are the resultant output.
Also the last index can be pointed out manually in the blue mark tosegregate the start/stop of the data.
|Figure3: Excel spread sheet for interpreting the Function Trace Results|
C or Visual Basic Software Tool. A parser also can bewritten using any programming language like C or Visual Basic. A parsertool will take the input from the function trace and update thefunction name and task/interrupt name through table look up in a datafile. The sample file in Figure 4below shows the parsed output of the function trace.
|Figure4: The Output of Software Tool|
A price to pay – Function trace tradeoffs
Function trace also has some points to be remembered duringimplementation:
* Function trace logging takes some CPU time and might pose as anoverhead to the system. Therefore in highly time-critical applicationsit may affect the timings of the application
* This method might become an overhead for slow processors executinglarge programs
* The log array may take a significant amount of memory. Thus memorycritical systems may face some memory crunch to implement thisdebugging mechanism
* The position of the logging within a function is another criticalpoint to be taken into account. The user may be confused by the datagenerated in some preemption cases
* This method neither locates the actual point of preemption nordoes it have the facility to watch the values of the parameter passedto the function. However this limitation may be minimized using theposition trace feature.
A last word
This method is successfully implemented in some of the live projects inthe PES IndustrialAutomation sub-vertical. It was observed that the turn around forcomplex defects had been reduced significantly. As it is a one timeimplementation, it requires very little maintenance.
It has also helped to measure the time required to execute aparticular function using the time stamp entry and without using aDSO/Logic Analyzer. Last but not the least, the logged output of theFunction Trace helps people understand the code flow, RTOS taskswitching and preemption and therefore improves the understanding ofprogram flow.
Anindya Dutta is a Embedded Software Engineer having over7.5 years of experience in the Industry. Currently he is working for Patni Computer Systems Ltd. Hiscore areas of skill are design and development using C, C++ andVx-Works in the Domain of Industrial Automation and Safety
Tridib Roychowdhury is a Technical Architect with more than11 years of total experience which spans over as a Firmware Developer,in the industry, and a Design Engineer in the engineering industry.Currently he is working for Patni Computer Systems Ltd. His experienceprofile includes firmware development using Assembly, C, C++. Hisindustry experience is in designing various products for the industrialautomation industry.