System level software centric power debugging using virtual prototypes - Embedded.com

System level software centric power debugging using virtual prototypes


Battery life has become theAchilles heel for the success of mobile software platforms, such as Android.The mileage users can get from their battery is severely impacted by software.Software implicitly and explicitly defines the power states of major systemcomponents when the phone is used and even when it is not used.

Just the LCD can consume as much power as the CPU in amobile phone. A software power inefficiency or malfunction can quickly cause adrop of 5x in terms of standby time. Evidence for this is available in the userforums of all the major phone vendors where people have updated their phonewith new firmware or applications.

Although power efficiency is a major quality and sign-offcriteria for hardware platforms, it is still a poor cousin in the qualitychecklist for software. There are multiple reasons for this.

The software community still considers power as somethingthat is taken care of by either hardware or the software layer they arebuilding the functions on. Application developers trust that the applicationframework will make sure that while using services the underlying resources areused in a battery-friendly manner.

But the reality is that a simple RSS feed application thatwakes up the phone every 10 minutes for just eight seconds to do some updatesvia the internet can cut the standby time of a phone in half. Softwareplatforms for mobile applications must be engineered in such a way that smartheuristics make sure only those services required in the active usage scenario areprovided. Components must be turned off when not in use.

When porting such platforms, the power control capabilitiesof the OS and their drivers must be made available to the applicationframework. During OS development and porting, software engineers have to makesure that all power saving features of the underlying hardware are leveragedand supported by drivers.

With dynamic frequency and voltage scaling of the CPU thiscan reach deep into the internals of the OS kernel, since it is impactingscheduling through work-load prediction.

Finally, in order to guarantee an efficient use of thebattery, all highly distributed software entities for power saving andmanagement have to be vertically integrated and need to cooperate.

Obviously, a minor malfunction or inefficiency in just onesoftware module can trigger a cascade that severely reduces battery mileage. Itis also obvious that debugging those cause-effect chains can become anightmare.

Debugging is extremely problematic since the cause-effect searchspace is not limited to one software layer. While software looks fine from afunctionality and performance standpoint, it could well be that the batterydrains four times faster.

Imagine a scenario where the phone is locked and it islocated in the pocket of a user who is hiking. After the hike the battery isempty because the accelerometer was not shutdown and constantly woke the phoneup when it would have been idle otherwise.

This example shows another challenge for power profiling anddebugging. In contrast to software for a desktop PC, the usage scenario playsan important role, as it defines how the phone interacts with the environment.How can you debug your phone however, while it is locked in your pocket? Howcan you make sure that scenarios are deterministic to compare differentimplementation options?

Debugging power defects moreover comes with another majorissue. In a low power mode, such as suspend, the embedded debug service is likelyto be suspended as well. In addition to that, any debug interaction with thedevice is intrusive and severely tampers the power figures in itself.

A debug daemon would not let the phone go to suspend,require USB/UART/Ethernet drivers, fire additional interrupts, increase the CPUload, change workload prediction and the frequency and voltage control. Furthermore,expensive lab equipment is required to perform a sufficiently fine granularpower profiling to determine which of the components is the most critical.

On top of all the technical challenges, software centricpower profiling and debugging is an educational challenge. Software engineersmay not be able to interpret what it means if 250mW are saved for a specificcomponent configuration. Here, it becomes much more important to providequalitative results, such as the stand-by time has dropped by 50 percentbecause the component is used 90 percent of the time in the profiling window.

Virtual Prototype based software centric power analysis and debug

Virtual Prototype (VP) basedsolutions enable software centric power analysis and debug to address thepreviously detailed challenges of:

§ Deterministic repetition of real-world scenarioson the developer’s desktop

§ Non-intrusive debugging during suspend and low-powermodes

§ Understanding system level hardware and softwarebehavior and interaction

§ Power measurement, debugging and root cause analysis

A VP is a software emulationof an entire system, such as a Smartphone ash shown in Figure 1 below . VPs can be made available early in the developmentstage of an electronic system, even before hardware is available. A VP is ableto run the exact same binary software that the hardware system would execute.

However, execution isperformed on simulation models of all the system components, including theapplication processor, the memories, DMAs, timers interrupt controllers etc. Those models can be instrumented with information that characterizepower.

The execution is controlledvia scenario scripts that drive the I/O of the VP, and include generatinguser-input via the touchscreen controller, setting GPS coordinates through theUART or initiating a phone call. Software optimization is made possible bycollecting power analysis data along with running other hardware and softwaretraces during simulation, in order to enable root-cause analysis and debugging.

Figure 1 -Virtual Prototype Framework

Software debugging

VPs easily integrate intoexisting software development flows as state of the art software debuggingtools and IDEs—GNU gdb, ARM RealviewTM, and Lauterbach TRACE32TM for examplecan be connected to the VPs, just like with real hardware. VPs however, do notrequire an embedded software service executing on the prototype to enable thelink to a software debugger. As highlighted earlier, this is a fundamentalrequirement to debug low power modes, like suspend or idle. Here, the VPframework itself provides debug interfaces for software debug tools.

Using the interface, thedebugger is able to read and modify the system state as well as control itsoperation. In this case debugging is non-intrusive, since the whole systemsimulation is suspended during debug operations and not only the CPU is underdebug.

Because each debug operationhappens in zero-time from the software’s point of view, the software is unableto recognize inspection or control by a debugger. When debugging, the user canbe sure that he is not accidentally impacting the power profile of the system. Thisdebugging mode is also referred to as stop mode debugging.

Of course VPs also supportrun-mode debugging, where only a process under debug is stopped and linked to asoftware debugger and a debug daemon is established via virtualized IO.

Debug and analysis scenarios

Virtual I/O is also essential for the creation of real-worldusage scenarios, such as navigating with a “maps” application when walking andusing a 3G internet connection. Virtual I/O refers to the ability of a VPinterface peripheral, such as USB, UART, Touchscreen IF, to connect either tothe real world (e.g. USB), or further (functional) simulation models of otherparts of the system (e.g. modem, GPS, accelerometer).

Using VPs it is possible to record and replay thosescenarios in a deterministic fashion for debugging, optimization and regressiontesting. Scenario recording is done via the ability to watch each and everyaspect of the system, including even signals and hardware registers next tomemory.

Breakpoints and watchpoints can trigger actions (i.e. callback)in a VP, such as reading the interface signal value and writing those, togetherwith a timestamp, into a file on the host for replay. Those actions areexpressed using a scripting language, like TCL (Tool Command Language).

The scripting interface provides the ability to peek andpoke data from/to a VP, as well as control the execution. It is important tounderstand that the script is not part of the embedded software.

The script is running on the host in the VP tool framework. Thus,in the replay script, data is pushed back into the VP using simulation timetriggered actions. The scenario replay allows the software developer to easilyreproduce complex defects during debugging and then compare analysis data foroptimization.

Power characterization& analysis

Power analysis data is recordedduring simulation by observing the power states of the system components. Thisobservation is again performed using watchpoints triggering actions as shown inFigure 2 below .

Each action re-computes the powerlevel for a component based on querying information, such as internal stateregisters or signal/pin values. On top of the basic scripting commands an easy-to-usepower framework is provided that allows the user to characterize power for componentswithout being familiar with all the details of the scripting command set.

The power data is used to computeenergy along with the time that the system is running. The software developeris then able to determine energy for a time interval and component underinvestigation. This allows finding abnormalities, such as the LCD being turnedto low instead of off during suspend or the CPU being loaded more than 70percent when the phone is idle.

Of course in order to identify thoseabnormalities the characterization must be accurate. Analysis is conducted on atrend basis and in a qualitative fashion. For this purpose it is sufficient tohave ballpark numbers provided by data sheets. A critical piece in the powerdebug flow however, is the ability to perform a root cause analysis.

Figure 2 – Stimuli,characterization and analysis

Power debugging

Root cause analysis requiresmore than merely hooking up a software debugger and investigating the executionof software along with the power levels. First, it is a challenge to link thesoftware execution with the power profile.

Power management is a highlydistributed function that is performed in many different processes in alllayers of an embedded software stack, such as Android. Thus, the tracing of softwareexecution and power is important in understanding the history of the HW/SW system.

This tracing must beperformed at a level that is meaningful to the software developer. Aninstruction level trace is far too detailed, but even tracing on the functionlevel is meaningless in an OS where processes switches happen thousands of timesper second (e.g. 1Khz timer interrupt).

A VP helps to abstract thetracing to a level where it becomes useful for the developer. Using anOS-awareness plug in, the VP can show OS processes over time along with thepower analysis data as shown in Figure 3below .

Figure 3 – Power analysis

At the process level thedeveloper is able to understand the relation between software entities in thestack. As shown in Figure 4 below ,he or she can determine for example that the battery status animation updateevery 500msec is triggering a cascade of actions, such as triggering interruptsin the kernel and event queues in the user space. It is important to determine forinstance why the Linux “tickles-kernel” power saving feature is not performingwell with Android in idle mode.

For a process that has beenidentified for a power level cause, the user can perform a top down debuganalysis looking at functions and even instructions in each process. The developeris able to pinpoint to the exact location in the application or the driverwhere the state change happens. The OS-awareness of a VP is again realizedusing breakpoints in the OS that trigger actions in the scripting framework topeek the process ID/name via the virtual memory read interfaces. Thus,OS-awareness in a VP does not require any instrumentation or change of the OSand is solely dependent on function symbol information.


Figure 4 – Tickless Kernel Process Trace & SoftwareLogging

 

Software logging and printf

Software logs, as well asuser defined debug messages, using the well known printf method can beextremely helpful to find the defect cause for a fault symptom. Those messages are typically at a level of abstraction where they nicelydescribe the function of the software at a detail level targeted for the user/developer.

Furthermore, Androidprovides logging mechanism for the application layer, the applicationframework, the run-time environment and the hardware abstraction layer. Unfortunately, the log generated via the Android logger is separatedfrom the logging performed by the Linux kernel, or the logging performed by theuser through ad-hoc printfs.

This becomes especiallyproblematic when log messages from the Android’s extended Linux powermanagement (i.e. wake-locks) need to be correlated with the kernel logs orother traces, like the process trace or power trace. VPs allow the unifying ofall traces without any change in the embedded software.

Thus, all trace informationcan be accessed in one environment that is correlated via time stamps. Themechanism behind it is once again based on the VP scripting capabilities. Asshown in Figure 5 below , the VP issetting breakpoints at each log function and triggering an action that isreading out the log string.

Figure 5 – Catching Linux Kernel Log Messages

The VP then examines theparameter of the log function, such as the virtual address of the string inmemory. The parameter can be inspected for example by reading the CPU registervalues that are typically used for parameter passing.


This way, arbitrarycomplex data structures can be de-referenced (e.g. Java object names, etc.) asshown in Figure 4. This is especially useful for software stacks that run applicationsin Java on an interpreter, since the link between the Java world and the nativeworld can be established to ease debugging.

Summary

In this article we have outlined how Virtual Prototypes (VPs)provide all the necessary elements for a debug solution that can spot and removepower related defects from software. Creating and executing deterministicscenarios for debug and regression has been described. It has also been shownhow debugging can be performed in a non-intrusive fashion with global visibility,even during low-power modes, such as idle and suspend.

We have reviewed how system power characterization can beperformed and how power can be analyzed over time and component. We have also shownhow debugging can be performed at the right level of abstraction using OS-awarenessunifying software logging with system traces.

Power debugging is just one formidable application for VPs.A key concept to understand is that debugging capabilities have been added atthe VP user level, purely based on scripting capabilities.

VP scripting allows for much more, including user defineddebug tools (e.g. memory corruption checker) or system level hardware/softwareassertions. Such solutions are being actively discussed in the context of theopen Virtual Platform User’s group (http://groups.google.com/group/virtual-platform-users).

(Achim Nohl is aSolution Specialist at Synopsys, responsible for Processor Designer and VirtualPlatforms. He drives the adoption of Virtual Prototypes for SoftwareDevelopment and the design shift from hardware to software. Achim holds a MSEEin Electrical Engineering from the Institute for Integrated Signal ProcessingSystems in Aachen .He can be reached at achim@synopsys.com .)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.