Although embedded software developers would like to think that they spend the bulk of their time designing software and coding it, that is not the case. Almost everyone actually spends more time getting the code to function correctly – a process that we call “debugging”. Different people have different ideas about what debugging actually entails and, with the advent of multicore designs, new challenges are presented. This article reviews debug approaches and technology and considers what is likely in the future.
What do you do?
If you met someone in a bar, say, and you got talking, they are likely to ask what you do. You will probably reply that you are an embedded software developer. (Actually, in my experience, this can be something of a conversation killer. You might be better saying you are an airline pilot or a brain surgeon.) If they carry on talking with you, they might ask you what that job entails and you would probably answer with something about designing software and writing C/C++ code. Of course, that is not really the case. Most embedded software developers spend the bulk of their time debugging.
This does beg the question: what does debugging actually mean? The answer is less simple than you might expect.
Debugging by hand
Many years ago, when computers were big and expensive, there were no debugging tools that we would recognize today. Frankly, programmers made fewer errors. One reason for this was that time on the computer was precious, so the programmer would write the code down on paper first, thinking it through very thoroughly. They would very carefully proof read to avoid syntax errors, as just one slip could waste a computer run. Then they would think through the operation of the code, dry running it on paper to check the logic. Eventually, this carefully scrutinized code would be submitted to the computer. After the run, the results would be studied and, if an error was detected, the code would be modified to include extra printf() (or equivalent) lines to show how values were changing. By using conditional compilation, “debug” and “production” versions of code could be compiled.
So, why not take this approach when writing embedded software today? The first part – writing the code on paper and thinking a lot – might be a very good practice. However, the ready access to computers that cost almost nothing has moved us away from such an approach. The practice of adding printf() calls to log values is still amazingly common, despite it being quite problematic for embedded applications. There are five key issues:
- Most implementations of printf() are quite large. The function needs to address a very wide range of formatting situations and that takes a lot of code. This does not matter in a desktop software context, but memory is rarely in abundance in an embedded system.
- There is the question of where the output of printf() actually goes. Many embedded systems have nothing that looks like a “console” and directing output back to a host computer can be challenging.
- In a multi-threading context – i.e. when using an RTOS – reentrancy is a concern. If two threads try to use printf() at the same time, confusion can result.
- Every time you want to change the variables you are looking at, a complete rebuild and download of the code is required.
- Lastly, of course, formatting and sending the output (somewhere) takes time. If it is a real time application, this overhead can introduce problems.
A real debugger
So, for most systems, using printf() as a debug tool is not viable and the use of a “real” debugger makesmore sense. Debuggers are available from a wide variety of suppliers ofembedded software development tools and can take a number of differentforms, largely characterized by how/where the code gets to execute:
- Simulation – It can be useful to start debugging before target hardware is available. One approach to achieving that is to use some form of simulation. This term covers a broad spectrum of technologies, which may include native execution of code on the host computer and instruction set simulation.
- Virtual prototype – A step on from simulation is the use of a virtual prototype. A number of vendors offer tools whereby code may be executed and interact with a detailed simulation of the real target hardware at a number of possible levels of abstraction.
- Network connection – If a target system is networked, this may be a good way to interface a debugger. However, it does demand fully working target hardware, functional network driver software and some kind of target “debug agent” software.
- JTAG – Many embedded CPUs support the provision of a JTAG port and its inclusion has a minimal impact on system design. Debuggers that support a JTAG connection are widely available.
For many developers, thesimplest solution is the last one listed – attach a debugger to a targetsystem or an evaluation board using JTAG. This can give ready access toall the target data and addresses the five problems with printf() debugging thus:
- There is no extra code on the target.
- Data is displayed on the host computer without effort.
- The debugger needs to be task-aware, but that is well understood.
- Code only needs to be rebuilt to fix bugs.
- A JTAG connection is slightly intrusive in the time domain, but the impact is typically quite small.
Debugging in the future – multicore and beyond
Debugging hasalways been challenging, as the manifestation of bugs can be verysubtle. This became harder with the widespread use of multi-threadedcode, but debugger suppliers responded well to the needs of developers.Many tools allow task-specific breakpoints and feature the option tostop the entire system or just the task of interest. However, this “stopand stare” approach becomes almost non-viable with a multicore system,which may also feature multiple operating systems and CPU architectures.
Going forward, it is likely that sophisticated tracing and analysistools (such as Sourcery Analyzer from Mentor Graphics) will address theneeds of developers of multicore systems, particularly with AMP(Asymmetric Multi-Processing) architectures. Although a tiny amount ofcode instrumentation is required, this has minimal impact. The largevolumes of logged data can be analyzed to yield a clear picture of asystem’s state and behavior in the lead up to an observed bug.
A final thought
I am left wondering whether, if developerswere rationed in their computer access and, hence, required to write outall code by hand and do some thinking, would better code result anddebug time be reduced?
Colin Wallshas over thirty years experience in the electronics industry, largelydedicated to embedded software. A frequent presenter at conferences andseminars and author of numerous technical articles and two books onembedded software, Colin is an embedded software technologist withMentor Embedded (the Mentor Graphics Embedded Software Division), and isbased in the UK. His regular blog is located at: http://blogs.mentor.com/colinwalls. He may be reached by email at email@example.com
Join over 2,000 technical professionals and embedded systems hardware, software, and firmware developers at ESC Boston May 6-7, 2015, and learn about the latest techniques and tips for reducing time, cost, and complexity in the development process.
Passes for the ESC Boston 2015 Technical Conference are available at the conference's official site, with discounted advance pricing until May 1, 2015. Make sure to follow updates about ESC Boston's other talks, programs, and announcements via the Destination ESC blog on Embedded.com and social media accounts Twitter, Facebook, LinkedIn, and Google+.
The Embedded Systems Conference, EE Times, and Embedded.com areowned by UBM Canon.