Engineers love to solve problems. It’s what we do. Unfortunately, one of the biggest problems with embedded software engineers is that we create a lot of our problems and then make ourselves the heroes by spending ungodly amounts of time fixing them (Debugging!). It’s pretty typical to find companies where embedded software engineers spend 20 – 40% of their time debugging! Thankfully, there are a lot of potential changes teams can make to decrease the time they spend debugging and get it into single-digit percentages. In this article, we will examine several tips for reducing debugging time.
Tip #1 – Embrace Test-Driven Development (TDD)
Test-Driven Development is a technique that allows developers to incrementally build their production software where they rely on tests to dictate the code they write. For example, TDD has developers first write a test case, make it fail, and then write only the code that allows that test case to pass. The process is then repeated.
Traditionally, embedded software developers will write entire code modules before testing it. It’s possible to write thousands of lines of code over several weeks. Then, when it comes time to test it, where is the problem if it doesn’t work? Only God knows! The developer has to painstakingly go back through the code and discover where the problem is and fix it. The amount of time required to do this can be considerable.
On the other hand, for developers using TDD, if a mistake is made and a bug is injected into the code, the test case will immediately tell the developer! Since they are incrementally writing their code, they are more likely to know exactly what they changed and can fix the problem immediately. TDD might seem like it takes more time to practice, but it creates a collection of test cases that can be run in regression testing to ensure that everything works as expected. TDD kills two birds with one stone: decreased time debugging and automated tests.
Tip #2 – Develop off-target as much as possible
When a project starts, one of the first instincts of nearly every embedded software developer is to get a development board and start writing embedded code. Unfortunately, in many cases, the embedded code is not the differentiator in our products; it’s the application code. While a lot of application code eventually needs to interact with hardware, many modules can be developed off-target, i.e., on a host computer.
Developing off-target code provides developers with many opportunities to decrease the time spent on each debug cycle. For example, typically, to write and test code for a target microcontroller, a developer must:
- Cross-compile the code
- Start a debug session
- Program the device over SWD
- Run the code on the target
- Verify that the code works by running it on the target (must have all the low-level code as well).
If the code is developed on the host, the developer must compile it for the host and then use a unit test harness, an emulator, or a custom program to run the code under development. If a problem is found, it’s much faster to fix, recompile, and go again. On an embedded target, just programming the target can add dozens of seconds to each cycle, let alone the temptation to single-step through code.
There are specific errors that off-target dev / debugging can create. However, I write about 75% of my code off-target now and have found that I’m much faster and more efficient. Instead of tracing issues through the embedded target, I can force issues in code quickly, determine the cause, fix it, and move on. Granted, some things will come up on target that won’t appear on the host.
Tip #3 – Master debugging strategies
The least efficient debug method known to humankind is single-stepping through lines of code. Don’t get me wrong, there is a time and place, but it often wastes a lot of time. Unfortunately, embedded software developers default to debugging with breakpoints and single-stepping. To get better at debugging, developers need to master the other debugging strategies available on modern microcontrollers.
At least eight different debug techniques are available to developers today. These techniques include, from the order of simplest to most complex:
Watch / Expressions: Provides the ability for a developer to examine the CPU and peripheral registers. They can often be used to spy on variables, perform calculations, or halt the CPU on a change.
Breakpoints: Provides the ability for a developer to stop CPU execution on a specific line of code. Advanced breakpoints can be used to set conditional statements.
printf: Provides the ability for a developer to print character data to a mapped serial interface. Depending on the implementation, this may or may not affect real-time performance.
Assertions: These are conditional statements used to verify assumptions at a specific point in a program. Failure of an assertion will often halt the CPU and provide the file and line location of the failed assertion.
Statistical Profiling: Periodic sampling of various registers within the application that occur simultaneously to its running. Often does not affect real-time performance. For example, one might want to sample the program counter (PC) to understand what code modules are being executed.
Data Profiling: Periodic sampling of various memory locations that contain variable data. Data profiling can be great when used with a real-time visualizer to monitor the system’s state, variables of interest changes, and so forth.
Task and Data Tracing: Provides developers with the ability to track events within a real-time operating system application. As a result, developers can gain insights into application performance, task latencies, run-times, and much more.
Instruction Tracing: Provides developers the ability to record every instruction executed on the processor. This can be used to understand code coverage during testing, debug compiler issues, and more.
Mastering all these techniques and knowing when to use them can dramatically decrease how much time is spent debugging when a defect does get into the system.
It’s possible to spend a lot of time debugging embedded software. Sometimes, debug time just can’t be avoided; however, developers may spend more time in many cases than they need to. We’ve explored several areas you can investigate further to decrease the time you and your team spend debugging. If you spend more than 20% of your time debugging, take an hour this week to identify what changes you can start to make immediately to get the time you spend debugging under control.
|Jacob Beningo is an embedded software consultant who specializes in real-time, microcontroller-based systems. He actively promotes software best practices through numerous articles, blogs, and webinars on topics from software architecture design, embedded DevOps, and implementation techniques. Jacob has 20 years of experience in the field and holds three degrees including a Masters of Engineering from the University of Michigan.|
- embedded.com: Debugging stories, from printf, just Flash and beyond.
- Debugging poor performing or unreachable code
- Software tracing in field-deployed devices
- Optimizing pre-silicon software development
For more Embedded, subscribe to Embedded’s weekly email newsletter.