Five more top causes of nasty embedded software bugs

Michael Barr

November 2, 2010

Michael Barr

The risk with priority inversion is that it can prevent the high-priority task in the set from meeting a real-time deadline. The need to meet deadlines often goes hand-in-hand with the choice of a preemptive RTOS. Depending on the end product, this missed deadline outcome might even be deadly for its user!

One of the major challenges with priority inversion is that it's generally not a reproducible problem. First, the three steps need to happen—and in that order. And then the high priority task needs to actually miss a deadline. One or both of these may be rare or hard to reproduce events. Unfortunately, no amount of testing can assure they won't ever happen in the field.⁵

Best practice: The good news is that an easy three-step fix will eliminate all priority inversions from your system.

• Choose an RTOS that includes a priority-inversion work-around in its mutex API. These work-arounds come by various names, such as priority inheritance protocol and priority ceiling emulation. Ask your sales rep for details.
• Only use the mutex API (never the semaphore API, which lacks this work-around) to protect shared resources within real-time software.
• Take the additional execution time cost of the work-around into account when performing the analysis to prove that all deadlines will always be met. Note that the method for doing this varies by the specific work-around.

Note that it's safe to ignore the possibility of priority inversions if you don't have any tasks with consequences for missing deadlines.

Bug 9: Incorrect priority assignment
Get your priorities straight! Or suffer the consequence of missed deadlines. Of course, I'm talking here about the relative priorities of your real-time tasks and interrupt service routines. In my travels around the embedded design community, I've learned that most real-time systems are designed with ad hoc priorities.

Unfortunately, mis-prioritized systems often "appear" to work fine without discernibly missing critical deadlines in testing. The worst-case workload may have never yet happened in the field or there is sufficient CPU to accidentally succeed despite the lack of proper planning. This has lead to a generation of embedded software developers being unaware of the proper technique. There is simply too little feedback from non-reproducible deadline misses in the field to the original design team—unless a death and a lawsuit forces an investigation.

Best practice: There is a science to the process of assigning relative priorities. That science is associated with the "rate monotonic algorithm," which provides a formulaic way to assign task priorities based on facts. It is also associated with the "rate monotonic analysis," which helps you prove that your correctly-prioritized tasks and ISRs will find sufficient available CPU bandwidth between them during extreme busy workloads called "transient overload." It's too bad most engineers don't know how to use these tools.

There's insufficient space in this column for me to explain why and how RMA works. But I've written on these topics before and recommend you start with "Introduction to Rate-Monotonic Scheduling"⁶ and then read my column "3 Things Every Programmer Should Know About RMA."⁷

Please know that if you don't use RMA to prioritize your tasks and ISRs (as a set), there's only one entity with any guarantees: the one highest-priority task or ISR can take the CPU for itself at any busy time—barring priority inversions!—and thus has up to 100% of the CPU bandwidth available to it. Also note that there is no rule of thumb about what percentage of the CPU bandwidth you may safely use between a set of two or more runnables unless you do follow the RMA scheme.