A new RTOS trace tool

It's tough to see what's going on when in an RTOS. A new toolhelps.

In a recent debate, one that mirrors so many similar discussions over the decades, a couple of smart developers vehemently objected to using an RTOS in most systems. Their complaint is valid; an RTOS brings in plenty of chances for unexpected system behavior. Scheduling isn't deterministic (using the most common approaches) and locks and priority problems can cause system hangs. But any abstraction has its own headaches. The use of an RTOS requires a new level of developer competence and use of different tools. I find that an operating system, despite the potential problems, can greatly improve a lot of systems.

The RTOS is particularly problematic as it's really tough to see what is going on. But most vendors have visualization tools. Examples include TraceX from Express Logic and µC/Probe from Micriµm. In this, the second decade of the 21st century, I think it would be foolish to develop an RTOS-based system without using a tool like these.

Percepio has come out with an OS visualization tool they call the Tracealyzer. It has a bit of recorder code that links into the firmware. Every event recorded on a kernel-call granularity (soon to be available down to the function-call level) stuffs four bytes of data into a buffer, which is passed to the Tracealyzer application on a PC. They claim a “typical” (whatever that means) application will generated about 10 KB/sec of data, and that the recorder's overhead will eat 1 to 3% CPU overhead on most processors.

Over 22 views are offered, and the best way to describe this is by showing some of the screens.

This screen shows kernel calls over time.


Click on image to enlarge.

I like the following screen, which shows where the CPU cycles are going:


Click on image to enlarge.

You can click on one of the peaks to get more detailed data.The next screen shows how the tool can display the value of a counting semaphore or size of a queue:


Click on image to enlarge.

Want to know the real-time relationship of various entities in your code? That's available as well:


Click on image to enlarge.

A huge number of filters make it easy to get more detailed views.

The Tracealyzer needs to know quite a bit about the RTOS. Currently versions are available for FreeRTOS, OpenRTOS, SafeRTOS, RTXC Quadros, and On Time RTOS-32. Prices range from $2k for the most capable version to free for a stripped down product useable with FreeRTOS.

Our applications are getting more complex all of the time, but tools like these make it easier to peer inside the code's behavior.

Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges, and works as an expert witness on embedded issues. Contact him at. His website is .

13 thoughts on “A new RTOS trace tool

  1. I'm not so sure tools like this would really assuage the objections of the people worried by RTOS impact on system behaviour. Very few people are concerned about how the system performs on the bench when everything is flying straight and level, What the na

    Log in to Reply
  2. Jack,
    The information displays look great, but how does the data being collected by the monitoring software get out of my packaged embedded controller to my PC for collection and analysis? You and Percepio's web site were silent on that matter. Do they a

    Log in to Reply
  3. The link wants 10kB/sec. Is that kbytes or kbits?? Either way that would take a big slice out of a J1939 bus. I guess you could perhaps use a limited feature set and load the bus less.

    Log in to Reply
  4. To avoid “unexpected system behavior” I write clean code in native C (or Assembler). To be independent of bugs, added by others, I use my own OS (StartOS). And I always clearly see what every byte and in what address is doing at every moment of time.
    It sa

    Log in to Reply
  5. Asimov_fan,
    The trace upload depends on the operating system. If there is a file system, you can store the data to a file and transfer it to the host PC. In other cases, typically for MCUs, we use the default debugger connection to make a RAM dump. We have

    Log in to Reply
  6. Chdmanning,
    In our current products we store the data to a RAM buffer, and upload on demand using the debugger connection (see the reply to Asimov_fan above). The RAM buffer usage is typically 4 bytes per event, although some events use multiple 4-byte slo

    Log in to Reply
  7. Chdmanning,
    We fully understand that some prefer “bare metal”. An RTOS certainly adds complexity, since it gives greater flexibility and makes it easier to create more complex designs. However, most embedded systems today use an operating system, about 70

    Log in to Reply
  8. Johan.Kraft,
    Thanks for the response.

    We generally do not have debug connections available on my controllers when they are installed in the production environment, but of course the production environment is where all the anomalous behavior occurs. If

    Log in to Reply
  9. Seems like a nice tool. What I would like to see is an effort towards desiging a standard testing interface for RTOSes for RTA, though given the differences in kernel design, it seems that this won't happen soon. This will encourage design and marketing of

    Log in to Reply
  10. Software tracing tools aren't new. Some of the greybeards might remember the ScopeTools products originally developed by RTI in the 1990's (StethoScope, TraceScope, ProfileScope, etc.), which were then sold to Wind River Systems in 2005.

    However, even tho

    Log in to Reply
  11. Tracing tools are certainly nothing new. Tracealyzer however features a novel visualization system with over 20 views (of 6 main classes). All views are interconnected, and the main time line shows both selected application data and system events. If usin

    Log in to Reply
  12. This is a cool tool but it is a very low level one. To raise the abstraction level to a more functional view, it is necessary to retrieve information from the application that is running on top of the RTOS. There is a free tracing tool that does such a thi

    Log in to Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.