A new RTOS trace tool
It's tough to see what's going on when in an RTOS. A new toolhelps.
In a recent debate, one that mirrors so many similar discussions over the decades, a couple of smart developers vehemently objected to using an RTOS in most systems. Their complaint is valid; an RTOS brings in plenty of chances for unexpected system behavior. Scheduling isn't deterministic (using the most common approaches) and locks and priority problems can cause system hangs. But any abstraction has its own headaches. The use of an RTOS requires a new level of developer competence and use of different tools. I find that an operating system, despite the potential problems, can greatly improve a lot of systems.
The RTOS is particularly problematic as it's really tough to see what is going on. But most vendors have visualization tools. Examples include TraceX from Express Logic and µC/Probe from Micriµm. In this, the second decade of the 21st century, I think it would be foolish to develop an RTOS-based system without using a tool like these.
Percepio has come out with an OS visualization tool they call the Tracealyzer. It has a bit of recorder code that links into the firmware. Every event recorded on a kernel-call granularity (soon to be available down to the function-call level) stuffs four bytes of data into a buffer, which is passed to the Tracealyzer application on a PC. They claim a “typical” (whatever that means) application will generated about 10 KB/sec of data, and that the recorder's overhead will eat 1 to 3% CPU overhead on most processors.
Over 22 views are offered, and the best way to describe this is by showing some of the screens.
This screen shows kernel calls over time.
Click on image to enlarge.
I like the following screen, which shows where the CPU cycles are going:
Click on image to enlarge.
You can click on one of the peaks to get more detailed data.The next screen shows how the tool can display the value of a counting semaphore or size of a queue:
Click on image to enlarge.
Want to know the real-time relationship of various entities in your code? That's available as well:
Click on image to enlarge.
A huge number of filters make it easy to get more detailed views.
The Tracealyzer needs to know quite a bit about the RTOS. Currently versions are available for FreeRTOS, OpenRTOS, SafeRTOS, RTXC Quadros, and On Time RTOS-32. Prices range from $2k for the most capable version to free for a stripped down product useable with FreeRTOS.
Our applications are getting more complex all of the time, but tools like these make it easier to peer inside the code's behavior.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges, and works as an expert witness on embedded issues. Contact him at. His website is .
I'm not so sure tools like this would really assuage the objections of the people worried by RTOS impact on system behaviour. Very few people are concerned about how the system performs on the bench when everything is flying straight and level, What the na
Jack,
The information displays look great, but how does the data being collected by the monitoring software get out of my packaged embedded controller to my PC for collection and analysis? You and Percepio's web site were silent on that matter. Do they a
The link wants 10kB/sec. Is that kbytes or kbits?? Either way that would take a big slice out of a J1939 bus. I guess you could perhaps use a limited feature set and load the bus less.
To avoid “unexpected system behavior” I write clean code in native C (or Assembler). To be independent of bugs, added by others, I use my own OS (StartOS). And I always clearly see what every byte and in what address is doing at every moment of time.
It sa
Asimov_fan,
The trace upload depends on the operating system. If there is a file system, you can store the data to a file and transfer it to the host PC. In other cases, typically for MCUs, we use the default debugger connection to make a RAM dump. We have
Chdmanning,
In our current products we store the data to a RAM buffer, and upload on demand using the debugger connection (see the reply to Asimov_fan above). The RAM buffer usage is typically 4 bytes per event, although some events use multiple 4-byte slo
Chdmanning,
We fully understand that some prefer “bare metal”. An RTOS certainly adds complexity, since it gives greater flexibility and makes it easier to create more complex designs. However, most embedded systems today use an operating system, about 70
Johan.Kraft,
Thanks for the response.
We generally do not have debug connections available on my controllers when they are installed in the production environment, but of course the production environment is where all the anomalous behavior occurs. If
Seems like a nice tool. What I would like to see is an effort towards desiging a standard testing interface for RTOSes for RTA, though given the differences in kernel design, it seems that this won't happen soon. This will encourage design and marketing of
Software tracing tools aren't new. Some of the greybeards might remember the ScopeTools products originally developed by RTI in the 1990's (StethoScope, TraceScope, ProfileScope, etc.), which were then sold to Wind River Systems in 2005.
However, even tho
Tracing tools are certainly nothing new. Tracealyzer however features a novel visualization system with over 20 views (of 6 main classes). All views are interconnected, and the main time line shows both selected application data and system events. If usin
This is a cool tool but it is a very low level one. To raise the abstraction level to a more functional view, it is necessary to retrieve information from the application that is running on top of the RTOS. There is a free tracing tool that does such a thi
Regarding .. A framework is an “application skeleton” .. we implemented something related to this that can show timeline from user “application” level. Check http://www.timetopic.net.