For embedded system developers, benchmarking seems to consume as much time aswriting code and debugging – finding the most appropriate way to evaluatehardware and software and analyzing the effect of the software code or hardwaredesign on performance, power and half a dozen or so other importantparameters.
But as Ron Wilson points out in his new #include column, “Benchmarking is just the start ,” it's alwaysbeen difficult to get even a suite of benchmarks to accurately represent a realchunk of critical-path code. A variety of factors conspire against benchmarknumbers, he says, even as systems get faster, more powerful and more programladen. All make accurate assessment even more important.
However, there are as many so-called benchmarking solutions – and benchmarkresults – as there are processor architectures, application environments, andspecific design and – unfortunately – marketing goals.
In “Benchmarkbamboozles,” Jeff Bier goes through some of the things you need to watch outfor when vendors use such numbers in a misleading way. For all the difficulties,though, in “Don’tban benchmarks,” Jeff points out that if used properly and interpretedcorrectly, benchmarking will continue to be an invaluable aid in your work.
An excellent aid in your quest to find the right benchmarking methodology ismy Editor’s Top Pick: “Why MIPS is just a number ,” by Gaurang Kavaiya,the cover story in the October 2010 issue of ESD Magazine (print anddigital ). Init he goes through the various ways benchmarking is done in embedded designs andpresents an alternate approach he thinks will be more useful.
Of the several dozen embedded design articles on benchmarking issues that wehave run recently, a few that I recommend include:
“Evaluating the performance of multicore processors ,” by Max Domeika,
“Howto choose benchmarks appropriate to the system you’re designing,” by JeffReilly and Dave Salvator,
“Tricks & techniques for performancetuning ,” by Peter Barrt abd Gerard Hartnett, and
“Howto calculate CPU utilization ,” by Michael Trader.
The challenge of determining correct and useful benchmarks is not going away,and will be more important – and difficult – to do in the future. So I hope toreceiving contributions from you for use on line, in ESD Magazine and in theEmbedded.com newsletter: your problems, your solutions, your tools and yourapproaches to benchmarking. (EET/Embedded.com Editor Bernard Cole,email@example.com, 928-525-9087)