Caveat To The General

Selecting a processor architecture for your next project is no trivial matter. How are you going to ensure a minimum acceptable level of performance? One answer is benchmarking. If you want to stir up controversy, nothing will do it faster than a rousing debate on the value of benchmarks. In principle, developing benchmarks is a noble undertaking. Benchmarks are potentially valuable tools for evaluating the performance of products. Being in favor of benchmarks is like being in favor of Truth, Justice, and the American Way. Nevertheless, the reality is not quite so glowing. It's difficult to create benchmark suites that reflect actual performance. Moreover, claims surface regularly that vendors will architect their products to excel on benchmarks rather than under real-world conditions. That's like teaching kids just how to do well on their SATs rather than providing them with a rich, well-rounded education.

Benchmarks for embedded processors could be a terrific boon to developers. Good ones could help spell the difference between outstanding end products and merely adequate ones. To its credit, EDN has launched an effort to develop a set of industry-standard benchmarking suites for embedded processors. At the helm is EDN technical editor Markus Levy. But while I applaud the idea of developing benchmarks in principle and have much respect for Markus, I have some concerns about how this program is being implemented.

To be a member of the benchmarking consortium, there are two criteria: you must be a microprocessor or microcontroller designer or manufacturer and you must pony up $20,000 a year. No tool vendors or developers are admitted to this select group-the consortium's board of directors voted to exclude them. This gives the chip vendors themselves control of the benchmark suites, which is tantamount to hiring foxes to guard the hen house. As members of this consortium, they can lobby to push through benchmarks that make their architectures look good and defend against those that make them look bad. On the other side of the coin, semiconductor vendors who are not members have no say in the benchmarks; hence, there is considerable pressure for them to participate, whether they want to or not.

This enterprise resembles a similar program that started a few years ago to provide a suite of FPGA benchmarks. Again, this was a situation in which only the chip vendors were offered membership, and again, it took a pretty healthy fee to join. Privately, several of the FPGA vendors expressed dissatisfaction with the methodology. This group was around for a couple of years searching for respect before finally admitting non-vendors. The benchmarking effort and the group's credibility benefited from their inclusion.

Good benchmarks are not easy to come by and developing them is a costly process. Who has deeper pockets to bear those costs than chip vendors? However, because they have a vested interest in the outcome, chip vendors' membership in this consortium is problematic. If they are members, then developers and tool vendors should be as well. After all, developers have the applications and know the performance they need, and vendors who create the tools to support the chips have an intimate acquaintance with the architectures. Any efforts at establishing benchmarks are commendable, but the benchmarking process must transcend vested interests if it's going to benefit the embedded development community.