LONDON — A benchmark in the works for embedded processors aims to provide a free, open-source alternative to the well-established suite of EEMBC benchmarks created by paying members. A small group of mainly academics are calling for support for EmBench, which they hope to release in a 0.5 version before the end of the year.
EmBench aims to deliver a single performance score based on a suite of about 20 real-world applications, mainly sourced from an earlier effort, the Bristol/Embecosm Embedded Benchmark Suite (BEEBS). It also plans to report metrics for code size and latency but not floating-point performance or power consumption.
Figures will generally be reported based on a geometric mean and geometric standard deviation to a reference platform. It tentatively picked PULP RI5CY, an open-source 32-bit RISC-V core from the Integrated Systems Lab at ETH Zürich in Switzerland, as its reference.
Since January, Patterson has worked on EmBench in monthly meetings with Palmer Dabbelt of SiFive, Cesare Garlati of Hex Five Security, G. S. Madhusudan of the India Institute of Technology Madras, Trevor Mudge of the University of Michigan, and Jeremy Bennett, head of an embedded software company that created BEEBS.
The idea emerged at a RISC-V workshop, when SiFive co-founder Yunsup Lee gave a presentation that used synthetic benchmarks to a group including Patterson. Patterson’s textbook on microprocessor design, co-written with John Hennessey, preached against use of synthetic benchmarks such as Dhrystone.
“Lee was embarrassed to do this in front of his former professor, and I was pissed off that people were still using synthetic benchmarks,” Patterson said in an interview. “My experience with MLPerf showed me [that] a small group could move fast, so I thought we could finally kill off Dhrystone and have a good benchmark for embedded computing. I talked to a few people at the workshop and they liked the idea. People are making changes to hardware to make Dhrystone run better. Everyone realized it was stupid, but no one took the initiative to fix it.”
EmBench also aims to be an alternative to the widely cited CoreMark, one of more than a dozen embedded benchmarks developed by the EEMBC trade group. EEMBC currently has about 20 active members, mainly embedded chip vendors such as Arm, Intel, and Renesas.
“EEMBC is fine for what they do with a paid membership model, but CoreMark is a synthetic benchmark — EmBench is an alternative model,” Patterson said.
The EEMBC website lists 17 benchmarks that it has developed since it was funded in 2000 or currently has in the works. Its most recent offerings include benchmarks for IoT, security, and advanced driver assistance.
For his part, EEMBC’s director left a door open for collaboration.
“My first reaction would be to see if we can work together [being that] EEMBC has support of the industry already,” said Peter Torelli, head of EEMBC. “I’m not opposed to permissively licensed benchmarks, so why not leverage what EEMBC has been running for 20 years instead of reinventing the wheel?”
An analysis of widely used processor benchmarks. Click to enlarge. (Source: David A. Patterson)