Evaluating the performance of multi-core processors - Part 2One common question when customers review data from microprocessor benchmarks such as those discussed in Part 1 is "How well will benchmark performance predict my particular application's performance if I employ your new processor or new compiler?"
In other words, if a new processor or different compiler increases the performance of benchmark X by Y%, how much will the processor or compiler benefit my application?
Of course, the answer is: it depends. Your application is not exactly like the benchmark program, and while the processor architects and compiler engineers may use the benchmark to assess the performance of features they are adding during development, they do not necessarily have access to or tune specifically for your application.
Therefore, you should be skeptical if someone claims that you will see the same performance benefit from using a new processor or compiler as what is shown from the benchmark data.
That said, you should expect some performance improvement and there are a couple of statistical techniques that can be employed to help improve the degree of confidence you have in benchmark data in estimating your end application performance.
There are three techniques to use when attempting to characterize your application's potential performance improvement based upon benchmark data and they are summarized as:
1) Assume the performance improvements correlate with the improvements observed in the benchmark.
2) Compute the correlation between your application and a number of benchmarks and then use the performance improvement from the benchmark with highest correlation.
3) Compute a multivariable regression using historical benchmark and application data to estimate a function, f ( x ) _ y , where x is the performance observed on the benchmarks and y is the expected performance of your application.
The first technique and second technique are fairly easy to implement. The third technique employs a multivariable regression to estimate your application performance using the following historical data as inputs in the calculation:
1) Benchmark data for a number of past processors and the new processor of interest
2) Your application ' s performance on the same past processors for which you have benchmark data
Consider the performance data in Table 3.1, downloadable as a two page PDF file. The performance results of each benchmark that comprises SPEC CINT2000 is shown for a number of processors including the Intel Pentium 4 processors, Intel Core Duo processors, and Intel Core 2 Quad processors.
The column labeled " Processor " indicates the processor on which the benchmark was executed. The column labeled " MHz " indicates the clock speed of the processor. The scores for the individual benchmark tests comprising CINT2000 are then listed.
The " Your App " column is hypothetical data 1 and indicates the performance of your application by assuming you have measured the performance on each of the processors listed. ( The data is actually from the CINT2000/300 Base benchmark that serves as the data to predict in this example.
The row labeled " Correlation " is the degree of correlation between the individual benchmark data and the data in the " Your App " column. The " MV Estimate " column was created by employing a multivariable regression where the CINT2000 benchmarks are the independent variables and the " Your App " data is the dependent variable.
Finally, the baseline column indicates the overall CINT2000 rating that is part of the historical SPEC data. If you were considering moving your application from a Dual-Core Intel Xeon 5160 processor-based system with a frequency of 3 GHz to an Intel Core 2 Extreme processor X6800 with a frequency of 2.93 GHz, Table 3.2 below shows a number of different performance estimates from the previous table.
The assumption in this scenario is that you have not executed the application on the Intel Core 2 Extreme processor X6800 and thus would not have the " Actual" performance number ( ??? in Table 3.2 below) for it yet.
|Table 3.2 : Overall performance estimate comparison|
The percentage difference for the "baseline" column suggests that your application would receive a 6.11% performance increase from moving to the new processor. If you applied the performance improvement from the benchmark that had the highest degree of correlation (technique #2) with your application (175 Base with .994 correlation), a 2.22% performance improvement is suggested.
If you applied a multivariable regression to the data set, the performance decrease is estimated at 1.68%. The actual performance decreased by 0.99% which in this example shows the multivariable regression estimate as being the closest.
There is no guarantee that employing correlation or a multivariable regression will lead to a better prediction of application performance in all cases, however, reviewing three estimates for performance improvement compared to one does provide a greater degree of confidence.
To summarize, armed with historical data from CPU2000 and your application on a number of processors, it is possible to generate two other estimates for the expected performance benefit of moving to a new processor without executing the application on the processor.
The estimates based upon degree of correlation between individual CPU2000 benchmarks and on a multivariable regression may provide a more accurate indication of expected performance.
|Figure 3.10 : Image rotation performance|
|Table 3.2 : Overall performance estimate comparison|