SoC Measurement Methodology
Accelerator benchmarks emphasize the performance of the accelerator and its path to system memory. SoC benchmarks emphasize the performance of all elements in the processing path, including the network interfaces, CPU(s), accelerator and memory subsystem. For networking-oriented SoCs such as those in the PowerQUICC family, an important cryptographic benchmark is IPsec packet processing.

The difference between encryption algorithms and a full security protocol such as IPsec were described in Part 1 of this article, along with software variables such as IPsec protocol stack efficiency and the quality of the API for exploiting hardware acceleration.

This section describes a methodology for creating normalized comparisons of different IPsec stacks on the same device, and comparisons of the same IPsec stack on different devices.

Network Test Environment
RFC2544 "Benchmarking Methodology for Network Interconnect Devices" describes a range of methods for the standardized testing of networking systems. Although the RFC doesn't specifically discuss IPsec, it introduces a number of important testing concepts.

The most logical way to test the IPsec performance of a system—Device Under Test, or DUT—could be to connect it to a network tester as shown in Figure 2 below. Coincidentally, this scenario is also Figure 2 below in RFC2544.

Figure 2: IPsec Testing Option 1

Using this option, the tester generates IPsec packets and sends them to the DUT, which then decrypts them and sends them back. The process can also be run in reverse so that the DUT's performance during encryption is also measured.

While this might be the most logical way to measure a DUT's IPsec performance, this setup is rarely used due to the cost of networking testers that support high IPsec data rates.

While this situation may change in the future, advanced networking SoCs with cryptographic acceleration often have more IPsec performance than the network test equipment that would purport to measure them. Figure 3 below (also Figure 3 in RFC2544) shows a better option for testing IPsec.

Figure 3: IPsec Testing Option 2

In option 2, IPsec is tested in a gateway-to-gateway configuration.

1. A network tester transmits clear text IP packets from port 1 to DUT 1 (labeled Home).
2. DUT 1 IPsec encrypts the packets and sends them to DUT 2 (labeled Office). 3. DUT 2 decrypts the packets and
4. Forwards them back to the network test's port 2, allowing the tester to measure received packet rate.

This process can be run bi-directionally so that each of the network tester's ports send and receive clear text packets, as shown in Figure 2 earlier. Many network testers are able to generate high rates of clear text IP traffic, allowing a broader set of users to recreate benchmarks measured using this method.

Because all packets are both encrypted and decrypted, IPsec performance is reported as the slower of the two operations. Also note that performance is measured in terms of clear text packets.

Depending on packet size, the Mbps measured inside the encrypted tunnel between the two DUTs could be nearly two times the Mbps reported by the network tester. The increase in packet size inside the tunnel also causes the 1 Gbps Ethernet link between the DUTs to saturate at approximately 950 Mbps, rather than approximately 990 Mbps for a non-encrypted Ethernet link.

Sources of variation in IPsec testing
It is beyond the scope of this two part series to cover all the potential variables impacting IPsec benchmarks using measurement option 2. However, there are two related testing options worth mentioning: injection rate and acceptable loss rate.

Injection rate is the rate at which the network tester transmits clear text packets to the DUTs. The tester can be configured to either flood the DUT (transmit at 100 percent line rate for all packet sizes), or start slowly and gradually increase the injection rate until the DUT starts dropping packets.

Flood testing generally produces better IPsec benchmarks than zero loss, but flood benchmarks are less useful to users because most systems are designed to achieve a given performance target without dropping large numbers of packets. Are those your zero loss packets on the floor?

RFC2544 doesn't have a concept of acceptable loss rate. Zero loss testing means performance is defined by the rate at which the DUT can forward packets for some time interval (typically 30-60 seconds) without losing any packets. This part of RFC2544 is generally ignored by networking equipment vendors when testing their end systems.

Independent testing labs, such as the Tolly Group, have been known to measure IPsec "zero loss" performance using an acceptable packet loss rate of <.001 percent. Both equipment and embedded processor vendors have been seen to report loss rates as high as .1 percent as zero loss.

Acceptable loss rate has a significant impact on performance. Freescale has reported zero-loss IPsec results using both absolute zero loss and <.001 percent, with the <.001 percent results typically 25 percent better than the absolute zero results.

Higher acceptable loss rates improve the results still further. Because there is no universally agreed-upon definition of acceptable loss rate for a benchmark reported as zero-loss, this is an area where a prospective buyer needs to do some extra homework before comparing two vendor-supplied benchmarks.