These instructions are distinct from accelerator device driver overheads, as these application/protocol stack overheads exist whether acceleration is available or not. The overheads of secure applications and secure protocol stacks are often shocking to users experienced with only the non-secure analogs.

Figure 4 below shows the throughput (Mbps) and CPU utilization of a cleartext forwarding scenario (IPv4 forwarding using the Linux 2.6 TCP/IP stack) and an IPsec forwarding scenario using the Linux 2.6 TCP/IP stack with OpenSwan IPsec.

To highlight the overheads of security protocol processing, IPsec is run in Encapsulating Security Protocol (ESP) mode, with both encryption and authentication disabled (null/null).

This is a non-compliant mode of IPsec (disabling authentication is not allowed) but it demonstrates the substantial performance degradation associated with security protocol processing.

Figure 4. Performance Comparison of IPv4 and IPsec ESP Tunnel (Null/Null)

The measurements shown in Figure 4 were measured on a Freescale MPC8548E device with a Power Architecture e500v2 core running at 1.33 GHz. The network test equipment used in this measurement had a throughput limit of 2 Gbps (2 X 1-Gbps Ethernet links), otherwise the IPv4 performance would vastly exceed the ESP Null/Null performance at all packet sizes.

In fact, it is possible to calculate the CPU frequency divided by the number of packets forwarded, and show that IPsec ESP Null/Null forwarding consumes about 3.2x more CPU cycles per packet forwarded than plain IPv4 forwarding.

At large packet sizes, where IPv4 and ESP Null/Null throughput converges because of test equipment limits, IPv4 has more than 80 percent idle CPU to perform other tasks while ESP Null/Null is still consuming the majority of the CPU's cycles.

Why is a security protocol so much "heavier"' than the non-secure equivalent? The specifics vary from protocol to protocol, but in almost every case, the security protocol requires more complex classification to determine whether the packet needs to be protected (and if so, using which security parameters).

Security protocols are also stateful, as the cryptographic keys that are used to encrypt and/or authenticate the data have lifetimes. These lifetimes may be measured in number of bytes encrypted, number of seconds since the key was first used, number of seconds since the key was last used (idle time-out), or all of the above (first to occur).

Finally, security protocols add header fields to packets, allowing the packet to be forwarded to the other end of the security tunnel without revealing information about the communicating parties.

These headers are often shimmed between existing headers, or they cause memory copies or memory buffer manipulations which aren't necessary in simple forwarding.

ESP Null/Null performance represents the theoretical best-case performance for IPsec ESP with look-aside crypto acceleration. A "perfect" look-aside engine, with zero driver overhead, zero processing latency and ample raw performance could equal ESP Null/Null performance, but not exceed it.

All Security Protocol Stacks Aren't Equal
The performance difference between IPv4 forwarding and IPsec ESP Null/Null forwarding is quite large, but there are also significant differences between IPsec stacks and the operating systems these stacks are running on.

Different Linux IPsec stacks running on the same device can produce very different results. For open source stacks (StrongSwan, OpenSwan, and Netkey), we found a 2x difference between the best and worst stacks, measured at small packet sizes where protocol stack overheads dominate.

While this is smaller than the 3.2x difference between IPv4 and IPsec ESP Null/Null, it is still a very large difference. Proprietary IPsec stacks can widen these differences still further.

The Mocana NanoSec IPsec stack running on Linux 2.6 performed 1.75x again better than the fastest open source stack IPsec tested, although some of this advantage comes from Mocana's crypto acceleration API (described in the next section).

Freescale doesn't have enough data to fully quantify the impact of running the same IPsec stack on the same PowerQUICC device but with different operating systems.

The limited comparison data (throughput and CPU utilization) that we do have indicates that the differences between OSs is smaller than the differences between IPsec stacks, assuming the network drivers are equally optimized for all OSs used in the comparison.