Related to the strong demands for virtualization technology, network I/O virtualization has recently become a hot topic and is one of the key topics being discussed at the upcoming Multicore Expo @ ESC.
In fact, analyst firm IDC, in a recent white paper titled “Optimizing I/O Virtualization: Preparing the Data Center for Next-Generation Applications”, stated that “If I/O is not sufficient, then it could limit all the gains brought about by the virtualization process.”
While first generation hypervisors, targeting commodity Intel-based servers, focused primarily on efficient CPU and memory virtualization, we are now seeing a flurry of activity focusing on network and network I/O virtualization. There are a number of key drivers for this trend:
* 10 Gigabit Ethernet (10GbE) has become the de-facto standard in data centers. Network packets at high data rates must be delivered efficiently to individual virtual machines.
* Data center deployments require integrated network management capabilities across all networking elements, including the network I/O virtualization layer providing network connectivity to virtual machines.
* I/O Convergence: Storage and network I/O are being consolidated onto the same networking infrastructure. These converged networks require different packet policies and additional network management interfaces being provided by the network I/O virtualization layer.
* Increased focus on network security: Network traffic in data centers is increasingly subjected to fine-grained security policies, such as firewalling or Intrusion Detection/Prevention Systems (IDS/IPS). Cloud-based data centers only exacerbate the need for fine-grained security policies to enforce isolation between untrusted entities.
* Emerging new use cases, such as VM migration, require additional mechanisms and policies being implemented at the network virtualization layer.
In order to meet the 10GbE performance requirements, the overheads of the current network I/O virtualization solutions need to be significantly reduced.
Additionally, the new solutions should provide more data center-like network management features and enforce fine-grained policies on network traffic.
Addressing Network I/O Performance Requirements
There are a variety of different approaches to address the challenge of network I/O virtualization.In one method, the Cisco Unified Computing System (UCS), as well as the recent Virtual Ethernet Port Aggregation (VEPA) proposal, move management and policy enforcement into a Top-of-Rack (ToR) switch.
It provides low overhead virtual links to VMs using virtualized Converged Network Adapters (CNAs). Essentially most of the network I/O virtualization layer is removed from the server and moved into the ToR switch.
While UCS and VEPA provide a solution to the challenges of management and efficient data delivery, they do not directly address the increased need for applying fine-grained security policies. In UCS and VEPA based environments, application of security policies has to be deferred to external security appliances, complicating the flow of network traffic.
In a second approach, common SR-IOV based NICs, like CNAs, offer a low overhead delivery mechanism of network packets directly to and from VMs by providing VMs direct access to a portion of the physical NIC. This suitably addresses challenges of delivering 10GbE efficiently to virtual machines.
However, available SR-IOV NICs only implement very basic switching of network packets and provide even less functionality for enforcing policies on network traffic, far from even being remotely suitable for supporting the key drivers outlined above.
In yet another solution, virtualization software vendors and their partners, on the other hand, are already offering or are actively developing a new breed of virtual switches (vSwitches). These vSwitches implement data center-level network management features and fine-grained policy enforcement as a software component integrated into the server's virtualization layer.
Prominent examples are VMware's distributed switch, the Cisco Nexus 1000V switches, and the Open vSwitch initially targeted at Xen environments. These vSwitches offer the flexibility and management features required by most of the key drivers outlined above.
However, since they are host-based software solutions they incur very high overheads on the data path. Furthermore, vSwitch implementations cannot take advantage of the efficient data delivery to VMs offered by common SR-IOV NICs as they are incapable of fine-grained policy enforcement.
Finally, I/O gateways offer one more approach. This new class of I/O virtualization solutions is being developed by various companies such as Xsigo, NextIO, Aprius and VirtenSys.
Conceptually this approach is similar to UCS and VEPA; however, a central I/O gateway typically performs network I/O virtualization by sharing standard-off-the-shelf SR-IOV NICs. And, unlike UCS or VEPA, the individual servers are not connected to the I/O gateway via Ethernet, but instead by using InfiniBand or, more commonly, via cabled PCIe.
This approach addresses the need to efficiently deliver network traffic directly to VMs and also offers a unique solution for emerging applications, such as VM migration from the VMS point of view, as its devices can transparently move with the VM. And, similar to UCS and VEPA, the application of fine-grained security policies must be deferred to external security appliances.
The Need for a Network I/O Coprocessor
The increased need for the application of complex network security policies in data centers, and in particular cloud deployments, requires fine-grained control and processing of individual network packets and flows of packets at high data rates.
Using commodity hardware to implement these security functions is appealing due to their ease of programming. Furthering this need is related to the fact that even the latest commodity, IA-based servers are ill suited for certain packet processing tasks such as processing small packets at data rates of 10GbE and beyond.
I/O coprocessors are ideally suited for implementing network I/O virtualization, while accelerating vSwitch capabilities. I/O coprocessors are best suited for both I/O gateways and server-based approaches above ” as a result, they need to integrate the following key functions:
* Intelligent, tasteful, flow-based switching
* Integrated IOV
* Load balancing
* Integrated security
* Glue-less interface to, and tight-integration with Intel CPU subsystem
In addition, such highly programmable network I/O coprocessors, designed to work closely with commodity Intel server platforms, make a perfect match for security appliances for the UCS and VEPA approach described above.
Summary & Conclusion
There are a variety of approaches for addressing the challenge of network I/O virtualization in the virtualized data center; some are here today while others are still in the standardization process. It remains to be seen how all these approaches will coexist, and which one(s) will be widely used in the short and long-term.
In all of these cases, programmable, stateful, flow-aware processors capable of providing 40Gbps of L2-L7 network I/O can help remove the bottlenecks in highly virtualized multicore systems.
Nabil Damouny is Senior Director of Marketing at Netronome where he is responsible for defining and positioning Netronome's silicon and board-level product offerings, including its programmable Network Flow Processor family (NFP-32xx) with built-in I/O coprocessor capability.
He was a founder and Vice President of Marketing and Business Development at Basis Communications, acquired by Intel Corp. in 2000. He then spent five years at Intel in strategic development. He earned a BSEE from IIT Chicago, and a MSECE from UCSB. Nabil holds three patents in computer architecture and remote networking.
Rolf Neugebauer is a Staff Software Engineer at Netronome Systems were he works on virtualization support for Netronome's line of intelligent Network Flow Processors. Prior to joining Netronome, Rolf worked at Microsoft and Intel Research. At Intel he was one of the initial researchers developing the Xen hypervisor in collaboration with academics at Cambridge University. Rolf holds a PhD and a MSc from the University of Glasgow.