It offers up to 20 Gb/s of accelerated network processing for avariety of applications including IDS/IPS systems, L2-L7 processing innetwork infrastructure equipment, and network security appliances. TheNFP is highly programmable, including many aspects of its PCIe x8(v2.0) host interface.
This programmability allows greater flexibility in how aspects ofthe device are presented to the host system. The NFP supports up 256queues to the host, which are grouped to form endpoints. The queues aregeneric in the sense that they can carry arbitrary descriptors, thus itis possible to create endpoints of different endpoint types.
Example endpoint types are standard NIC style interface with RX andTX queues, optimized packet capture interfaces, or look-a-side cryptointerfaces. The details of these interfaces are under software controlrunning on the NFP, which defines, amongst others, the purpose of theassigned queues, the descriptor formats used, and how and when DMA isinitiated. Endpoints of these different types can be createddynamically at runtime.
Furthermore, for a variety of applications both in the data centeras well as in network infrastructure equipment it is necessary forthese different endpoints to be accessible by different Guest VMsexecuting on the host with low overhead — a IOV solution is required.However, as pointed out earlier in this series, SR-IOV is not designedto support this type of highly dynamic and highly flexible devices.
A more flexible IOV solution
The key insight is that SR-IOV relies on a number of PCI standards andessentially only adds a device enumeration and resource discoverymechanism. It is this mechanism, which is the primary reason for thelimitations of SR-IOV.
With the Netronome IOV architecture, device enumeration is delegatedto a driver running on the host. This driver is specific to the NFP andis capable of managing endpoints on the NFP: creation and removal ofendpoints of arbitrary types.
To the host OS (or hypervisor) the host driver acts as PCI busdriver and enumerates NFP endpoints as standard PCI devices. Itessentially implements a virtual PCI bus. All configuration spaceaccess for devices on this virtual PCI bus are passed to the hostdriver which either emulates them or translates them to accesses on theNFP.
This solution is not dissimilar to the SR-IOV approach. The hostdriver performs the same function as the PF driver for a SR-IOV device(management of VFs) and the PCIM (translating SR-IOV VFs to PCI devicesto the host OS or hypervisor).
However, the Netronome architecture does not require VFs to also beenumerated in hardware. We therefore refer to the NFP endpoints asSoftware Configurable Virtual Functions (SCVFs). Because the Netronomehost driver is not restricted by the SR-IOV device enumerationlimitations it can enumerate arbitrary types of functions on itsvirtual PCI bus.
|Figure4: SR-IOV (left) and Netronome's IOV (right) compared. The primarydifference is that with Netronome's IOV solution different types ofdevices are easily supported.|
Figure 4 above depicts the commonality and difference betweenSR-IOV and Netromone's IOV solution. With the Netronome solution,different types of endpoints (indicated by the different colors) areeasily supported while SR-IOV mandates that all VFs associated with aPF are of the same device type.
With Netronome's IOV solution the PF driver in combination with avirtual PCI implementation performs the same function as the PF driverand the PCIM for SR-IOV.
For both SR-IOV and for Netronome's IOV solution virtual functionsare presented to the host OS or the hypervisor as standard PCI devices,thus they both leverage the mechanism provided by modern hypervisors toassign PCI devices to Guest VMs.
During initialization, each SCFV gets assigned a unique PCI functionID, which the NFP uses to tag DMA requests originating from thecorresponding endpoint. Thus IO MMU page tables can be set upappropriately to provide DMA protection between different SCVFs. EachSCVF also gets assigned one or more MSI vectors from the PFs set ofMSI-X vectors, thus they can directly notify the Guest VMs they areassigned to.
Network virtualization with NFP
The host side interactions of the NFP are identical to a SR-IOVsolution. SCVFs can be assigned to Guest VMs just like VFs or PCIdevices. This provides Guest VMs with low overhead and low latencyaccess to network devices.
Guest VMs run standard device drivers to talk to the SCVF andinterrupts can be delivered directly to the cores on which a Guest VMis executing.
However, since the NFP is a programmable network device, themultiplexing and de-multiplexing of packets from and to SCVFs is notlimited to some fixed function hardware implementation as in mostSR-IOV or MQ NICs.
Instead extensive packet processing can be performed includingflow-based classification, load balancing or filtering. This provides asignificantly more flexible solution than other hardware based IOVapproaches.
Comparing the Various IOV Implementation Options
Table 1 below summarizes and compares the four different IOVoptions discussed above.
|Table1. Four different IOV options.|
The Netronome IOV solution has the advantage of being SR-IOVcompliant while providing flexible device support – Most notably, thecapability to dynamically assign different kinds of virtual functionsat run time.
The result is that a physical NIC can provide multiple virtual NICtypes including a dumb NIC, an intelligent NIC, a crypto NIC or aPacket capture (PCap) NIC.
IOV in the Network Data Center. Next generationnetworked data centers need to address a complex set of issues, suchas: consolidation, service continuity, service flexibility and energyefficiency. Virtualization of servers is already seen as a key factorin moving to a next generation data center.
Without IO virtualization the limitations already discussed in thisseries will severely restrict the extent to which the goals outlinedabove can be met.
A single, multi-core server can easily support 10 to 100 virtualmachines (VMs), allowing numerous applications, which today require adedicated server, to share a single physical server. This allows thenumber of servers in the data center to be reduced while increasingaverage utilization from as low as 5 ” 15% to up to 50 ” 60%.
With multiple VMs running on a single physical machine, there areopportunities for the VMs to cost-effectively share a pool of IOresources, such as intelligent NICs.
In the single application, single server model, each application hasaccess to the entire server's bandwidth. In the virtualized servermodel, however, network bandwidth is shared by multiple applications.
As more applications are consolidated on one server, bandwidthrequirements per server, and server utilization, both increasesignificantly. The result is an intelligent NIC is needed to offloadnetwork processing for the host, keeping the host CPU from becoming thebottleneck that limits application consolidation.
This trend required low overhead delivery of network data directlyto Guest VM. The move to a unified network for all traffic in the datacenter with data and storage networks consolidating onto standardEthernet using technologies such as Fiber Channel over Ethernet (FCoE)or iSCSI over 10Gbps Ethernet is also a driver for IO virtualization.These consolidated data center networks require lower latency andhigher throughput than traditional data only networks.
As noted earlier the different approaches to IO Virtualization havevery different characteristics with regard to latency. Next generationdata center requirements for low latency and low overhead delivery ofnetwork IO directly to Virtual Machines can only be provided byhardware based IOV solutions such as SR-IOV based NICs or Netronome'sIOV architecture.
Software based IO virtualization imposes too high an overhead tohandle the expected network IO demand and IOV solutions based on MQNICs are unsuitable for latency sensitive applications such as FCoE.
As servers and network appliances in the data centers are builtaround commodity multi-core CPUs ” specifically around x86architectures ” and network IO around PCIe, implementing IOV over PCIebecomes critical in allowing the many VMs to share network IO devices.
Data centers also deploy a wide range of advanced network andmanagement technologies within the Ethernet infrastructure, such asextensive use of Access Control Lists (ACLs), sophisticated VLANsetups, Quality of Service, and even some limited Layer 3 and 4processing.
These technologies are readily available in modern networkinfrastructure equipment such as Top of the Rack (TOR) switches.However, even modern SR-IOV based NICs only provide very limited, fixedfunction switching capabilities, creating a disconnect between thesophisticated physical network infrastructure and virtual networkinfrastructure implemented on the host.
An IOV solution combined with intelligent IO processing (IOV-P)bridges this gap and extends sophisticated network processing andmanagement into virtualized servers. An IOV-P based intelligent NIC canimplement the same functionality as modern data center switches,including monitoring and policy enforcement (ACLs), within the server.This enables these policies to be applied even for inter-VM networktraffic without having to be passed through the TOR switches.
Furthermore, the proliferation of encrypted network traffic (IPSecand SSL) provides the opportunity to offload some or all of itsrequired processing to an intelligent NIC, freeing up host CPU cyclesfor application processing.
In summary, Intelligent Ethernet NICs with IOV capability are keyingredient of the virtualized system, as they greatly reduceutilization of the host CPU for network processing, allowing the systemto support a larger number of applications, while saving power.
Adding IOV capability to the intelligent NIC ensures that eachapplication can be configured with its own virtual NIC, allowing anumber of applications to share a single 10GbE physical NIC. At thesame time the IOV-P concept allows a single physical NIC to providemany different “intelligent” functions to the VM and even to create andrefine these functions at run time.
IOV in the networkinfrastructure. In much the same way as in network datacenters, control plane and application layer functions ininfrastructure equipment are built around commodity multi-corevirtualized CPUs.
These will also need an underlying IO subsystem that is alsovirtualized. Such IOV subsystems will be used to implement intelligentservice blades, network appliances, intelligent trunk cards, andintelligent line cards in the core network infrastructure.
Such cards usually serve the various line cards in the system, andwill be best implemented based on the IOV-P. These cards can be anintelligent virtualized NIC supporting 10GbE and above, service bladessupporting multiple services, or trunk cards supporting nested tunnels.
|Figure5: Service blade / intelligent NIC architecture for infrastructureequipment. Integrating IOV-P function in the data plane allows thevirtualized equipment / appliance to support multiple different virtualfunctions.|
Netronome's NFP-3200 integrates the IOV-P
Figure 5 above depicts a virtualized multi-core system, with IOVcapability running multiple applications. Such applications run onmultiple instances of the same OS, or different OS's – These in turnrun on a single core, single VM or multiple cores, multiple VM's.
By classifying network IO traffic into flows, applying securityrules, and pinning flows to a specific VM (virtual machine) on aspecific core on the host, and/or by load balancing various flows intovarious VMs, the IOV-P enables the overall system to achieve fullnetwork performance at 10Gbps and beyond.
In today's environment, servers and network appliances in the datacenters, and control plane and application layer functions ininfrastructure equipment, are increasingly being built around commoditymulti-core CPUs ” specifically around the x86 architecture.
This CPU subsystem is being virtualized for efficient use of CPUs,better isolation, security, ease of management, lower cost, and lowerpower. This trend is expected to accelerate.
As these network servers, appliances and equipment are virtualizedat the CPU level, they need an underlying IO subsystem that is alsovirtualized. The IOV-P provides an ideal solution for IOvirtualization. What remains to be seen, however, is the speed at whichvendors will adopt SR-IOV for such virtualization.
Nabil Damouny is the senior director of businessdevelopment at NetronomeSystems . He has a BSEE from Illinois Institute of Technology(IIT) and a MSECE from the University of California Santa Barbara(UCSB). He holds 3 patents in computer architecture and remotenetworking.
Rolf Neugebauer is a Staff Software Engineer at NetronomeSystems were he works on virtualization support for Netronome's line ofIntelligent Network Processors. Prior to joining Netronome, Rolf workedat Microsoft and Intel Research. At Intel he was one of the initialresearchers developing the Xen hypervisor in collaboration withacademics at Cambridge University. Rolf holds a PhD and a MSc from theUniversity of Glasgow.