With the mainstream adoption of server virtualization technology over the past few years, I have noticed a proliferation of partitioning technology across all layers of the technology stack – from the server to the network and storage layers, all the way up to the application. And, while the concept of partitioning (the act of dividing something into parts) is simple, the capabilities and features that come from the use of this powerful design pattern are fairly complex.
In this two-part series, I will discuss some key places where we see partitioning technology utilized and explore some of the impacts and implications that these technologies have within the data center.
Partitioning has been a fundamental concept within computer science for over 50 years. Early mainframe computers leveraged partitioning to share compute, memory, and storage resources among different users on the system. And in the late 1960’s, IBM figured out how to partition a single physical machine into multiple virtual machines, each of which could run an independent copy of an operating system.
Although it took more than 40 years for the industry to commoditize this concept of server partitioning, once it took hold, server virtualization instantly became a pervasive technology within data centers all across the globe. It is interesting to take a look at the impact that this trend is actually having on the technology ecosystem.
One of the first sets of problems that server virtualization provoked was related to device sharing. As expected, when you introduce the concept of sharing resources among different users, there is an obvious concern over both security (is my data secure?) and performance (How much slower is my IO going to be? Can another user take all the bandwidth on the device?). These concerns have driven major changes in both the network and storage ecosystems.
On the network side, the industry response was two-fold. The first was to develop a specification (Single–Root IO Virtualization or SR–IOV for short) for allowing the bare metal operating system (the hypervisor) to partition the physical network interfaces into separate “virtual” interfaces, each of which would be treated fundamentally like it was an independent network endpoint.
The value of this approach was to create an industry standard way for PCI-based devices to effectively partition a physical network port. This approach to partitioning requires that the bare metal operating system take an active part in configuring these devices.
The second response was that the leading NIC vendors began to provide proprietary partitioning capability directly in their hardware (Figure 1 ). These multi-channel capabilities allowed a single physical network port to look like multiple physical network ports to the operating system without the need for the operating system to configure the device – it is discovered by the operating system just like a standard NIC port.
Of course you still need to configure the virtual ports, but in this case the configuration is done through the BIOS or special management software provided by the NIC vendor. Some of the vendors even provide quality-of-service capabilities to control and monitor the amount of traffic on that interface, making it feel like an independent device.
In the end, both of these technologies provide the same high-level benefit, which is to allow server virtualization software to carve off a “virtual” NIC when it needs to provide an isolated “dedicated” network endpoint to a guest operating system.
This use of partitioning at the network interface level has had a substantial impact on the data center network. The proliferation of network endpoints has caused a dramatic increase in number of MAC addresses (and IP addresses) that have to be managed. It has put pressure not only on how we manage these devices but also on network infrastructure itself.
Switches have had to extend their models to support this one-to-many paradigm. Configuration items that were done once for a given physical port now have to be done potentially multiple times on each port (one for each virtual NIC). This puts pressure on the scalability of the switches, including their internal table limits.
The larger number of endpoints also drives an increase in the number of layer2 networks; this has created pressure on the fundamental way in which networks get partitioned today (i.e. VLANs). For example, the number of VLANs is limited to 4096 on most switches. This represents a scalability concern.
The other interesting impact on the network is the dynamic nature ofguest operating systems. Virtualized environments support fluid mobilityof a guest operating system from one host server to another to helpbalance workloads across the cluster and to manage server maintenance.
Inorder to support this capability, data center administrators have hadto provide open access to the network for every host in the cluster.This has driven concerns not only with regard to security, but alsoregarding the large increase in the amount of configuration that isneeded in the switch for each host.
On the storage side, thefiber channel (FC) industry's response has been fairly analogous to thenetwork industry response – both in terms of benefit and approach.Server virtualization drove the need for isolation and performanceoptimization in the shared storage networks used in these environments.
Toattack those problems, the storage adapter vendors leveragedpartitioning technology to allow a single host bus adapter (HBA) port toappear as multiple HBA ports to the operating system. This allows eachguest operating system to have an independent HBA, which provides asecure, isolated storage endpoint. And, much like the network, theimplication of this had consequences into the data center environment.
Theneed for HBA partitioning caused the FC storage industry to extend theFC specification (i.e. NPIV) so that the FC fabrics could deal with asingle physical port having multiple storage names (i.e. WWN). And thesheer number of storage end points puts pressure on the scalability ofthe FC switches both in terms of the management and internal resources.
Anotherplace where the industry has leveraged partitioning is in convergednetwork adapters. Leading industry NIC providers have figured out how topartition a network interface port into a combination of virtualnetwork and storage ports. This is interesting because it provides amajor simplification for the data center environment by reducing theinfrastructure that is needed to run their applications.
This technology has become particularly important with the mainstream adoption of 10Gb Ethernet networks (Figure 2 ).Servers are now being built with 10Gb network interfaces directly onthe motherboard, which provides the needed bandwidth to share bothnetwork and storage traffic on the same physical port.
Figure2: Over the next five years, partitioned 10 Gb Ethernet will constitutethe majority of network interface design implementations.
Thiskind of advanced partitioning has also influenced the technologyecosystem. If you want to take advantage of this capability, then theswitches in your datacenter need to have the ability to deal with theconvergence of both network and storage protocols on the same physicalwire. In order to support this, the industry has created newspecifications to allow fiber channel traffic to flow over standardEthernet networks (i.e. FCoE).
This is important because storagenetworks typically have different requirements than traditionalEthernet networks. Storage traffic tends to be much more sensitive todata loss and latency (the amount of time it takes for data to get tothe storage array) and so new capabilities had to be added to theEthernet network to allow storage traffic to be prioritized with aguarantee to avoid data loss. This technology has even made its way intothe storage arrays – the major storage array vendors now providesupport for FCoE ports on their storage arrays.
One of the hiddengems in this partitioning technology is the ability to provide IOvirtualization for bare metal operating systems. This is powerful as abuilding block in creating an extremely flexible infrastructureenvironment – one where all the servers can be anonymous resources thatcan run any application and operating system by dynamically programmingthe IO personality that is needed for that application.
This isuseful in both virtualized environments and in native environments. Itallows a host server to be provisioned through software and allows foran extremely efficient use of compute resources with capabilities likeN+1 hardware failover, capacity on demand, day/night sharing, and adramatically simplified disaster recovery for your infrastructure. Forenterprises and cloud providers, in their quest to have the mostflexible infrastructure, these partitioning technologies can help drivethe operational efficiency needed to manage the most complexenvironments.
In the second part of this series I will focus onpartitioning for IO virtualization, how it is achieved, and the benefitsthat result in providing a more flexible and efficient infrastructure.
Scott Geng is CTO and Executive Vice President of Engineering at Egenera ,and has been instrumental in the design and development of thecompany’s Processing Area Network (PAN). Prior to joining Egenera, Gengmanaged the development of leading-edge operating systems and middlewareproducts for Hitachi Computer Products and was consulting engineer forthe OSF/1 1.3 micro-kernel release by Open Software Foundation. He holdsBachelor of Arts and Master of Science degrees in computer science fromBoston University.