PCI Express MR-IOV Solution
As noted earlier, PCIe has been perceived as a single-host interconnect technology. But thanks to the recent release of new standards and off-the-shelf silicon solutions, PCIe is now a feasible solution in multi-host systems as a switch fabric technology for data centers and enterprise IT applications. The presence of native PCIe interfaces (ports) in all servers and I/O sub-systems makes PCIe a very attractive candidate for future switch fabrics in small and mid-sized clusters.

In 2007, the PCI-SIG (the consortium responsible for PCI, PCI-X, and PCIe) released the Single-Root I/O Virtualization (SR-IOV) specification that enables the sharing of a single physical resource such as a NIC or HBA in a PCIe system among multiple SIs running on one host.

This requires I/O vendors to develop devices that support the SR-IOV specification. This is the simplest approach to sharing resources or I/O devices among different applications. It solves the problem of the CPU being burdened by having to run the software for virtual I/Os. Instead, it moves the burden to the SR-IOV-capable endpoints.

Last year, the PCI-SIG completed work on its Multi-Root I/O Virtualization (MR-IOV) specification that extends the use of PCIe technology from a single-host domain to a multi-root (i.e., multi-host) domain. The PCI and PCIe standards are developed around a single host model where one host controls and uses all I/Os in its domain.

The MR-IOV specification enables the use of a single I/O device by multiple hosts and multiple system images simultaneously, as illustrated in Figure 2 below. This illustration shows a multi-host environment where a single MR-IOV-capable NIC and HBA are shared across multiple systems via an MR-IOV switch.

Figure 2. MR-IOV specification enables the use of a single I/O device by multiple hosts and multiple system images simultaneously.

Such a solution would unify the servers, switch fabrics and I/Os used in data center applications. In this approach, servers would run multiple applications and SIs, MR-IOV-capable PCIe switches would provide connectivity, and MR-IOV-capable endpoints would provide the sharing and virtualization of I/O resources.

In order to implement MR-IOV specifications, three components of the system need to be developed " MR-IOV PCIe switches, endpoints, and management software. All three of these components must be available simultaneously and work seamlessly.

Figure 3 below illustrates an example of a PCIe MR-IOV switch. This switch would allow association of a switch downstream (or endpoint) port to any of the host ports. The endpoint connected to a downstream port can be legacy, SR-IOV- or MR-IOV-capable, but the endpoints servicing multiple hosts must be MR-IOV-compliant.

Figure 3. Example of a PCIe MR-IOV switch

Unfortunately, the complexity and manageability of the MR-IOV switches and endpoints make the implementation of the MR-IOV standard very difficult, as no single vendor has the capabilities and the commitment to develop a completely seamless solution.

Unless the system (software) vendors, PCIe switch vendors, and PCIe I/O (endpoint) developers collaborate and commit resources to develop and deliver an interoperable solution, we will not see an MR-IOV implementation anytime soon.