Using multi-root (MR) PCIe to extend nextgen multi-host storage & server switch fabrics
MR PCIe Switch Solution
As the market was looking for an MR-IOV solution,
PCIe switch vendors have created MR PCIe switches that allow the implementation of a subset of the features MR-IOV offered, which some industry experts have dubbed a "poor man's MR-IOV." (
In our case, PCIe Gen 2 MR switches range from 96 lanes and 24 ports to 48 lane and 12 ports, which give system designers quite a bit of latitude in MR implementation.)
This new generation of MR switches allow multiple hosts to be connected to a single switching device, which can be portioned under user control in such a way that each host will be connected to a desired set of downstream ports of the switch.
In this MR switch solution, each host operates independent of other hosts and controls I/Os in its domain without seeing or impacting traffic associated with the other hosts. Figure 4 below illustrates the internal architecture of an MR switch where particular sets of downstream ports are associated to particular host ports through partitioning (as shown by the dotted lines in Figure 4).
 |
| Figure 4. Internal architecture of an MR switch |
The internal peer-to-peer (P2P) bridges, virtual bus, and downstream ports for a specific host port are completely isolated from the other host ports and their associated P2P bridges. These switches can either be used as standard PCI Express switches or partitioned in up to eight independent switches under user control (configuration).
The switches can be configured through in-band configuration by the default host, the I2C interface or an optional eeprom download. Some common configurations are also supported through strapping options.
In order to take advantage of the advanced features of the MR switch, the user must denote one of the host or I2C ports as the manager of the switch. Although the configuration of the switch portions is done statically at boot time, the user has the ability to change the partitions of the switch or move ports from one partition to another under management control.
However, caution must be exercised when the port associations are being changed, as some request or data may still be in transit. Once a port has been moved from one partition to the other, the new host must enumerate the device into its domain.
