For three decades, industrial PCs have used two types of internal architecture: motherboards and passive backplanes. A new PCI Express over cable concept will improve the capabilities of the next computers on smaller foot prints by ten times, according to the manufacturer.
Figure 1. Concept of myOPALE
By Frederic Aupetit, Ecrin Systems Download PDF version of this article
For three decades, industrial PCs have used two types of internal architecture, motherboards and passive backplanes: ATX motherboards with slots for ISA, PCI and PCIe I/O extensions connected perpendicularly to the motherboard, or smaller form factors like EBX/miniITX variants for 1U chassis for up to 2x I/O cards in parallel to the motherboard, and since 1994, PICMG 1.0 and 1.3 passive backplanes, where a single board computer and its I/O cards are connected vertically to offer more I/O sites and better MTTR for industrial and test bench systems.
The major evolution during the last 25 years has essentially concerned only the size of the housing, which has shrunk from standard 4U/2U 19’’ rackmount to Box PCs, so-called fanless IPCs. In all cases, the SBC and its I/O cards have remained physically and mechanically connected through Peripheral Component Interconnect, nowadays PCI Express, a local bus standard developed by Intel Corporation in 2004 and now promoted by PCI-SIG. The PCI Special Interest Group is the consortium responsible for specifying the PCI, PCI-X and PCIe computer buses since 1992. This PC backplane interface standard is not optimal in many cases. In 1U 19’’ chassis, only two I/O cards can be mounted, and mechanical fixing and cards exchange are complex in 1U and 2U chassis due to riser or butterfly backplane interconnects. Reliability of the contacts is not optimal and customization NRC will be necessary in most appliances. Furthermore, connectors of all I/O boards and SBCs are located at the same side of the chassis, usually at the rear panel. If specific connectors are needed (BNC, XLR, SMA, RJFTV, 38999…), all I/O cards and computer boards have to be moved, as an entire block, inside the housing, and chassis depth will therefore increase drastically. There is also no universal method to dissipate waste heat from hot spots, and in most of the cases, chassis customization will generate NRC and extra cost for recurrent. Customized appliances will usually require a complete mechanical and electronic redesign.
Figure 2. Configuration based on one myOPALE-CPU driving three myOPALE-IO, allowing to integrate up to five I/O cards in 1U/19’’ rackmount. Common variants: 2 systems in one, I/O connectors on front or rear panel.
Ecrin’s myOPALE concept is based on four major principles. It breaks the mechanical link between the computer mainboard and its I/O cards thanks to PCI Express over cable interconnection. It uses building blocks in a standard form factor. The re-use of universal deployed interconnect standards from SNIA (Storage Networking Industry Association) that encompass cables, connectors, form factor sizes and housing dimensions, transceiver interfaces and related technologies is possible. These specifications enable multi-source products and solutions. And finally, it includes the cooling system as a building block.
PCI Express was introduced by Intel in 2004 as the new Peripheral Component Interconnect serial bus. Today, PCIe is very popular as the PC backplane interface standard, but it is still less known as a high-speed cabling interface. The serial technology and embedded clocking within each differential signal pair allow PCIe protocol to be used at full speed across a motherboard or over a cable with the benefits of high performance, low power and reduced costs. The cabled version of PCIe features the same high-speed differential line pairs as one that transfers data on PCBs, as well as a number of additional signals – known as sideband signals. One PCIe lane consists of two pairs of cables individually shielded for EMI protection – one transmit-pair and one receive-pair. Sideband signals provide additional functionalities, but are not directly involved in the PCIe data transfer.
Figure 3. PCIe over cable miniSAS-HD connector allows to move back I/O cards in myOPALE-IO to provide custom connectors independently from myOPALE-CPU, on front or rear panel.
The myOPALE concept integrates most advantages of PCI Express over cable inside the rack. In this concept the PCIe backplane bus is conceptually stretched to operate over a cable from CPUs to secondary I/O backplanes, physically located anywhere in the chassis. Thus it offers many advantages in a small footprint 1U 19’’ rackmount, but also very interesting capabilities for other integration formats beyond the limits of the previous IPC generation.
The choice of the Mini-SAS HD (SFF-8643) connector for PCIe over cable was also electrically the best solution thanks to its multi-source and price/quality ratio. It is COTS ready and able to use cables from 30cm to 1m and lockable connectors to protect against shock and vibration in harsh environments. The high bandwidth of 12Gbps for PCI Express Gen3 protocol and 8 differential links for 4X lanes PCIe Gen3 enable high data throughput. Eight sideband signals for the control plane between myOPALE-CPU and myOPALE-IO building blocks are available. Compatibility with new NVM Express storage media and networks, availability of standard cable from Mini-SAS HD (SFF-8643) to U.2 (SFF-8639) is ensured. Availability of M.2 and PCIe 4X form factor boards to Mini-SAS HD will allow to integrate myOPALE-IO modules into standard PCs.
Figure 4. Blade with expansion slot demonstrates the capability to integrate 6- in-1 system in 4U/19’’ rackmount. Each blade can drive up to two I/O cards.
Mechanically Ecrin has adopted the SNIA SFF-8500 specification that defines the configuration characteristics associated with 5.25″ drives bay for the following reasons: ideal format for 1U/19’’ rack mount (41.3mm x 146.1mm), well adapted to full-height PCIe I/O cards, compliant form factor with custom COM Express carrier boards, easy integration into rack, desktop or tower chassis. The set of specifications includes external dimensions, connectors, connector placement, mounting holes and interface pinouts to assist manufacturers for system integration.
The CPU module incorporates all features of embedded computers into an SFF 5.25’’ site with 200mm depth only. These are: 6/7th Gen Intel Core, Intel Xeon E3-1500 processors, up to 32GB Dual Channel DDR4, with ECC support for Intel Xeon E3-1500 SKU, legacy I/O on front panel; DisplayPort (x2), GbE, USB 3.0 (x4), legacy I/O on rear panel; 2x SATA 6.0Gb/s ; 3x USB 2.0 (Micro-Fit 24-pin connector), internal bay for 2.5″ SSD/HDD, SATA 6.0Gb/s, 24x PCIe Gen3 lanes through six Mini-SAS HD connectors (4 PCIe lanes per Mini-SAS HD connector), and 12VDC power supply (+5Vstb for ATX mode).
The design is based on COM Express Type 6 that integrates embedded low power SKUs with 10 years life support. In Skylake and Kaby Lake versions, Intel Core i7 or Xeon E3-1500 CPUs with Chipset platform are selected, which directly drive 24 PCIe Gen3 lanes as their expansion bus. They can be grouped by 8/8/4/4 or 8/4/4/4/4 which allows to drive up to five I/O cards per processor. PCIe cable adapters on the carrier card is quite simple by routing to the connector, and provides some signal conditioning to guarantee the signal integrity is met at the other ends of the cable. Because these adapters are simple, they are inexpensive. Because they don’t convert the PCIe protocol into anything else, they provide higher performance and don’t require any software drivers (as do such others as thunderbolt, InfiniBand, firewire, USB, Fiber channel…). Throughput and latency, offered by PCIe over cable in myOPALE system, are the best of any other protocol or I/O adapter.
Power supply may be used in AT or ATX-mode. In AT-mode, 12V input is fine. +5VStb is generated internally to support Wake-on-LAN function and AMT Remote Power-on. In ATX-mode, +5VStb signal input is provided via a 6-pin power connector to myOPALE-CPU which monitors PS-ON to drive the power supply unit. The cooling system is based on an aluminum mono-block composed of two areas: one conduction-cooled cold plate that takes away heat from its CPU hot spot up to fins cooled by a forced air-flow system (2x 40mm fans in push-pull mode). Temperature and fan speed are monitored by the embedded CPU.
Figure 5. Just by integrating one COTS PCIe or M.2 adaptor to miniSAS-HD card, you can add myOPALE-IO on the front panel of your PC for audio, broadcast, communication extensions.
Sideband control signals selected from Mini-SAS HD cable allow myOPALE-CPU to manage and monitor each myOPALE-IO without extra cabling. Power and Reset buttons, HDD activity and Default LEDs are available on the front panel and simultaneously on a rear Micro-Fit 24-pin connector for remote control. The myOPALE-IO module can integrate up to two full-height I/O cards in a standard 5.25’’ drive bay with 200mm depth. Half-length I/O cards will be completely enclosed in the housing, while full-length cards can protrude from the back without any inconvenience. A retaining bracket system for card lock will protect low profile and full-height I/O modules against shocks and vibrations stress.
Two versions of myOPALE-IO exist with different passive backplanes: two PCIe Gen3 4X I/O sites and one PCIe Gen3 8X I/O site. For both versions, PCIe connectors are mechanically in PCIe 16x format to accept all types of I/O boards in PCI Express (x1, x4, x8 and x16) format. The myOPALE-IO backplane converts Mini-SAS HD connectors to standard PCIe direct linear connectors, ready to accept any type of COTS or Custom PCIe I/O cards. Ecrin Systems integrates also two other functions: a DC/DC converter from 12VDC to 3.3VDC and an automatic 2-fan control system from a Look Up table. Sideband control signals from Mini-SAS HD connector allow the control of the power supply, and monitor local temperature and fan speeds.
The ability to run PCIe over cable at full performance with software transparency opens up a range of new applications for CPU and I/O system re-positioning in a rack. myOPALE extends the PCIe bus structure to dedicated PCIe I/O hardware. PCIe over cable provides a simple method for extending applications that need more I/O boards than could be fitted in a standard Industrial PC based on backplanes. Designing compatible end-points is straightforward because the PCIe interface is available as a gate array library.
Having backplane performance available over a cable expands the myOPALE usage model to encompass many high-end mono- and multi-chassis applications including more I/O expansion with different system integration capabilities, disk array subsystems based on NVMe interface, high speed video and audio editing equipment, quick trading with many pairs of CPU/FPGA in one system, medical imaging systems, etc.