Does PCI-and its industrial derivativeslike CompactPCI-feature a multiprocessing capability, or not?Arguments abound that CompactPCI is not suitable for asymmetricalmultiprocessing, a type of multiprocessing where each of aPCI-based system's multiple processors has its own operatingsystem. Closer scrutiny reveals a different reality, proving thatCompactPCI can address the multiprocessing real-timeenvironment.
Originally, the PCI (Peripheral Component Interconnect) bus wasdeveloped as a high-speed I/O bus for PC systems. In recent years,the PCI bus has also found its way into the embedded market invarious form factors. The most popular representative is PCI'selectrical and logical equivalent, CompactPCI, with its provenEurocard format.
The increasing demands for asymmetrical multiprocessingcapabilities have triggered new solutions that address therestrictions previously imposed on this type of multiprocessing byPCI/CompactPCI architectures. Asymmetrical multiprocessing tiestogether several PCI-based boards via the PCI/CompactPCI bus. Eachboard is “intelligent” in that it has its own processor and copy ofthe operating system. Figure 1 shows this basic configuration, withthe user interface and the network part of the system implementedon the host CPU, with additional real-time processing powerprovided by I/O modules.
Figure 1: Example of a PCI/CompactPCI System WithAsymmetrical Multiprocessing
Central System Controller Functionality in PCI/CompactPCISystems
A PCI/CompactPCI system requires a central system controller(host) with the following multiprocessing properties:
- Central clock signal generation for the PCI bus
- Configuration management
- Arbitration (multimaster system)
- Central interrupt handling.
Let's look at the fundamental aspects of PCI to form the basisfor further multiprocessing considerations.
Central Clock Signal Generation
The PCI bus is a synchronous bus that requires the systemcontroller to generate a central clock signal for the entiresystem.
Both PCI and CompactPCI busses are based on a hardware and softwarespecification that defines the auto-configuration cycles which inturn enable Plug 'n Play capabilities. During theauto-configuration cycle that occurs after power-up, the centralsystem controller uses special bus transfers to detect which PCIdevices are connected to its local PCI bus, as well as which PCIdevices may be connected on other PCI buses behind a PCI-to-PCIbridge (PPB). The central system controller first configures andassigns address ranges for its local PCI devices, then repeats theprocess for the PPB and for any PCI devices found on PCI backplanesbehind the PPB.
The PCI bus is also a multimaster bus, allowing several masters tobe connected on one PCI bus competing for possession of the bus viaa request/grant mechanism. Possession of the PCI bus is allocatedby a central arbitrator usually located in the processor-PCIbridge. One master on the PCI bus may, for instance, be a SCSIcontroller that collects data, then requests for possession of thebus, and finally transfers its data to main memory by means of DMAtransfer.
Central Interrupt Controller
The PCI bus has four interrupt lines that can be used by severaldevices. The central interrupt controller is located directly onthe local PCI bus of the system controller board. This is adequatefor a PC system, but processing interrupts in a CompactPCI systemare more complicated because the central controller must alsoprocess interrupts generated by other CompactPCI boards on thebackplane, and not just its local PCI bus interrupts. A standardCompactPCI system has eight slots and if more than fourinterrupt-capable boards are installed, some of the four availableinterrupt lines must be shared by more than one interruptsource.
Problems for Multiprocessing
Problems with multiprocessing can occur with CompactPCI systemswhen the local I/O bus of the system controller board and theintelligent I/O boards are PCI-based.
There are three different methods that can be used forconnecting various PCI buses:
- Direct connection of two PCI buses without a bridge
- Connection via standard (transparent) PPB
- Embedded PCI-to-PCI bridges.
Figure 2: Generic CompactPCI Architecture
The first and most obvious method for connecting various PCIbuses is to use a direct and parallel connection between the localPCI bus and the bus designated as the PCI backplane. Onedisadvantage of using this method is that it restricts the numberof loads that can be driven by a PCI module.
In a PCI system, the signals are driven directly from the ICwithout any intermediate buffer, and are subject to certain timingrestrictions on the high-speed PCI bus. This simple hardwarerestriction limits the number of loads in any PCI system to 10.From the view of the PCI bus, a PCI device on the motherboardcounts as one load and each additional slot counts as two moreloads. For a standard PC this means a maximum of four expansionslots.
Additional disadvantages to the direct PCI method becomeapparent when multiprocessing is required and the slots arepopulated with intelligent I/O rather than just “dumb” I/O modules.These disadvantages are identical to the problems with thetransparent bridge connection method discussed in the nextsection.
Connection Via Standard (Transparent) PPB
In 1994, the PCI-to-PCI bridge Architecture Specification wasdefined by the PCI Special Interest Group (PCISIG). Thespecification defined a transparent or standard PCI-to-PCI bridge(PPB) that avoids the load restrictions of the direct connectmethod.
After normal identification during the auto-configuration cycle,the PPB becomes transparent to the host processor. From themultiprocessing point of view, this transparent bridge behaves inexactly the same way as the direct connection of two PCI buses(review Direct Connection of Two PCI BusesWithout a Bridge).
This type of PPB does not contain any resources such as DMA ormailboxes that would require a separate device driver. Nor does itconvert any addresses from one PCI bus to another (flataddressing). Interrupt lines are simply routed around thetransparent PPB.
Although transparent PPBs solve the load restriction problem,other challenges exist:
- No Plug 'n Play Capability
The auto-configuration cycles from the two different processorswill collide. After power-up, each processor will first configureits local PCI I/O subsystem. It will then configure its transparentPPB, and finally attempt to identify and set registers and addressranges of the attached devices found behind the PPBs on the PCIbackplane. The first problem is that transparent PPBs can only beconfigured from the primary (local) PCI side via a specialconfiguration cycle coming from their local processor. PCI to PCIbridges cannot be selected by configuration cycles coming fromtheir secondary or backplane interface, making it impossible forthe host board to identify the intelligent I/O board and viceversa.
- No Communication Feature (e.g. Mailboxes) Between the TwoProcessors
A transparent bridge does not have a device driver that couldmanage such a resource.
Alternative Solution Involving Embedded PCI-to-PCIBridges
Fortunately, another type of PCI-to-PCI bridge has beendeveloped in order to avoid communication and Plug 'n Playproblems. Alternative PPBs are called embedded, or non-transparentbridges, and have been designed specifically for intelligent I/Oboards and thus for asynchronous multi-processing.
Plug 'n Play capabilities are now possible without collision ofthe configuration cycles. The embedded bridge separates the areasthat can be scanned and configured by the local and hostprocessors. After power-up, the intelligent I/O processor scans(auto-configures) its own I/O subsystem found on the primary orlocal side of the embedded bridge. It then configures the embeddedbridge, installs address windows, defines address translations andmaps the address registers for its local PCI devices into theembedded bridges' primary (local) and secondary (PCI backplane bus)sides. The local processors' configuration range ends at theembedded bridge and it will not scan past the bridge to finddevices on the PCI backplane bus.
Figure 3: Asymmetrical Multiprocessing with Embedded(Non-Transparent) PPB on the Intelligent I/O Board
During the same power-up sequence, the processor on the hostboard is configuring its PCI I/O subsystem (PCI1 in Figure 3) and its transparent PPB. It is alsoscanning for PCI devices found on the PCI backplane bus. When thehost processor finds the embedded bridge on the intelligent I/Omodule, it only sees the secondary side of the bridge (which lookslike a normal PCI target device) and configures it accordingly. Theunderlying PCI bus of the intelligent I/O module (PCI2 in Figure) is not visible to the hostprocessor.
Addressing conflicts are avoided by means of address conversion.For instance, the processor of the I/O board is able to hide theresources it requires from the host processor (hidden resources).In addition, address conversion enables a device on the local PCIbus of the intelligent I/O board to be allocated a PCI address thatalready exists on the backplane without causing any addressconflicts.
The host processor drives the clock signal via its transparentPPB to the PCI backplane bus and to the attached PCI devices. Theembedded bridge on the intelligent I/O board receives the hostclock signal on its secondary (PCI backplane bus) side, but doesnot forward the host clock signal to its primary side (PCI2).Instead, the primary side of the non-transparent bridge runs withthe clock signal from its own local processor. Both clock signalsare still synchronous for their PCI bus, but asynchronous withrespect to each other.
Functions on the non-transparent PPB (e.g. mailboxes and I2OFIFOs) can be used for communication between the host processor andthe intelligent I/O processor.
PCI2 in Figure 3 illustrates how ahardware solution for asymmetrical multiprocessing in PCI andCompactPCI systems is possible by using transparent PPBs on thehost board and non-transparent PPBs on intelligent I/O boards.Transparent PPBs could also be used on dumb I/O boards.
Embedded Bridges Bring Asymmetrical Multiprocessing toCompactPCI
Asymmetrical multiprocessing systems have been implemented onCompactPCI architectures by combining transparent and embeddedbridge technologies.
The key multiprocessing requirementsclocking, Plug 'n Playconfiguration management, arbitration and interrupthandlinghave all been addressed by the embedded bridgesolution. Embedded bridges also offer additional features likemailboxes, I2O support and hidden resource capabilities.
Before deciding to implement a specific asymmetricalmultiprocessing system, the application of such a system should bescrutinized with respect to the real-time conditions and alsosubsequent expandability.