Does PCI-and its industrial derivatives like CompactPCI-feature a multiprocessing capability, or not? Arguments abound that CompactPCI is not suitable for asymmetrical multiprocessing, a type of multiprocessing where each of a PCI-based system's multiple processors has its own operating system. Closer scrutiny reveals a different reality, proving that CompactPCI can address the multiprocessing real-time environment.

Originally, the PCI (Peripheral Component Interconnect) bus was developed as a high-speed I/O bus for PC systems. In recent years, the PCI bus has also found its way into the embedded market in various form factors. The most popular representative is PCI's electrical and logical equivalent, CompactPCI, with its proven Eurocard format.

The increasing demands for asymmetrical multiprocessing capabilities have triggered new solutions that address the restrictions previously imposed on this type of multiprocessing by PCI/CompactPCI architectures. Asymmetrical multiprocessing ties together several PCI-based boards via the PCI/CompactPCI bus. Each board is "intelligent" in that it has its own processor and copy of the operating system. Figure 1 shows this basic configuration, with the user interface and the network part of the system implemented on the host CPU, with additional real-time processing power provided by I/O modules.

Figure 1: Example of a PCI/CompactPCI System With Asymmetrical Multiprocessing

Central System Controller Functionality in PCI/CompactPCI Systems

A PCI/CompactPCI system requires a central system controller (host) with the following multiprocessing properties:

• Central clock signal generation for the PCI bus
• Configuration management
• Arbitration (multimaster system)
• Central interrupt handling.

Let's look at the fundamental aspects of PCI to form the basis for further multiprocessing considerations.

Central Clock Signal Generation
The PCI bus is a synchronous bus that requires the system controller to generate a central clock signal for the entire system.

Configuration Management
Both PCI and CompactPCI busses are based on a hardware and software specification that defines the auto-configuration cycles which in turn enable Plug 'n Play capabilities. During the auto-configuration cycle that occurs after power-up, the central system controller uses special bus transfers to detect which PCI devices are connected to its local PCI bus, as well as which PCI devices may be connected on other PCI buses behind a PCI-to-PCI bridge (PPB). The central system controller first configures and assigns address ranges for its local PCI devices, then repeats the process for the PPB and for any PCI devices found on PCI backplanes behind the PPB.

Arbitration
The PCI bus is also a multimaster bus, allowing several masters to be connected on one PCI bus competing for possession of the bus via a request/grant mechanism. Possession of the PCI bus is allocated by a central arbitrator usually located in the processor-PCI bridge. One master on the PCI bus may, for instance, be a SCSI controller that collects data, then requests for possession of the bus, and finally transfers its data to main memory by means of DMA transfer.

Central Interrupt Controller
The PCI bus has four interrupt lines that can be used by several devices. The central interrupt controller is located directly on the local PCI bus of the system controller board. This is adequate for a PC system, but processing interrupts in a CompactPCI system are more complicated because the central controller must also process interrupts generated by other CompactPCI boards on the backplane, and not just its local PCI bus interrupts. A standard CompactPCI system has eight slots and if more than four interrupt-capable boards are installed, some of the four available interrupt lines must be shared by more than one interrupt source.

Problems for Multiprocessing

Problems with multiprocessing can occur with CompactPCI systems when the local I/O bus of the system controller board and the intelligent I/O boards are PCI-based.

There are three different methods that can be used for connecting various PCI buses:

• Direct connection of two PCI buses without a bridge
• Connection via standard (transparent) PPB
• Embedded PCI-to-PCI bridges.

Figure 2: Generic CompactPCI Architecture

Direct Connection of Two PCI Buses Without a Bridge

The first and most obvious method for connecting various PCI buses is to use a direct and parallel connection between the local PCI bus and the bus designated as the PCI backplane. One disadvantage of using this method is that it restricts the number of loads that can be driven by a PCI module.

In a PCI system, the signals are driven directly from the IC without any intermediate buffer, and are subject to certain timing restrictions on the high-speed PCI bus. This simple hardware restriction limits the number of loads in any PCI system to 10. From the view of the PCI bus, a PCI device on the motherboard counts as one load and each additional slot counts as two more loads. For a standard PC this means a maximum of four expansion slots.

Additional disadvantages to the direct PCI method become apparent when multiprocessing is required and the slots are populated with intelligent I/O rather than just "dumb" I/O modules. These disadvantages are identical to the problems with the transparent bridge connection method discussed in the next section.

Connection Via Standard (Transparent) PPB

In 1994, the PCI-to-PCI bridge Architecture Specification was defined by the PCI Special Interest Group (PCISIG). The specification defined a transparent or standard PCI-to-PCI bridge (PPB) that avoids the load restrictions of the direct connect method.

After normal identification during the auto-configuration cycle, the PPB becomes transparent to the host processor. From the multiprocessing point of view, this transparent bridge behaves in exactly the same way as the direct connection of two PCI buses (review Direct Connection of Two PCI Buses Without a Bridge).

This type of PPB does not contain any resources such as DMA or mailboxes that would require a separate device driver. Nor does it convert any addresses from one PCI bus to another (flat addressing). Interrupt lines are simply routed around the transparent PPB.

Although transparent PPBs solve the load restriction problem, other challenges exist:

• No Plug 'n Play Capability
  The auto-configuration cycles from the two different processors will collide. After power-up, each processor will first configure its local PCI I/O subsystem. It will then configure its transparent PPB, and finally attempt to identify and set registers and address ranges of the attached devices found behind the PPBs on the PCI backplane. The first problem is that transparent PPBs can only be configured from the primary (local) PCI side via a special configuration cycle coming from their local processor. PCI to PCI bridges cannot be selected by configuration cycles coming from their secondary or backplane interface, making it impossible for the host board to identify the intelligent I/O board and vice versa.
• No Communication Feature (e.g. Mailboxes) Between the Two Processors
  A transparent bridge does not have a device driver that could manage such a resource.

Alternative Solution Involving Embedded PCI-to-PCI Bridges

Fortunately, another type of PCI-to-PCI bridge has been developed in order to avoid communication and Plug 'n Play problems. Alternative PPBs are called embedded, or non-transparent bridges, and have been designed specifically for intelligent I/O boards and thus for asynchronous multi-processing.

Plug 'n Play capabilities are now possible without collision of the configuration cycles. The embedded bridge separates the areas that can be scanned and configured by the local and host processors. After power-up, the intelligent I/O processor scans (auto-configures) its own I/O subsystem found on the primary or local side of the embedded bridge. It then configures the embedded bridge, installs address windows, defines address translations and maps the address registers for its local PCI devices into the embedded bridges' primary (local) and secondary (PCI backplane bus) sides. The local processors' configuration range ends at the embedded bridge and it will not scan past the bridge to find devices on the PCI backplane bus.

Figure 3: Asymmetrical Multiprocessing with Embedded (Non-Transparent) PPB on the Intelligent I/O Board

During the same power-up sequence, the processor on the host board is configuring its PCI I/O subsystem (PCI1 in Figure 3) and its transparent PPB. It is also scanning for PCI devices found on the PCI backplane bus. When the host processor finds the embedded bridge on the intelligent I/O module, it only sees the secondary side of the bridge (which looks like a normal PCI target device) and configures it accordingly. The underlying PCI bus of the intelligent I/O module (PCI2 in Figure) is not visible to the host processor.

Addressing conflicts are avoided by means of address conversion. For instance, the processor of the I/O board is able to hide the resources it requires from the host processor (hidden resources). In addition, address conversion enables a device on the local PCI bus of the intelligent I/O board to be allocated a PCI address that already exists on the backplane without causing any address conflicts.

The host processor drives the clock signal via its transparent PPB to the PCI backplane bus and to the attached PCI devices. The embedded bridge on the intelligent I/O board receives the host clock signal on its secondary (PCI backplane bus) side, but does not forward the host clock signal to its primary side (PCI2). Instead, the primary side of the non-transparent bridge runs with the clock signal from its own local processor. Both clock signals are still synchronous for their PCI bus, but asynchronous with respect to each other.

Functions on the non-transparent PPB (e.g. mailboxes and I2O FIFOs) can be used for communication between the host processor and the intelligent I/O processor.

PCI2 in Figure 3 illustrates how a hardware solution for asymmetrical multiprocessing in PCI and CompactPCI systems is possible by using transparent PPBs on the host board and non-transparent PPBs on intelligent I/O boards. Transparent PPBs could also be used on dumb I/O boards.

Embedded Bridges Bring Asymmetrical Multiprocessing to CompactPCI

Asymmetrical multiprocessing systems have been implemented on CompactPCI architectures by combining transparent and embedded bridge technologies.

The key multiprocessing requirements—clocking, Plug 'n Play configuration management, arbitration and interrupt handling—have all been addressed by the embedded bridge solution. Embedded bridges also offer additional features like mailboxes, I2O support and hidden resource capabilities.

Before deciding to implement a specific asymmetrical multiprocessing system, the application of such a system should be scrutinized with respect to the real-time conditions and also subsequent expandability.