How DMA adds value in a PCIe switch system
Figure 1 below shows a PCIe switch with DMA in control planes. Unlike data planes, control planes are latency-sensitive rather than bandwidth-intensive. Since latency is of the utmost importance, control planes prefer components that help reduce latency as much as possible.

The particular PCIe switch shown in Figure 1 has 12 x1 downstream ports and one x4 upstream port. These downstream ports are connected to a large number of ASICs and FPGAs. The DMA engine is used to gather statistics from the ASICs and FPGAs and update them with the latest configuration details from the CPU.

Data in the control plane needs to be written as fast as possible, since configuration of devices is very important; as they need to be instructed on the action, they also need to be taking over the data passing through them.

A DMA engine built into a PCIe switch helps achieve this objective; without it, the control processor has to update each and every end-point, one after the other, which delays the updating process and negatively impacts the performance of the system.

The usefulness of DMA in a PCIe switch is more visible when the upstream port of the PCIe switch is x1 wide and there are 15 downstream ports connecting to FPGAs and ASICs.

Figure 1. PCIe Gen 2 switch with DMA function in a control plane

Figure 2 below shows a PCIe switch with DMA in an intelligent PCIe adapter card. The DMA engine transfers data from the intelligent I/O adapter cards while the non-transparency (NT) function in the same switch provides host-isolation.

If a PCIe switch without DMA was used in this application, the CPU would have to take over this function, the result of which is the performance of the system decreases as the CPU tries to manage the data transfers between the intelligent I/O adapter cards rather than focusing on the computational tasks of the entire system. Additionally, having both NT and DMA in the same switch provides designers with the advantage of using a single-chip solution for two very important functions in the adapter card.

Figure 2. PCIe Gen 2 switch with DMA function in an intelligent PCIe adapter card.

Figure 3 below shows a PCIe switch with DMA in a PCIe cluster. The DMA engine transfers data between servers, while the NT function in the same switch provides isolation between each server. Again, in this example, if a PCIe switch without DMA was used, the CPU on each server would have to take over this function for its own card and thus resulting in a performance drop as the CPU is trying to manage the data transfers between the servers rather than focusing on the computational tasks of the system.

Figure 3. PCIe Gen 2 switch with DMA function in a PCIe cluster

Conclusion
The new generation of PCIe Gen 2 switches with built-in engines that have emerged can support four channels of DMA each and transfer small and large blocks of data between all switch ports. Each DMA channel can saturate a x8 link at Gen 2 in one direction.

The DMA engine in these devices implements a descriptor ring approach. Each descriptor provides support for large transfer sizes (up to 128MB), giving the user the ability to perform very large data transfers in any direction (memory to device, device to device, memory to memory).

Descriptors can exist in host memory or, alternatively, inside the DMA switch. Up to 256 descriptors are supported internally in these PCIe switches, which also support 32-bit and 64-bit transfers as well as programmable quality of service (QoS).

These devices support multiple configurations and can be used in various applications, such as control planes in the networking and communications market, intelligent I/O adapter cards in the embedded market and in PCIe clusters.

DMA engines in a PCIe switch provide more options for designers when architecting a system. This enables embedded-systems designers to lower their costs, while at the same time increase performance and differentiate their products.

Krishna Mallampati is senior product marketing manager at PLX Technology, Sunnyvale, Calif. Previously, he worked as an applications engineer and product marketing manager at Lucent Technologies and Agere Systems, and was a senior product marketing engineer at Altera Corp. Mallampati holds a master's of science degree from Florida International University and a master's of business administration from Lehigh University.