Using PCIe & intelligent DMA to achieve blazing data rates in real-time recording instruments
Chris Tojeira of Pentek describes how the use of PCIe, intelligent DMA and a real time front end as well as an efficient API messaging structure resulted in the design of a real-time data recorder capable of sustained data to disk rates of 2 GB/sec and higher.
In today’s world of high-speed A/D converters, operating in the Gigahertz range and beyond, real-time signal recording has become a challenging feat that requires specialized hardware and intelligent application software.
To build a real-time recorder capable of streaming sustained data to disk at rates of 2 GB/sec and higher, the system developer must be able to overcome the limitations presented by the recorder’s I/O interfaces, memory, operating and file system as well as consider the optimal use of disk drive technology, RAID controller technology and user-friendly control tools for data capture and analysis.
Meeting real-time requirements
For a recording system to be considered “real-time” it is required that the recorder captures every sample of data that the front-end provides with absolutely no loss. This must happen consistently to provide confidence that the system will run in the most mission critical situations. It is also important to consider the ongoing requirement to move data faster.
With this in mind, the commercial PC server is an excellent platform choice as microprocessor technology is constantly advancing in both processing speed and architecture: memory interfaces are capable of streaming data at rates of 10 GB/sec and higher; serial fabric PCI Express Gen 2 x16 is capable of maximum data rates of 8 GB/sec; and SATA II provides disk storage rates of 3 GB/sec to a single drive.
The commercial PC motherboard’s roadmap includes transitions to PCI Express Gen 3, capable of doubling the current data rates. SATA III is beginning to hit the commercial PC market, providing storage rates of 6 GB/sec to a single drive.
The latest Intel processors are approaching 4 GHz rates clock rates with hex and octal cores, while memory continues to get smaller, cheaper and faster. This provides engineers with a solid foundation to build upon; one that can grow as new high-speed front-ends are introduced.
Keeping Up with Real-Time Streams
Once the foundation of the recorder is established, it is essential to provide a front-end I/O device that is capable of streaming data in real-time. This front-end device typically consists of one or more high-speed A/D converters or digital interfaces that acquire data at a constant rate, and a set of DMA engines that move data off of the device.
In a real-time system, these DMA engines are the most critical feature of the I/O device, as the design that is implemented on the internal hardware will dictate how well the instrument can stream data and maintain its real-time performance requirement.
The latest high-performance I/O devices should provide Intelligent DMA chaining that give the system developer the ability to optimize data flow and maximize performance.
The Intelligent DMA engine permits the user to chain many large buffers of data in a link list. Buffering data in such a way will allow the system to absorb any momentary latency hits, which can be caused by a number of external factors, assuring the instrument meets its real-time requirements.
While the DMA engines are responsible for moving data off of the device, it is the PCI Express (PCIe) engine, inherent within the device, that supplies the path to the server PC’s system memory.
Figure 1 – Real-Time Recording Instrument System Level Block Diagram
It is essential that the I/O device supply a sufficiently fast enough PCIe interface, to provide engineers with the bandwidth to maintain the data rates required by the front-end. Buffered data must be sent to disk at data rates that can match those performed by the front end I/O device. Figure 1 above shows the block diagram for one typical real-time recording Instrument.