I/O synchronization strategies for complex embedded designs - Embedded.com

I/O synchronization strategies for complex embedded designs

Measurement and automation systems involving multiple devices often require accurate timing in order to facilitate event synchronization and data correlation. For example, an industrial automation application may need to synchronize distributed motion controllers, or a test and measurement application may need to correlate data acquired from sensors distributed across a device under test.

To achieve this synchronization, devices in the system must either have direct access to timing signals from a common source, or the devices must synchronize their individual clocks to a common timebase. There are advantages and disadvantages to both methods of device synchronization.

In systems where the devices are located nearby each other, typically a few meters, sharing a common timing signal is generally the easiest and most accurate method of synchronization. However, a system in one physical location may be composed of several different types of processing units. Systems can be composed of PCs, running a modern operating system like Windows or Linux; embedded devices, sometimes running a Real-Time operating system such as VxWorks; and FPGAs. Synchronizing these logically distributed systems can be complicated.

This is further complicated when systems are physically distributed over a large area. In these situations, distributed clock synchronization becomes necessary. Using this approach, devices act on timing signals originating from a local clock which is synchronized to the other clocks distributed throughout the system. Examples of distributed clock synchronization include devices synchronized to a GPS satellite, a PC’s internal clock synchronized to an NTP time server, or a group of networked devices synchronized using the IEEE 1588 protocol. Instead of sharing timing signals directly, these devices periodically exchange information and adjust their local timing sources to match each other.

Another complication to synchronization is the type of ADC that is used. The most commonly used types of ADC’s for measurement and control applications are SAR (Successive Approximation Register) and sigma-delta (also known as a delta-sigma). These two types of ADC’s are relatively easy to synchronize if all of the ADC’s in the system are of the same type. Synchronization gets challenging when both types are used due to the different ways that they sample data.

Synchronization and Measurement Accuracy
When synchronizing a system, it is important to consider the level of synchronization that is required. For some applications, having correlated data is important to have a snapshot of your entire system. If the parameters you are measuring are changing rapidly, synchronization within a few milliseconds or even a few nanoseconds may be required. It is also important to know if synchronization to an absolute time is important, or just a relative time from an event. Jitter, any undesired deviation from a truly periodic event, affects general synchronization, but may also have an impact on the accuracy of your measurements.

Clock jitter and accuracy. When a system is comprised of multiple clock domains, jitter is introduced whenever a signal crosses between them. For example, say there is a system that pulses convert on a SAR ADC. The system has a trigger input that comes from an external timing source. This could be a pulse generator or a done signal from a sigma-delta ADC. The system that generates the CONVERT pulse to the SAR ADC must sample the trigger on one of its clock edges. This can introduce up to one system clock cycle of jitter into the conversion pulses.

Figure 1 shows a system where the trigger coming into the system must be re-synchronized onto the system clock. There is an uncertainty of one system clock period for when the trigger is seen by the system. This will cause one clock period of jitter on the conversion pulse.

Figure 1. Conversion Jitter

This clock jitter will impact the measurement accuracy. As seen in Figure 2 below, the error introduced by the clock jitter is related to the input signal’s frequency. High frequency signals have a greater voltage difference (dv) for the same amount of clock jitter (dt).

Figure 2. Voltage Difference Dependent on Frequency

The equivalent number of bits (ENOB) is a metric of how much resolution a measurement system can maintain. The more clock jitter the system has, the lower the ENOB will be. Figure 3 shows how ENOB is related to the input signal frequency and the amount of jitter in the sampling clock. Each line represents a different number of nanoseconds of jitter on the signal. As the signal frequency increases, the ENOB decreases for a given amount of jitter.

Figure 3. Equivalent Number of Bits

Heterogeneous I/O Synchronization
Embedded systems used in control and measurement applications typically have to measure a variety of signal types. Depending on the measurement range, bandwidth and signal conditioning requirements, different sensors may be measured by different types of hardware. The methods used to synchronize these different types of hardware vary by the type of ADC used – successive approximation register-based (SAR ADC),or sigma-delta based (Sigma-Delta ADC) – and the system architecture..

A successive-approximation ADC uses a comparator to successively narrow a range that contains the input voltage. At each successive step, the converter compares the input voltage to the output of an internal digital to analog converter which might represent the midpoint of a selected voltage range. At each step in this process, the approximation is stored in a successive approximation register (SAR). This type of ADC has a CONVERT pin that is used to tell the ADC when to take a sample. Each pulse on the CONVERT pin causes the ADC to take one sample. Synchronizing multiple SAR ADC’s can be achieved by providing the same CONVERT signal to each ADC.

A sigma-delta ADC oversamples the desired signal by a large factor and filters the desired signal band. Generally, a smaller number of bits than required are converted using a Flash ADC after the filter. The resulting signal, along with the error generated by the discrete levels of the Flash, is fed back and subtracted from the input to the filter. This negative feedback has the effect of noise shaping the error due to the Flash so that it does not appear in the desired signal frequencies.

A digital filter (decimation filter) follows the ADC which reduces the sampling rate, filters off unwanted noise signal and increases the resolution of the output (sigma-delta modulation, also called delta-sigma modulation). This type of ADC uses a free-running oversample clock to acquire its data. The ADC returns data at a rate determined by the decimation filter that is synchronous to the oversample clock. Synchronizing multiple sigma-delta ADC’s can be achieved by providing the same oversample clock to each ADC.

Synchronizing Multiple SAR ADCs
How well you can synchronize different ADC’s will depend greatly on the level of control you have over the conversion and data transfer (Figure 4 ). Let’s take a system that is attempting to synchronize two different pieces of hardware that both contain SAR ADC’s. One piece of hardware has some setup that must be performed before the conversion.

This can be setting input multiplexers, changing gain settings, input settling time or some other operation that the hardware needs before it is ready for the conversion. The other piece of hardware is always ready for a conversion. For both of the ADC’s, there is some time spent transferring the data back to the control unit after the conversion. The conversion process is started by a software command that triggers both pieces of hardware simultaneously.

Figure 4. Conversion Timing Delays

You can see in Figure 4 that if software starts the acquisition on both pieces of hardware at the same time, the data may not be acquired at the exact same point in time. Depending on the application, this phase offset between the two ADC’s may be acceptable. If this application needed precise synchronization of the two ADC’s, the developer would need lower level control of the hardware signals (i.e. convert lines).

Synchronizing SAR and Sigma-Delta ADCs
Mixing SAR and sigma-delta ADCs presents other complications. The sigma-delta ADC is synchronous to the oversample clock. It returns data regularly after a number of oversample clock periods that is set by the decimation filter in the ADC. The SAR ADC converts data based on the convert pulse which is synchronous to the system clock.

If the oversample and system clocks are not synchronized, they will drift apart over time and introduce delays between when the signals are sampled by the ADCs. Figure 5 shows a system with a sigma-delta ADC returning data synchronous to its oversample clock. The SAR ADC is synchronous to the system clock. As the two clocks drift apart, the conversion of the SAR ADC moves away from the sigma-delta ADC conversions.

Figure 5. Clock Drift Error

One method of overcoming the drift is to trigger the SAR conversion on the data ready signal of the sigma-delta ADC. This will synchronize the two measurements and prevent them from drifting apart over time. Depending on how this is done, jitter may be introduced into the SAR conversions. If the system (running off of the system clock) samples the data ready signal and drives the convert pulse, there will be jitter on the convert pulse on the order of one system clock period. This is because the data ready signal is coming from the oversample clock domain.

A better way to synchronize the sigma-delta and SAR ADC is to use the same clock for both ADCs. This way, the timing engine that drives the SAR conversion will be synchronous to the oversample clock going to the sigma-delta ADC. However, this solution has several downsides. The clock speed needed for the sigma-delta is often much slower than what is needed for the communication and timing engine for the SAR ADC. To get around this problem, you may be able to use an FPGA to phase align the two clocks, but this is a complex and difficult design that is only needed if you require precise synchronization.

Analog sampling delays. A system that is perfectly synchronized in the digital domain is still subject to synchronization errors in the analog domain. Many times a system needs to sample different types of signals. Some ADC’s may be connected to IEPE accelerometers while others are connected to strain gauges. Measurement hardware typically has some analog conditioning circuit that is specific to the type of sensor being used. These circuits can have different analog delays. If the two ADC’s sample the data at the exact same time, they may be measuring signals that occurred at two different points in time due to these different analog delays.Logically Distributed System Synchronization
For the purposeof this paper, we will define a logically distributed systems to be asystem that has all of its components in the same physically location.They are considered distributed because they have multiple processingunits that must be synchronized. These systems may have their processingunits connected by standard busses like PCI, PCIe or USB. They may alsobe connected with dedicated timing and synchronization signals.

Standard bus synchronization. Somestandard buses like PCIe do not have any built in synchronizationschemes. However, you can still synchronize devices on these buses usingadjustable timers and synchronization points similar to what 1588 doesover Ethernet. Unfortunately, many communication buses have difficult topredict jitter caused by other devices and the unpredictable nature ofoperating systems like Windows.

The device or software processthat keeps the master time needs to broadcast the current time and settimers on each device that should expire when the next synchronizationpoint is reached. At the time of the next synchronization point, themaster time keeper can then ask all the devices how early or late eachdevice is. Using this information, the master time keeper can increaseor decrease the timers on each device. Overshoot and undershoot can beminimized by using a PID algorithm to find the values for each timer.The entire system must treat the process of checking and adjusting thetimers as a high priority task.

Synchronization signals. Onetypical synchronization topology consists of a master device thatcontrols the synchronization of several slave devices through cabledsignals. Because the clock and trigger signals are physically connectedwith cables, they can achieve higher precision and are less vulnerableto clock signal problems, such as clock skew and clock drift, than thetime-referenced systems discussed later. The master node creates a clockand then shares that clock with all other nodes by means of cabling.

Whenthe action, such as data acquisition, is started on the master node, atrigger signal is shared with the other nodes to begin the action.Variables like trace length can affect how tightly synchronized thesignals actually are. A tightly configured system will use timers andadvanced FPGAs to slightly move the edge of each individual trigger lineto help achieve synchronization within a few hundred picoseconds. Figure 6   shows an example of a physically connected system, along with the clock and trigger lines.

Figure 6. Physically Connected Synchronization System

Physically Distributed System Synchronization
Aphysically distributed system has nodes that perform I/O in multiplephysical locations. Examples of this include everything from temperaturesensors spread across a factory to health monitoring sensors onturbines spread across a wind farm to voltage and current sensors on apower grid spread across a continent. These physically distributedcannot share a common clock to synchronize their I/O. Instead, they mustuse other methods of synchronization.

Time referenced systems. The blue portion of the precision versus distance graph shows theprecision achievable by time-referenced synchronization. In atime-referenced system, protocols such as GPS are used to convey timeinformation across greater distances than are prqactical with cabling.This time information is used by each system node to determine thepresent time and create a clock based on that reference. You can use afuture time event, which is when an action starts after a defined timeis reached, to trigger an action across all nodes simultaneously. Figure7 shows a time-referenced synchronization system.

Figure 7. Time-Referenced Synchronization System

Anothermethod of visualizing this transfer of time data via time protocols isthe clock tower/wristwatch analogy (Figure 8 below). Think of the masternode as being the clock tower and the slave nodes as being the peoplein a small town. Everyone in the town must be at work at 8:00 a.m. andthey all have their own wristwatches. But each person’s wristwatch couldbe different, and there might be confusion over the correct currenttime. If each person looks at the clock tower and resets, orsynchronizes, his or her wristwatch to the clock tower time reference,then everyone has the right time and arrives at work on time. The sameconveyance of time information takes place in time-referenced systems.

Figure 8. Time Reference and Time Source

Time Protocols
Time protocols are the tools you use to transport time information across large distances.

Pulse per second (PPS)  is a simple form of synchronization that outputs a pulse once a second.The rising edges of two consecutive pulses should be exactly one secondapart. The signal does not contain information about the specific timeof day or year. The pulse width is generally 100 ms, but many receiversallow the user to specify the pulse width, as long as it is less thanone second. Figure 9 shows a PPS signal.

Figure 9. Pulse per Second

IRIG-B – This older protocol was developed in the 1950s and is used totransmit time data. The signal is similar to PPS, but instead of asingle pulse once a second, IRIG-B sends coded bits that make up a dataframe that is one second long. This data frame presents time informationin seconds, minutes, and days and provides a status byte. IRIG-B has asynchronization precision of tens of nanoseconds.

Figure 10 shows how time is sent using IRIG-B. The entire frame is only onesecond long. It is based on pulse-width modulation (PWM), where a 25percent duty cycle represents a 0, a 50 percent duty cycle is a 1, and a75 percent duty cycle is a Pause (P) to separate the pulses forseconds, minutes, days, and the status. Two pause cycles (R in thisdiagram) signify the end of a timestamp. Figure 6 shows an IRIG-Bsignal, which is read from right to left.

Figure 10. IRIG-B

IEEE 1588 – IEEE 1588 is a packet-based protocol that you can use over Ethernet(Figure 11, below). It defines a standard set of clock characteristicsand value ranges for each characteristic. By running a distributedalgorithm, called the best master clock (BMC) algorithm, each clock inthe network identifies the highest-quality clock; that is the clock withthe best set of characteristics.

The highest-ranking clock,called the “grandmaster” clock, synchronizes all other “slave” clocks.If the grandmaster clock is removed from the network, or if itscharacteristics change in such a way that it is no longer the “best”clock, the BMC algorithm helps participating clocks automaticallydetermine the current best clock, which becomes the new grandmaster.This algorithm offers a fault-tolerant and administrative- free way ofdetermining the clock used as the time source for the entire network.

Thegrandmaster clock periodically issues a “sync” packet containing atimestamp of the time when the packet left the grandmaster clock. Thegrandmaster may also issue a follow-up packet containing the timestampfor the sync packet. The use of a separate follow-up packet allows thegrandmaster to accurately timestamp the sync packet on networks wherethe departure time of a packet cannot be known accurately beforehand.

Figure 11. IEEE 1588

Global positioning system (GPS) – GPS is a radio frequency encoded signal that provides time, position,and velocity information by means of a network of triangulationsatellites. This time-reference signal, which is globally available,delivers synchronization precision between tens and hundredsof nanoseconds. To use this signal, you need a GPS antenna with a clearview of the sky. Many system designers will use a simple PPS signal fromtheir GPS radio to synchronize their distributed system.

Isochronous Data Transfer
Whileisochronous data transfer is typically associated with USB, the conceptof scheduling periodic, low latency data transfers can be applied tomost communication buses. Isochronous data transfers are an importantpart of USB, EtherCAT, and TSN (formerly AVB Gen 2).

Since alldata transfers are scheduled on an isochronous bus, you can achieve thelowest latency data transfer possible for the physical network. In orderto properly meet this schedule, everything from the processing unit tothe IO must be synchronized. If any part of the system misses theschedule, data is either late or missed entirely. This is why manyisochronous standards, like TSN, include time synchronization as part ofthe protocol.

Distance between System Nodes
Twoimportant variables in designing a timing and synchronization scheme aresynchronization precision and the distance between the system nodes.System designers must account for the limitations created by thesevariables because as transmission distance increases, it is moredifficult to share signals between systems to keep them synchronized.

Thistrade-off between precision and distance presents a problem: to have ahigh precision of synchronization, you must have a clock with highfrequency and accuracy, which can degrade as the distance betweenchassis, or nodes, increases. In most systems, you know the distancesyou must design for. You may have a single node, a group of nodes in onelocation, or multiple nodes that are spread out over a greaterdistance. Based on this, you must decide if you can successfullytransmit the clock and trigger signals over this distance without toomuch degradation. If you cannot, then you must use a time reference torelay the clock domain information. Figure 12 shows the precision versus distance graph for physically connected and time-referenced synchronization systems.

Figure 12. Precision versus Distance Graph

Theprecision versus distance graph shows that as the distance betweennodes increases past a certain point, it no longer makes sense to usephysical signals for synchronizing nodes in your system. You needanother method of conveying the synchronization signals from the masternode to the other slave nodes in the system such as a timesynchronization protocol.

1. SynchronizationBasics. (2012, May 9). Retrieved March 1, 2013, from National Instruments
2. Introduction to Distributed Clock Synchronization and the IEEE 1588 Precision Time Protocol. (2012, May 21). Retrieved March 1, 2013, from National Instruments
3. Timing and Synchronization Systems. (2009, July 23). Retrieved March 1, 2013, from National Instruments

Salvador Santolucito is a Senior Hardware Engineer in the C Series modules group at National Instruments. Over the past seven years, Salvador has focused ondeveloping support for C Series modules in LabVIEW FPGA andunderstanding the challenges related to synchronizing modules withdifferent hardware architectures. He holds a bachelor's degree inElectrical & Computer Engineering from the University of Texas atAustin. This paper was/part of a class(ESC-327) presented for theEmbedded Systems Conference at Design West.

Brian Ruhmann is a Senior Engineer at National Instruments. He has spent the past decade focusing on embedded system design and FPGA technologies. While working at National Instruments, Brian has been a developer and technical lead on several products including LabVIEW FPGA and the CompactRIO product line.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.