Clock generation and distribution is a significant challenge in the design of large and complex digital systems., Ordinarily, clock generation and distribution can be well controlled in small systems and subsystems. Synchronous signaling, where the data signal timing is related to a single timing reference, can be used for practically all critical high-speed signals (see Figure 1). In large systems, however, where designs are typically partitioned into two or more subsystems, clock distribution becomes increasingly complex. Each subsystem can have its own local clock generation and distribution tree. The intra-system interfaces require data transmitted synchronously from one subsystem, to be re-timed or synchronized to the local reference in the receiving subsystem. This article discusses the methods for retiming digital signals in data communications systems such as the emerging InfiniBand Architecture. But the re-timing issue is relevant to other serial communications architectures as well.
Synchronous clocking systems, such as the Synchronous Optical Network (SONET), use a single timing model. Originally developed for transmission of telecommunications signals such as voice, Sonet is now the prevalent transport infrastructure for the wide area network backbone. However, the majority of information now being transmitted over SONET networks is data, due to the widespread use of the Internet. The primary benefit of the synchronous transmission and multiplexing hierarchy defined by SONET is that multiple data streams at the defined rates can be combined (multiplexed) without bit stuffing into a higher rate stream and can be extracted without demultiplexing the entire higher rate stream.
The SONET network consists of point-to-point links providing connectivity to the various access nodes in the network with the capability to aggregate streams of data by either injecting into the system (ADD) or taking out (DROP). Network timing between these points is accomplished through a synchronous system of clocks that are very accurate, and only need adjusting periodically, with a hierarchy for distribution of this timing information throughout the network. The clock is not transmitted separately from the data, but is encoded into the data for extraction at each end, with circuits known as "clock and data recovery" (CDR) circuits.
Figure 1. A synchronous digital system uses a single clock reference
The most accurate clocks are called Stratum 1, and are accurate to better than 0.000001 ppm (parts per million). As the timing signals propagate, stratum 2 clocks are generated from stratum 1 clocks with slightly less accuracy, and so on for stratum 3 clocks. The timing is akin to synchronizing many people's accurate quartz watches to a master clock. The frequency differences between all the wristwatches to the master clock frequency are good enough where re-synchronization only needs to occur periodically, such as once a day. For stratum 1 clocks, they are accurate in frequency to better than 1 second in 3000 years.
Although the SONET method works well and has been in use for large backbone telecom networks, it is an expensive and complex system. For transmitting data over shorter distances, a lower cost system must be used to distribute network timing. A system for high speed serial data to be transported in data communications from one node to another node asynchronously is used, where each local system or node is based on reference frequencies that are close, but not exactly the same as the system reference frequency. This is known as a plesiochronous system, wherein each local clock has small frequency differences of typically 100-200 ppm.
Ethernet has been the local area network standard for more than a decade. A new standard is emerging for communications between servers, storage, and network adapters, the InfiniBand Architecture. The InfiniBand Architecture offers a scalable, switched, modular fabric for servers and storage with features such as QOS (quality of service) and high RAS (reliability, availability, security), scalability, and flexibility. Serial speeds are 2.5Gb/s and are scalable to 10Gb/s and 30Gb/s in the 4x and 12x configurations, respectively. InfiniBand uses local reference frequencies that are sub-multiples of 2.5GHz clock frequency (i.e. 125 MHz) that are stable and accurate to within +/-100ppm.
To accommodate the differences in data periods due to the +/- 100ppm frequency differences, plesiochronous systems employ a technique known as "bit stuffing", where special bits are either added or deleted to adjust the rate of an incoming data stream to the frequency of the system receiving the data stream. If bit stuffing were not employed, bits would be duplicated (incoming data rate too slow) or skipped (incoming data rate too fast) due to the differences in frequency between the transmitting node clock rate and the receiving node clock rate.
Each node in an InfiniBand plesiochronous system uses a local frequency reference. Each node is comprised of a synchronous digital subsystem based upon a clock rate Fc derived from a common local frequency reference Fref, where Fref1 and Fref2 are different by 200ppm or less. The specification allows "clock wander", which means that the local data rate can vary with time within +/- 100ppm from the 2.5Gb/s specification. In the example of Figure 2, the incoming data from node 1 is transmitted at Fc1. Node-2 receives this data and must perform rate compensation upon the data stream to enable it to be used at the node-2 clock rate of Fc2.
Figure 2. Plesiochronous data communication systems use multiple clock references
To accommodate the frequency rate compensation at each node, InfiniBand Architecture protocol-aware nodes employ a bit stuffing protocol wherein a group of bits are either inserted or deleted if the incoming frequency is either faster or slower, respectively, than the local system clock rate (as shown in Figure 3). As is typical with other data communications systems, InfiniBand employs valid windows where these bits, typically referred to as "skip symbols", can be inserted or deleted in a circuit block known as an "elastic buffer". The input side of the elastic buffer operates in the received clock domain and the output side of the elastic buffer operates in the local clock domain. The number of reserved bits in each window and the frequency of occurrence of these skip-symbol insert/delete windows determine the clock rate differences, or clock wander, that can be accommodated.
Figure 3. Skip-SKIP symbols for rate compensation in InfiniBand systems
Systems such as Ethernet, Fibre Channel, and InfiniBand employ these techniques for rate compensation, with InfiniBand allocating 5 spaces for skip symbols approximately every 4000 symbol clocks. One space is allocated for the frequency compensation at the receive node. The remaining four symbol spaces are half filled when transmitted, allowing up to two skip symbols to be added or two skip symbols to be deleted along the link by other devices, such as retiming-repeaters. Thus, only two additional devices containing elastic buffers can be inserted between nodes.
Transmission of Data Over Media
The InfiniBand Architecture specifies the transmission of data between nodes over channels on various media: printed circuit board, copper cable, or fiber optics. In typical links between nodes, the signals traverse over a combination PCB and copper cable or PCB and fiber optic cable. The distance the data can be communicated over the channel is dependent upon signal integrity, channel loss characteristics, noise and interference that affect the receivers ability to recover the clock and data at the required bit error rate (BER) of better than 1E^-12.
At the InfiniBand Architecture data rate of 2.5Gb/s, the minimum transmission distances specified are 24 inches over PCB, 17 meters over copper cable, or 250 meters over multi-mode fiber. In large chassis, much of the signal can become attenuated and noisy (increased clock jitter) and the data eye can become severely degraded as the signals traverse the printed circuit board and connectors prior to launching out of the chassis. (Figure 4a shows a photo of a 2.5Gb/s data eye as it is initially transmitted, and Figure 4b shows the same signal degraded in amplitude and with increased timing jitter after 35 inches of circuit board trace.)
Figure 4a. A 2.5Gb/s data eye as the signal is initially transmitted
Figure 4b. The same signal with a degraded data eye after transmission over 36 inches of PCB
Prior to transmitting the signal on either cable or fiber, it is desirable to improve the signal by amplifying and removing jitter, so that the data eye quality is improves to the original transmitted quality (like that in Figure 4a). This can be accomplished by devices known as retiming-repeaters.
Retiming Repeaters amplify the signal and open the data eye in amplitude, but do not improve the noise and jitter characteristics of the signal. Retimers use a clean clock to retime the signal to improve the jitter and increase the duration that the data eye can be validly clocked, i.e. opening the data eye in the horizontal direction. Retiming-repeaters both reset the jitter budget and increase the signal amplitude. A retiming-repeater can be employed either in the chassis or between chassis to extend the distance between nodes.
Table 1 compares four methods for retiming and repeating a signal: CDR, back-to-back SerDes, elastic buffer, and a specially-developed Skip-Free Retiming-repeater architecture. Key metrics compared are: the ability to reset the jitter budget, the number of times the retiming repeater can be cascaded between nodes, and the jitter transfer and jitter peaking characteristics of each method.
Table 1. The same signal with a degraded data eye after transmission over 36 inches of PCB
Method 1 is based on recovering the clock from the input data in a clock-and --data-recovery (CDR) loop, and retiming the data with this recovered clock. The phase-locked-loop (PLL) filter removes some jitter, but the residual jitter is transferred to the output. Thus, devices employing Method 1 cannot be continually cascaded without jitter peaking that can exceed the system jitter requirements.
Method 2 is based on using back-to-back serializer-deserializer (SerDes) functions. The serializer PLL performs additional filtering of the recovered clock, which results in better jitter transfer characteristics than the CDR method. The ability to cascade this function multiple times is still limited by jitter transfer and jitter peaking. The number of times this function can be performed in a link is dependent on the jitter characteristics of the incoming signal into the first repeater. Thus, it is not known for all situations how many times that this type of retiming-repeater can be used in a link.
Method 3 is based on using a local reference clock with low jitter characteristics to retime the signal, and is the typical method used to reset the jitter budget.
Figure 5 shows two of these retiming repeaters in the serial data path between 2 nodes. To fully reset the jitter budget, retimers should employ a "clean" local frequency reference such as Fref3 and Fref4. This type of retiming-repeater employs the same approach that the end nodes use to perform rate compensation by adding or deleting skip-symbols in an elastic buffer. However, because the number of bits reserved for skip symbol addition or deletion is fixed in InfiniBand architecture systems, this limits the number of times that typical retiming-repeater can be used in a data communication link between nodes. As Figure 5 shows, the data rate is changed each time the signal is retimed, and thus only a fixed, finite number of retiming functions (two in InfiniBand Architecture) can be employed before the frequency difference at the end node is greater than can be compensated for.
Figure 5. Typical retimers alter the data transmission frequency, insert skip symbols, and are limited in the number of times retiming can be performed in a data link
The retiming-repeater's function is to reset the jitter budget, amplify the signal, and retransmit a clean data signal along the selected medium. Because there are no attached digital subsystems to the retiming repeater, there is no requirement to change the data rate to a local digital subsystem data rate. Therefore, a new technique that resets the jitter budget, amplifies the signal, and retransmits the data signal at the incoming data rate is desired. This fourth method is shown in the last column of Table 1.
Method 4 is the Skip-Free Retiming-Repeater, which is new class of retiming-repeater developed by Primarion. Figure 6 shows the system implementation of this method, which employs a retiming technique that uses a clean local clock to reset the jitter budget, but does not alter the frequency of the data. The data is re-transmitted at the same rate as the received data rate, with the low jitter characteristics of the local clock. By performing rate compensation to the retimer local clock instead of the data, the attributes of the clean retimer local clock can be applied to the data signal without changing the data rate. This enables retiming to be accomplished indefinitely without regard to the number of skip symbols allocated for frequency rate compensation at each digital subsystem node.
Figure 6. Skip-free retiming resets jitter budget, but retains the original data transmission frequency (Fc1)
The retimer-repeater has applications in situations where long board traces degrade the signal prior to transmission in chassis based equipment, such as switches, severs, and adapters. They can also be used as signal conditioners to clean up signals prior to optical conversion by optical transponders or for the emerging, flexible, SFP based optical or copper I/O configurations. Finally, the ability to use the retiming-repeater to extend the backplane, copper, or optical link is desired in larger fabric installations. A skip-free retiming-repeater enables the equipment OEM or data center manager to have maximum signal integrity without worrying about jitter peaking or using up part of the skip symbol budget.
Differences in the clock distribution of data communication systems and SONET systems enable the data communications infrastructure to be lower cost at the expense of having to use bit stuffing techniques at each end node. For optimum performance, retiming and repeating is often desired at the edge of equipment or to extend a link. Several methods are available to perform the retiming-repeater function. Some devices do not employ the use of a local reference to retime the signals, but instead recover and filter the clock from the incoming data stream. Because these systems are dependent on filtering the "dirty" input data clock, jitter peaking may occur due to jitter transfer characteristics of the filtering. In data communications systems such as InfiniBand, when a limited number of retiming repeaters based on elastic buffers can be used, a skip-free retiming-repeater offers a solution that resets the jitter budget while operating completely transparently to the link.
