Although copper cables have represented a major obstacle to building large Infiniband-based computing clusters, many system administrators have been reluctant to make the move to optical cabling due to concerns about performance and reliability. A new range of high-performance (20 Gbps) optical interconnect cables designed to enable computing clusters to scale out substantially further (up to 100 meters) than existing 24 AWG copper cables, with strong signal quality, a low bit-error rate (BER), the potential for increased reliability, and other advantages have recently become available.
A key element in the development process of advanced interconnect designs is verification of performance and, in particular, compliance with the required industry standards. As multi-gigabit data rates become common in digital systems, signal integrity (the quality of the signal necessary for proper operation of an integrated circuit) is becoming a paramount concern for designers and customers, and any new cable must be proved not to impact the quality of signals passing through. In this article we will look at the test configuration for a new Infiniband optical cable from Intel.
Growing Clusters
Clusters have almost single-handedly driven the recent rapid growth in the high-performance computing market and are now the dominant configuration for technical servers. At the same time, cluster sizes are growing very rapidly, with an average six-fold increase in processor count and a threefold jump in node count in just the last two years.
With these larger processor and node counts come much larger physical footprints, plus new challenges in areas such as power supplies and cooling requirements. In addition, interconnect-related shortcomings such as latency, bandwidth, and the limited physical reach of existing Infiniband copper cables further constrain the physical size and performance of cluster configurations.
New optical Infiniband cables are designed to address the limitations of 24AWG copper cables. The fiber-optic cables are effective at transmitting 20 Gb/s double data rates (DDRs) at distances of up to 100 meters, compared with a typical range of only 10 meters using 24AWG copper cables.
The performance specifications of the optical cables also top the specifications of their copper counterparts. Each cable provides four 5 Gb/s lanes in each direction (20 Gb/s unidirectional total bandwidth) at InfiniBand DDRs, and somewhat lower rates with SDR or 10GbE CX4-powered sockets. The cables add only 550 pSec of signaling latency per dual optical/electrical conversion, giving them latency characteristics on a par with 24 AWG copper cables. In addition, with a BER of 10-15, the cables produce 1000 times fewer errors per day than copper cables rated at a 10-12 BER. This low BER is especially important for maintaining the stability of the computing fabric, and should contribute to increased cluster reliability.
Other notable advantages of optical cables are size, weight, the bend radius of the connectors, and the impossibility of electromagnetic interference (EMI)-related ground loops. Because they contain no copper cores, the cables are lighter (84%) and less voluminous (83%) than copper cables (Figure 1). The volume advantage is important for alleviating the airflow issues that tangles of bulky copper cables can cause in scale-out clusters.
1. Optical cables are 84 percent lighter than copper equivalents, eliminating the need to reinforce floors and cable ladders.
With smaller connectors (11%), a tighter bend radius (40%), and lighter cabling, optical cables will be easier to install, modify, and maintain. And with no copper wiring, unintended ground loops should not occur.. In sum, optical cables promise to provide substantially better data reliability than copper cables, an especially important consideration for large scale-out clusters with thousands (sometimes tens of thousands) of components. The new cables also present an attractive economic proposition, with pricing close to copper cable pricing at 10-meter lengths and increasingly lower than copper at greater lengths.
Performance verification
The Infiniband cable test specification requires that a signal with a specific impairment is provided from a signal source. This was done using a Tektronix AWG7102 arbitrary waveform generator (AWG) transmitting a test pattern at 5 Gb/s. The signal impairment is created using a direct synthesis method. The direct synthesis technique used in the instrument is a sampling-based technology which creates analog waveforms from a series of sample points. The sample points in the AWG's memory can be used to define essentially any waveform, including digital pulses. The units also offers a sample rate up to 20 GS/s which is supported by additional features including multiple outputs and ample memory to support long pattern sequences.
Direct synthesis techniques allow engineers to create any wave shape, including signals that embody the effects of propagation through a transmission line. Parameters such as rise times, pulse shapes, delays and aberrations can all be generated and controlled, which is what is required for rigorous serial bus testing applications like the Intel cables.
Intentionally applied anomalies are not the only benefit of the direct synthesis AWG approach. The AWG can be used to originate impaired test signals up to 6Gb/s with all the required timing, amplitude, and distortion characteristics rather than producing a clean signal and degrading it after the fact.
To examine the signal after it has passed through the cable, a high-speed sampling oscilloscope equipped with eye-diagram analysis capabilities is required, and the engineers used a Tektronix DSA8200 sampling digital signal analyzer to make these measurements. Jitter, noise and BER analysis software was used to analyze the BER down to the cable's 10-15 BER specification.
The team used the AWG to generate a high-speed InfiniBand serial signal, transmit it down the 100-meter Intel Connects Cable, and observe the waveform at the far end with the digital serial analyzer sampling scope and 80E10 sampling head. Results were analyzed using the jitter, noise and bit-error-rate software.. From these, the team proved that optical cables performed fully to specification and very favorably when compared to conventional copper cable solutions. An eye comparison between the two is shown in Figure 2.
2. As the open eye illustrates, optical cables offer excellent signal quality from 1 to 100 meters, and better than copper at 5 meters.
The official launch of this new cable occurred at the ISC 2007 annual high-performance computing (HPC) conference in Dresden, Germany. For the event, the team provided the full test setup, and Tektronix engineers worked alongside Intel's team to support the launch and demonstrate the cable's performance. Based on the response, it appears that a significant roadblock in the deployment of very large computing clusters has effectively been resolved and that optical cables present a viable alternative to copper for deployment of large-scale computing clusters.
About the Author
Chris Martinez is the worldwide strategic marketing director for instrument business at Tektronix. A 30-year veteran of the company, Chris has held various marketing, sales and engineering positions in his career. His experience includes advanced technical work, product planning, professional selling, strategy development and over fifteen years of marketing advanced instrument test solutions. He has an MBA from the Oregon Executive MBA, University of Oregon, holds a B.S. from Marylhurst College, Oregon, and an Associate of Science Degree from Yuba College, California.
