The advent of delivering 10 Gigabit Ethernet connections to the enterprise and data center opens up a myriad of test challenges. Constructing a comprehensive and standards-based test environment is critical to ensure both product compliance and a high level of confidence in the inherent ability of devices to operate in real-life applications.
Recently ratified, 10GBASE-CX4 is a copper breakthrough that resolves the price challenge of delivering 10-Gigabit Ethernet to stackable switch connections in the enterprise. CX4 provides ten times the speed at three times the cost of the Gigabit solution it replaces, and is approximately 80% lower in cost compared with existing 10-Gbits/s solutions. In addition to price/performance, what enhances CX4's appeal is that it is based on well-known and widely accepted technology—the XAUI interface and InfiniBand cable.
To test CX4's higher-frequency signals in an environment of greater constraints than its predecessor, designers must develop new XAUI interface tests while also improving older tests. For example, whereas eye-pattern testing was sufficient for XAUI, CX4 requires template tests. And, compared with the testing of an individual IC, today's CX4 products are housed in XENPAK and X2 modules that must comply with multi-source agreement (MSA) specs and CX4 standards.
In this article, we will discuss the testing challenges faced when developing 10GBASE-CX4 transceivers. We'll also examine as direct and indirect techniques available both at the IC and system level to provide reliable standards-based testing of 10GBASE-CX4 transceivers. In the article, attention will be paid to the real-life challenges faced by the end customer including measurement of bit error rate (BER), jitter, and a variety of others. Also, the discussion will be based on experience gathered during the successful development of 10GBASE-CX4 XENPAK modules.
Standards Conformance Testing
Conformance is the process of ensuring that standards are adhered to—and in the case of CX4, involves such elements as the testing of the physical coding sub layer (PCS), as well as the physical media dependant layer (PMD).
PCS testing is digital in nature. Different state-machines in the PCS are tested by injecting symbols (data packets) through the device. This is to verify both the functionality of the decoding/encoding engine and the state machines, as defined in the XAUI spec (802.3ae).
Physical media dependent (PMD) layer testing has a reputation as a challenging and time-consuming element of conformance testing for high-speed transceivers. Within the 802.3ak CX4 standard the tradition not only holds true, the challenge is even more pronounced.
An important conformance issue regards the topology and location of the test points for transmit (Tx) and receive (Rx) signals, as defined by the CX4 standard. While IC vendors primarily concentrate on meeting standard requirements at the physical interface of the chip (pins), the new standard expands the focus to include real-life impairments. This is accomplished by selecting the point for PMD transmitter side parameter measurements, defined as TP2, to be the output of the InfiniBand connector (Figure 1). In practice, this requires a high quality test-fixture that has an InfiniBand plug on one side, and usually eight SMA pairs on the other side, connected to the testing equipment.
Figure 1: Diagram showing 10GBASE-CX4 link and test points (Source: IEEE).
As shown in Figure 2, the high-quality test fixture becomes a mandatory element in building a reliable laboratory setup to test for CX4 compliance. The resulting method of signal analysis takes into account impairments of the board traces and connector.
Figure 2: Laboratory test setup for PMD testing of XENPAK-CX4 products.
A key element to ensure compliance of the transmitter is the template test whereby a special low frequency pattern (LFP), consisting of a sequence of quintet zeros and ones, is transmitted from the PHY IC. The device output is captured by a high-speed oscilloscope and overlaid on a predefined template (Figure 3).
Figure 3: Example of Template Test Results.
In order to pass the test shown in Figure 3, all measured data points must be inside an allowed area restricted by high and low borderlines. Meeting this requirement ensures not only that the device under test is set to a pre-defined level of pre-emphasis (emphasizing high-frequency components in the transmitted spectrum), but also verifies that parasitic elements causing severe return loss are not hidden in the signal path.
Dealing with Jitter
Jitter measurement remains the most challenging and controversial segment of the conformance-verification procedure. Although several high-speed oscilloscope vendors claim a comprehensive solution for jitter measurement, the procedure pushes currently available equipment past the limits of their capabilities, and as a result does not always yield consistent results.
Test equipment features critical to enabling reliable and repeatable jitter measurements are a sampling rate of at least 20GHz, and sufficient memory to capture a few million samples. In addition advanced post processing is required to separate the measurements into the different components of jitter since the CX4 standard specifies limits for total jitter and also its basic components: random jitter (RJ) and deterministic jitter (DJ).
Random jitter originates from several sources including thermal and shot noise and is usually characterized as a Gaussian distribution. It continually increases with time, which is the reason why random jitter is described by standard deviation rather than peak-to-peak values.
In comparison, deterministic jitter is always bounded in amplitude and has specific causes such as crosstalk, inter symbol interference (ISI), duty cycle distortion, EMI, and reflections. Because of its bounded nature it is common to describe deterministic jitter using its maximal (peak-to-peak) values. It should be noted that in CX4 applications, part of the deterministic results from the pre-emphasis on the transmit signal, causing a favorable ISI phenomenon. This improves the quality of the signal on the receive side enabling reliable communication over long cables (in contrast to random jitter which is always detrimental to the received signal quality).
Measuring BER
Another aspect of testing standard conformance is measuring bit error ratio (BER), which is performed at TP3, the output of the InfiniBand cable on the far-end (see Figure 1 above). To conform to the standard, the CX4 physical layer (PHY) should have a BER equal to or less than 10-12 over worst case cable as defined by the standard. Although the standard defines two transmit test patterns (CJPAT and CRPAT), mainly for jitter analysis and interoperability testing, in order to simulate real traffic, the PHY should also include the high order PRBS pattern (pseudo-random data), which is more stressful for copper media.
The measurement of BER at 10-12 levels always presents a problem because it requires a very long run time. A single test can take between 30 to 60 minutes for a 100-Gbit/s link. Compounding this issue even further, is the fact that most OEM customers require not just the specified 10-12 BER, but 10-15, 10-16, or 10-17 BER. Performing measurements at these extremely low levels of BER, using conventional methods, could take weeks and even months.
It should also be noted that standard test equipment—the BER tester (BERT)—typically features only one differential lane. However, since CX4 consists of four lanes, measurement has to be repeated four times.
Finally, CX4 solutions are implemented utilizing a variety of cables, provided by multiple vendors. These cables have diverse characteristics and as often occurs in an emerging market, not all CX4 cables (originally designed according to the InfiniBand standard) will meet all 802.3ak (CX4) standard requirements. In addition, due to pre-emphasis, short cables are potentially a challenge for some PHYs, which create a need to test over the entire span of cable length. Consequently, the complex and time consuming BER measurements have to be repeated many times to provide adequate coverage for all types of cable characteristics and lengths.
Indirect BER Measurement
Evaluating the quality of the channel in systems that require BER in the order of 10-12 or better is time consuming because of the necessity to accumulate an adequately large number of bits to calculate the actual BER. Therefore, one of the key challenges in testing the robustness of a system, either in characterization or final production, is to have a capability to provide a good estimation of channel performance/quality within a few seconds.
Performance and quality estimation becomes possible by using a forward error correction (FEC) algorithm, which when used, enables an accurate estimation of link quality within a few seconds. This algorithm operation is somewhat similar to the well-known "importance sampling" method, where noise is added to artificially increase BER, allowing measurement with a lower number of processor bits. Real BER is then calculated without added noise.
Because the signal-to-noise-ratio (SNR) is reduced, the effective BER becomes in the range of 10-5 to 10-8, which is convenient to measure. Actual BER for normal operating conditions can be estimated from that resulting measurement.
The indirect BER measurement system consists of a test board with a resident PHY. The test board connects through a CX4 cable to the device under test (DUT). Both PHYs are configured to work in enhanced mode, activating the FEC and pre-code BER estimator. The result is a numerical indication of the quality of the channel. The numerical indications are provided for both the DUT receive side and the test board receive side (the transmit side of the DUT).
Programmable Parameters
In addition to the indirect BER measurement that allows "quick" performance testing, adjustable parameters such as amplitude, pre-emphasis, equalization, and impedance can be optimally configured during system characterization to achieve the best performance.
The performance of the system must be measured over the entire CX4 link channel. The "channel" is comprised of the transmitter (signal conditioner, I/O), the media interface (cable, connector, PCB traces), and the front-end of the receiver (I/O, clock/data recovery block, and usually an equalizer). As part of the characterization and testing of the system, the PHY requires flexibility to adapt to media impairments resulting from different connectors, trace topology and characteristics, and possibly also for cables that may be outside the standard specification.
To identify the best set of parameters, the PHY features a special integrated indicator, referred to as the signal quality index (SQI). SQI is a special algorithm within the clock data recovery block that evaluates the quality of input symbol samples. During testing the SQI is read by software to identify the set of parameters that give the best performance (highest SQI value).
Let's look at the four adjustable parameters in more detail:
1. Pre-emphasis: This is a mechanism to boost the high-frequency component of the signal on the transmitter side to compensate for high-frequency attenuation occurring on the copper cable. Today's PHYs are able to set pre-emphasis in the range of 0% to 80%, with increased resolution in the range defined by the CX4 standard (21% to 41%).
2. Impedance: The output impedance of the transmitter, and input impedance of the receive side determines the matching of the respective I/O to the media, and is also known as return loss. The better the matching, i.e. the closer the impedance values, the lower the reflection generated on the channel, which directly translates to a better SNR and performance. The channel is assumed to have a differential impedance of 100 ohms. However, this is not always the case due to diverse impairments (PCB traces connectors, etc.). The impedance levels on transmit and receive sides can be adjusted within the PHY, allowing increased matching in various system environments.
3. Amplitude: In some cases, either in systems with higher attenuation, or in those sensitive to strong signals, there is a need to program the transmitter output signal level. Amplitude reduction is also a key factor to reduce near end cross talk (NEXT) noise. Transmitter output can be programmed to amplitude between 600 to 1200 mV.
4. Equalization: This is required on the receive side to compensate for phase and amplitude distortion and to remove remnants of ISI not removed by pre-emphasis. Configuration of the equalization levels can either be accomplished manually, or by using a sophisticated embedded algorithm based on SQI indications.
Module-Level Testing
By way of example, the testing of Fujitsu's x-mGC series IEEE802.3ak XENPAK CX4 modules takes into consideration connector and mechanical reliability, power dissipation, endurance, EMI/EST, and CX4 compliance. Such tests are critical to product quality and the functional assurance of the module.
To guarantee reliable electrical connections, the CX4 connector on the front side of the module and the 70-pin card edge connector at the system side (which is a XENPAK interface) are tested. Each connector is tested to ensure contact resistance and durability in mating with the printed circuit board (PCB) and cable plug. On the CX4 side, transmission characteristics are also tested.
Testing of the module's power dissipation is key to ensuring that the total power dissipation budget is met for the end system. Thermal simulation and measurement of the module are conducted according to test environment and conditions specified within the XENPAK MSA. In addition, the temperature of the re-timer chip used in the module is also measured.
The mechanical integrity of the module is evaluated to ensure meeting the MIL-STD-883 specification. Test parameters include mechanical shock, vibration, and thermal shock. Endurance testing includes aging acceleration, high-temperature storage, temperature cycling, damp heat, and moisture resistance. Additional tests performed under MIL-STD-202, 883, and Telcordia GR-453 standards, validate the reliability of the PCB, components, and solder joints, and also validate the module's life. Appropriate EMI/ESD characteristics are also important factors for telecom/datacom equipment, and tests are conducted according to FCC class B and MIL-STD-883.
Finally, the module is tested for reliable data transmission, signal transmit and receive, and conformance to the IEEE802.3ak CX4 standard. These tests are to ensure hot pluggability and other module-specific capabilities.
Wrap Up
Recently ratified 10GBASE-CX4 is a copper breakthrough that resolves the price challenge of delivering 10-Gigabit Ethernet to the enterprise. However, testing of CX4's higher-frequency signals introduces additional challenges that require new tests be developed and old ones improved.
Direct and indirect techniques are now available at the IC and system level that provide reliable standards-based testing of 10GBASE-CX4 transceivers. Solving real challenges of BER, jitter and a variety of other tests involved several innovations to accomplish rapid and comprehensive testing coupled with performance optimization.
About the Authors
Amir Bar-Niv is a founder and director of applications of Mysticom Semiconductor. Amir can be reached at amirb@mysticom.com.
Dimitry Taich is an application manager at Mysticom. Dmitry holds BCS degree in Physics and Master's in Electrical Engineering from Technion, Israel Institute of Technology and can be reached at dimitryt@mysticom.com.
Osamu Daikuhara is a design engineer within Fujitsu Component's Development Department's Connector Business Group. Osamu can be reached at daiku@fcl.fujitsu.com.
Shigeyuki Takizawa is currently managing development of module projects within Fujitsu Component's Connector Business Group. Shigeyuki can be reached at s-takizawa@fcl.fujitsu.com.
