The Palo Alto, Calif., campus will play host to top designers from Cray, IBM, Intel and a host of other companies and universities disclosing their latest work in chip-, board- and system-level links. This year's confab will unveil research in optics and exotic approaches that hold promise for systems due out in the next several years. It will also show stepwise advancements for today's computers, which are increasingly eyeing Ethernet as a viable option, as well as more expensive and specialized commercial alternatives like Infiniband.

Specifically, Sun Microsystems Inc. will provide more details on the promise and perils of Proximity, its effort to craft a terabyte/second-class chip-to-chip interconnect based on capacitive coupling. IBM Corp. will disclose development work with Corning Inc. on a new approach to optical cell switching that could deliver throughput of tens of gigabytes/second. And university researchers will detail new high-water marks for 10-Gbit Ethernet-based links.

In one panel, computer builders and users will argue whether Ethernet will become the king of the interconnect hill, thanks to its high-volume, low-cost commodity status. "It's an open question," said Fabrizio Petrini, a technical-program co-chair for the conference and a researcher at Los Alamos National Laboratory.

"Today 10G Ethernet is still very expensive," he said. "A card can cost thousands of dollars. . . . On the other hand, the high-performance computing market doesn't seem to be large enough to accommodate multiple standard and proprietary interconnects."

Interconnect plays a key role in the cluster designs that dominate high-end systems today. "The high-performance network in a cluster computer is the computer, because it largely defines achievable performance — as well as defining its scalability, fault tolerance, system software and overall usability," Petrini said.

Recognizing that reality, researchers from Sun will disclose fresh details of the Hero supercomputer under development as part of a project for the U.S. Defense Advanced Research Projects Agency.

Sun says its Proximity scheme will use capacitive coupling to deliver a hundredfold improvement in chip-to-chip interconnects, potentially bringing "the most distant gigabytes of memory" as close as "a first-level cache," according to the paper.

"Using 10 percent of a 150-mm2 chip for Proximity communication with a gap misalignment of under 5 microns yields a bandwidth per square millimeter of about 3.1 Tbytes/s and a chip bandwidth of about 46 Tbytes/s," which potentially exceeds on-chip bandwidth, according to Sun.

Researchers admit, however, that a number of unresolved implementation issues are pushing Sun toward the bleeding edges of chip and module interconnect design. The company faces mechanical, thermal and power challenges packing large arrays of chips via capacitive coupling. Those issues lead to requirements for a water-cooled plate and careful placement of power-conversion circuits.

On-chip links
The ultrafast chip-to-chip links force on-chip and module-level interconnects to make similar advances to keep up. Thus, Sun is developing a new capacitively coupled driver circuit to speed on-chip links while keeping power consumption to a tolerable level.

"We have built silicon implementing these designs in our lab and have found the ideas promising for reducing power," according to the Sun paper.

In addition, "the Hero platform requires [vast numbers of optical] modules with a significant increase in optical interconnection capacity over existing systems," the paper said. That slams Sun up against a reliability shortfall from using huge numbers of vertical-cavity surface-emitting lasers. It also necessitates a module interconnect based on wavelength-division multiplexing to provide an I/O capacity in excess of 2 Tbytes/s.

Sun also found that Hero must support a "sea of anything," or SOX, architecture in which "all resources in the system — including processors, memory controllers and I/O buses — individually interact with each other in a single shared memory address space," the paper said. The SOX architecture has its own Achilles' heel.

"When a system has many components that have been isolated due to failure, the updating of path tables becomes complicated. Research into algorithms that can cope with the scale of a Hero-class system and handle large numbers of random failures is ongoing," the paper said.

Nevertheless, Sun's Proximity work "is probably the major breakthrough of this decade in high-speed interconnects. It will change the architecture of supercomputers," said Petrini of Los Alamos.

Also at the conference, IBM and Corning will provide a first peek at a joint development effort on an optical cell-switching architecture for high-performance computers called the Optical Shared Memory Supercomputer Interconnect System.

Such optical switches can cover clusters spread over large distances while using relatively low power and showing less EMI sensitivity than copper-based solutions. However, they are still more costly than their copper counterparts.

In the copper era
University researchers will show tests of the 10-Gbit Ethernet T110 adapter from startup Chelsio Communications that indicate commodity copper technologies can deliver adequate performance for high-end clusters. Thanks to its TCP offload engine (TOE), the Chelsio card delivered a 9.6-µs latency (against ~5 µs for Infiniband) and 7.6-Gbit/s throughput. The Chelsio card provided three times the performance of a non-TOE 10G Ethernet card when used on an Apache Web server.

But other technologies are not standing still. In the Infiniband world, Mellanox Technologies will describe a prototype implementation of a "zero-copy mode" that ups the bandwidth from 500 to 700 Mbytes/s, even as "CPU utilization decreases eight times," said the paper.

Separately, Ohio State researchers will detail work on PCI Express-based Infiniband cards that eliminate all local memory yet show "practically no performance impact" on some key benchmarks. That work could open the door to Infiniband cards that sink to costs more closely in line with Ethernet clustering adapters.