We've all heard the story. Parallel shared-bus architectures are showing their age, especially in communication designs pushing into the 2.5-Gbit/s range and beyond. As a result, designers are requesting new point-to-point serial architectures that provide better scalability and reliability in a comm equipment design.
The challenge is picking the right architecture. Over the last two years, a ton of serial interconnects have been hyped in the market, making the choice of an interconnect scheme a real headache.
With the development of the advanced-switching feature set, the PCI Express specification is quickly emerging from the dust as one of the better choices for designers. Through advanced switching, PCI Express provides the serial capabilities needed to handle higher data rates while providing the PCI compatibility needed to evolve architectures.
In this article, we'll detail how the advanced switching set builds on the base PCI Express architecture. We'll also show why the advanced-switching feature set makes PCI Express a better option for comm designs.
The Path to Performance
The advanced-switching feature uses the identical physical layers as the base PCI Express standard. The fundamental PCI Express physical layer is developed around a building block that includes two point-to-point unidirectional paths. Each path consists of a low-voltage differential signaling (LVDS) differential pair operating at 2.5 GHz. Utilizing the 8B/10B coding scheme, each PCI Express lane provides an effective bandwidth of 2 Gbit/s in each direction, or a full duplex bandwidth of 4 Gbit/s.
Lanes can be combined to provide higher bandwidth when necessary. PCI Express allows interconnects with x1, x2, x4, x8, x12, x16 and x32 bi-directional lanes to be configured for a maximum 128 Gbit/s of full duplex bandwidth today. And there is more headroom still in the architecture, as future serializer/deserializer (serdes) technology pushes PCI Express beyond today's frequency and bandwidth limits by merely changing the physical layer and not affecting the data link layer or higher layers.
PCI Express, both base and advanced switching, also provides a quality-of-service (QoS) capability using differentiated traffic classes that are mapped to virtual channels within the product. Each virtual channel has associated with it a separate path within the chip and over the links, providing the ability to allow some data to be given a higher priority than others, and to enable higher priority data to flow through the system even if lower priority data is blocked.
Key Differences
As shown above, there is quite a bit of overlap between the base and advanced switching PCI Express specs. There are, however, important differences between the base PCI Express spec and the advanced switching spec that make the advanced switching spec a better solution for comm systems. Let's look at these differences.
The base PCI Express architecture is at its root a serialized, packetized version of PCI with a memory-mapped addressing model and a host-centric hierarchical tree topology (Figure 1). This is appropriate for ease of migration from PCI, and is one of the main reasons that PCI Express will be adopted widely.
Click here for Figure 1
Figure 1: Topology for a base PCI Express interconnect.
The memory-mapped model, however, has limitations when used in communication designs. Requiring the system to route all traffic through the host node creates a bottleneck and puts extra loading on the host processor and host memory. Second, memory-mapped systems do not easily support multi-host systems since the two hosts must figure out who is going to set up the memory map. Typically, this creates the need for non-transparency to isolate the hosts to separate areas of the memory map. Lastly, memory-mapped architectures do not easily support failover mechanisms since the only option when traffic fails to reach its destination is to keep re-trying the transaction.
To solve the limitations described above, the advanced switching specification contains a new transaction layer that supports a wide range of topologies and the features to make them work efficiently, providing scalable bandwidth and predictable traffic control for communications-oriented equipment. In these new topologies there is really no concept of a central host, and traffic can be efficiently redistributed across all of the system resources, eliminating the host-centric bottlenecks. These new topologies do not require host configuration of a memory map but instead use path routing so that each device knows how to get to every other device. When the main path is blocked due to a failure, devices can re-route packets to their intended destinations through alternate paths, allowing seamless failover to occur.
The key new features in advanced switching include peer-to-peer communications, multicast operations, path routing, reliability and universal protocol encapsulation. We'll look at each of these below, starting with peer-to-peer and multicast.
Peer-to-Peer and Multicast
The traffic in a base PCI Express system flows to and from a peripheral and a host. In order for a peripheral on a line card to send information to another line card, say in a router, the data must first travel from the source line card, through the host, then back out from the host to the destination line card. Under normal circumstances the host will inspect and modify the data prior to sending it back out, but even if no modification is necessary the path through the host taken.
The operating scenario described in the paragraph above creates several bottlenecks in a communication system design. The first bottleneck is the movement of data through the switch fabric. Under the base PCI Express spec, data must move through the fabric twice—once to go from the source line card to the host, and once to travel from the host to the destination line card. This adds latency and uses bandwidth that could be put to better use. This is often a bottleneck at the host even if the fabric can handle the extra traffic, since the I/O and memory paths to the processor are normally a performance-limiting scarce resource.
The second problem designers will encounter with the base spec is strain on the host processor. In systems employing a base PCI Express interconnect, the host processor strains to keep up with the traffic flow since in this model it has to process the data before sending it back out. This problem has been made worse as line speeds have increased, and multiple ports are shared on a line card. Multi-gigabit Ethernet or fibre channel systems put a burden on even very powerful processors.
PCI Express advanced switching provides the ability for a line card to send information directly to another line card in a peer-to-peer manner. This ability is especially useful when the line card has intelligence on it, such as a network processor—an approach used today in routers and media gateways, where heterogeneous data types are flowing through the system at wire speed.
In addition to the ability to send information directly from one point to another without host processor intervention, PCI Express advanced switching provides for multicast capabilities. There are situations where designers want to send information to a variety of line cards simultaneously, but without sending the information to everyone (which would be broadcast) and without having to explicitly and serially target each node one by one. An example of where this capability is important is in storage, where the systems has a large number of hard disks, and it is necessary for the information to travel to many disks at once for redundancy.
This is where multicast comes in. Using advanced switching, data can be replicated at the switch based upon a multicast table. It then can be sent to the selected target nodes without having the sending node replicate the packet transmission.
Routing by Path
Above, we discussed the topologies and capabilities used to send information from one point to another with an advanced switching interconnect. The underlying mechanism that makes this work is called path routing.
Advanced switching uses a path route that provides the ability to offer virtually any topology and to scale to extremely large numbers of nodes. This is much more efficient for communications systems, which tend to have a significant amount of concurrent traffic in many parts of the system
With path routing, data travels from point A to point B by means of a series of directional instructions through each switch in the fabric (Figure 2). The information about the path is contained in the header of the message itself, rather than in a universal table in the switch.
Click here for Figure 2
Figure 2: Diagram of path routing in a PCI Express interconnect.
Keeping path information in the header simplifies the routing elements in the system, and eliminates the need for packets to carry source and destination node IDs. And since the path through the system is explicitly and uniquely contained in the message, it is easy to backtrack through the system, the reasons for which include a faulty link or a completion response.
Reliability and Predictability
Communications and storage systems form the backbone of most businesses today, from their web site to their financial and accounting systems. The ability to offer always-on capability is mandatory, and such systems by their nature need to be extremely reliable.
PCI Express provides a variety of mechanisms to ensure reliable operation. The specification includes a link layer CRC to protect transfers from point-to-point between devices, and an end-to-end CRC in the core spec to ensure that a packet travels through the entire fabric without being distorted.
In order to allow reliability at the system level, PCI Express includes a mechanism for hot add/remove. This lets the fabric be dynamically changed or repaired without having to take the entire system down.
The advanced switching spec builds on the capabilities provided by the base spec, adding the ability to include multiple hosts in a system for an additional level of redundancy. With redundant hosts sharing the same switch fabric, the failure of one host does not take down the entire system. The second host can take over the responsibilities of system mastering.
Advanced switching puts up no barriers to the type or degree of redundancy incorporated into a system. In addition to redundant hosts, it is permitted to have redundant fabric masters, ports, links, switches or even entire fabrics.
Universal Protocol Support
One of the most powerful features of the advanced switching specification is its ability to tunnel other protocols in a transparent manner. This feature is called encapsulation. The standard accomplishes this by identifying the packet in the header in a field called the protocol encapsulation interface (PEI). This allows the fabric to transmit the payload without actually needing to understand the underlying protocol. PCI Express Advanced Switching allows 247 different possible payloads to be encapsulated.
Encapsulation allows a heterogeneous system to be created, combining other standards such as Ethernet, SPI-4, H.110, T1/E1, Sonet, fibre channel, etc. while using advanced switching as the backbone of the system.
In order to use the protocol encapsulation feature in a system, a bridge is needed to translate the native protocol, such as SPI-4, to an advanced switching-compatible entity. This is accomplished by creating an advanced switching header identifying the underlying non-PCI Express protocol, and including the native protocol as the payload.
An example of a heterogeneous system is one containing only PCI Express nodes, both base and advanced switching. Since the physical layers are identical between the base and advanced switching spec, the same LVDS I/O can be used to connect to either a base device or an advanced switching device. With a switch that can handle both PCI Express nodes, mixed base/advanced switching systems can be easily created by plugging in both base and advanced switching components to the same switch (Figure 3).
Click here for Figure 3
Figure 3: Diagram of a system featuring base and advanced switching PCI Express nodes.
The Bottom Line
The ability of advanced switching systems to incorporate base PCI Express elements and the protocols from legacy interfaces is a major boon to investment protection, and provides the ability to leverage initial investment in base PCI Express into future advanced switching systems. Additionally, by adding functions like peer-to-peer and multicast, the advanced switching spec makes PCI Express a more effective and attractive solution for communication equipment designs.
About the Author
Jeff Dodson is an ASIC architect at PLX Technology. Jeff holds a BSEE from Duke University and can be reached at jdodson@plxtech.com
Michael Sarpa is PCI Express program manager at PLX Technology. He holds a BSEE from California Polytechnic State University, San Luis Obispo, and an MBA from Santa Clara University. Michael can be reached at msarpa@plxtech.com.
