The advent of switch fabric interconnect (SFI) technology has added a new degree of complexity to the world of backplane design, revealing a need for thorough analysis and understanding of traffic requirements to identify corresponding solutions. In these designs, engineers must contend with four primary system traffic patterns—equal access, primary access, multiple access and centralized access.
In this article, we'll take a look at the different SFI architectures available to designers as well as at the four primary traffic patterns. We'll also take a detailed look at a proposed switched card interconnect architecture that splits the control and addressing from the data plane on a channel-by-channel basis in order to maximize available bandwidth on the data channel. But, before diving into the existing and new architectures, let's start by taking a basic look at the differences between serial and parallel backplanes.
Clarifying the Terms
Since backplanes have become a focus of activity with many new terms used to describe them, it is good to begin with a very brief discussion of terminology.
Serial and parallel are often used to describe the two major categories of backplanes. This is unfortunate due to the dual nature of each of these terms. In the sense of a parallel connection, the term parallel is correct for describing a multi-point backplane. But, since most multi-point backplanes have multiple connections of parallel signals, the ambiguity of the term can cause confusion. The same is true of serial (or serialized) data and serial connections. The terms multi-point and point-to-point (or single point) might be more descriptive of the actual nature of the two types of card interconnect configurations.
Interest in point-to-point card interconnects, often called SFI, has become intense. The number of solutions available for card interconnects is expanding rapidly, representing a major change affecting all aspects of system implementation. It is important to carefully select a card interconnect solution, since a backplane implementation may exist for a long time.
The various types of backplane topologies demanding the most attention today are: SFI mesh, SFI semi-mesh, SFI star, modified SFI star, modified SFI mesh, and multipoint. The term SFI, used in five of the topology names, simply indicates that it is based on a switch fabric interconnect architecture. The mesh topology provides a direct connection to each line or function card without using a separate switch card, but some form of routing is placed on the line or function card. Semi-mesh indicates that not all cards are directly interconnected.
The star topology provides a separate switch card through which all traffic is routed before reaching the destination line or function card. The multi-point topology, also known as a parallel backplane topology, connects all cards in parallel, thus providing only one active channel. The concept of modified, in all cases, involves variations on the base topology to suit certain applications. An indefinite number of options certainly exists, some of which will almost assuredly become viable as SFI evolves and new solutions to problems are found.
What's a Suitable Solution?
What systems are best suited for the various card interconnect solutions? The answer to this question depends on what is required of the card interconnect scheme. In general, the multi-point bus is an extremely robust solution that can work for any system application. Its primary drawback has always been extracting enough bandwidth to suit the application.
A point-to-point card interconnect solution is good when certain cards—but not all cards, such as broadcast—need to communicate and when multiple streams of simultaneous communication are helpful. A point-to-point switched card interconnect solution is suitable when all cards need to communicate, when multiple simultaneous channels are valuable, and when the required aggregate bandwidth exceeds that which can be attained with a multi-point backplane.
Keep in mind that each point-to-point link in any of the point-to-point solutions will often have bandwidth that is much lower than can be attained with a multi-point configuration (Table 1). However, the total bandwidth potential can be many times the total bandwidth capacity of a multi-point solution. This is appropriate when many communications links to multiple points are required. Of course, the single-channel bandwidth limitations of the point-to-point solution can be easily overcome by adding more signals to each path. Note that the bandwidth of the multi-point interconnect in the table is reduced with additional cards due to the increased load on each signal.
Table 1: Bandwidth Comparison of Different Interconnect Architectures
Click here for Table 1
System Implications
The next consideration when selecting a backplane is the type of system traffic that will be encountered. There are many ways to group systems into operational classes, but the following four categories&151;equal access, primary access, multiple access, and centralized access—work well for understanding the implications of backplane operations and configurations.
Figure 1 shows an equal access system, in which each function card requires equal access to the backplane. The best example of this is a data transmission switch system where each line card provides a bandwidth link of equal magnitude to the outside world.
Figure 1: Equal access traffic patterns, used in systems such as network switches and base stations, employ an SFI mesh, semi-mesh, or star topology.
A primary access system, as shown in Figure 2, can be viewed in two ways: (1) as a system in which one function card needs equal and continuous access to the backplane and every other card in the backplane; or (2) as a system in which every other function card of the system needs equal and continuous access to one specific function card. An example of this is a data transmission router system in which one system card provides a high-speed link between two networks. Every other system card provides local links at a much slower rate to switches or end terminals within a single network.
Figure 2: : SFI modified star or mesh configurations are ideal for network routers, base stations, and other primary access systems.
The multiple access system in Figure 3 is one in which many, but not all, system cards need varying amounts of access to the backplane. A good example of this is a server or a RAID system in which many function cards in the system need access to the backplane, but not always equally or continually, and where only some cards perform access requests at all. Complicating the picture further is the need in some systems for different communications paths, traffic types, and information interaction depending on which two entities are communicating.
Figure 3: Servers, RAID systems, and other devices employing multiple access traffic patterns require varying access to the card interconnect data.
Finally, centralized access (Figure 4) is a system in which a single card controls all access to the backplane and requires most, if not all, of the access. The best example of this is the standard PC in which the central processing unit on a single card or motherboard requires most, if not all, of the access to the backplane. Another example is the control portion of a backplane in a data transmission system that sets up and maintains the system's configuration and position within the network, reports on system and network performance, and may also control the data transmission or switching function.
Figure 4: Centralized access systems can employ multipoint or modified SFI star topologies, since a single card/function controls all access to the card interconnect data.
It's important to note that the PC is moving away from centralized access to something closer to multiple access, where each function can control its own transmission rather than depending on microprocessor intervention for every transaction over the backplane. Once this happens, the PC will function more like a multiple access system with an expected dramatic performance improvement.
Varying Techniques, Requirements
To optimize performance, each system implementation requires a different communications technique and backplane configuration.
In the case of an equal access system, a multi-channel, point-to-point switched backplane is perfectly suitable, especially if it is a data transmission switch system. Effectively, the star topology of the network is brought in-system to the backplane. In the past, a multi-point interconnect was often used for this backplane. A mesh backplane topology using on-card switching or multiplexing may be considered for situations where even less blocking or higher throughput is desired, though it may prove very expensive.
For the primary access system, a number of alternatives exist. A direct point-to-point connection to all the secondary cards provides a straightforward solution, although this may entail a number of traces on the backplane. This point-to-point solution will, more than likely, have a switch or high-speed multiplexer on the primary access card depending on requirements and configuration. However, this could be looked at from another point of view, such as an equal access system modified for primary access by designing a high-speed link onto the switch card(s). Multi-drop or multi-point architectures can also be considered.
In the case of a multiple access system, the degree to which the traffic varies between cards affects backplane selection. For fairly well-behaved traffic, either a multi-channel, point-to-point backplane or a multi-channel, point-to-point switched backplane provides the most likely solution. Whether to use switch architecture or not depends, for the most part, on how many cards need to be interconnected through point-to-point channels. If the traffic varies widely between the cards, then the ability to use the total bandwidth of the backplane can be advantageous. This can be done with the single-channel multi-point backplane.
For centralized access systems, the multi-point backplane is probably still the best architecture since there is little need for multiple channels talking to multiple cards. A direct point-to-point backplane can be considered if the card with the centralized access requirements has performance capability well beyond the accessed cards and if the overall performance requirements of the system exceed what is reasonable for a multi-point backplane. In this case, as in the primary-access case, connections are point-to-point from the access card to each of the accessed cards. This may change in the future as more computing elements co-exist in a single system. Of course, this also changes the system to a multiple access system.
Obviously, multiple traffic patterns and multiple backplane implementations may exist within a single system. An example of this is when a multi-point backplane is used for control and out-of-band addressing while some form of point-to-point is used for data transmission.
Another factor that may affect a backplane implementation decision is broadcast and multicast capability. In some applications a card will need to transmit the same information to more than one other card (multicast) or, in some cases, all other cards in the system (broadcast). So far, the multi-point backplane still succeeds better at this than other card interconnect schemes, since once a card has taken control of the backplane it has access to all other cards.
Whether it is required to send data to one card or all cards is not an issue with a multi-point implementation. In the case of a point-to-point switched backplane, each of the individual channels needs to become free before a multicast or broadcast can be accomplished. Alternative methods, such as sending multiple multicasts to a group of ports, can achieve the same result, but this inhibits other traffic on the sending channel for extended periods. If many short multicasts are required by the application, this could become prohibitive.
Traffic Flow
The flow of traffic through a system varies dramatically depending on the type of system. The two primary types of system traffic are through flow and interactive flow.
As seen in communications systems, through flow data is sent through the system with little or no required interaction, other than seeking the correct output port and checking for errors. Latency is typically accepted, within reasonable bounds, in most communication systems.
Computing systems, on the other hand, require an interactive flow in which the data is requested and/or modified. Because a microprocessor awaits the data to continue operations, latency becomes a critical component of the efficiency and cost/performance trade-off of the system. Pipeline efficiency, another consideration in interactive flow, is concerned with keeping the flow of data constant. The potentially high latency provided by packet-switched backplanes may surprise some backplane designers, especially those in the computing environment.
Problematic Differences
It is important for designers to recognize that the point-to-point card interconnect environment is extremely different from the system interconnect environment. What are the differences, and how can designers capitalize on them? Since signal travel may be measured in a matter of inches rather than meters or miles, is there a more efficient way to transmit the data that takes advantage of the characteristics of the environment?
Obvious changes are that the drive required and the frequency attainable in a backplane will be quite different than the drive or frequency to send an electrical signal across a room or a country or under the ocean. Another big difference is that the destination is one of a very limited number of cards in a backplane.
In the network environment, there are billions of possible destinations; the media also changes. Traces, free air, or fiber may be used in a backplane, whereas wires or fiber will normally be used to interconnect systems. The cost of adding a number of traces to a card or backplane is radically different than running a new set of wires through a building, campus, country or ocean.
When does fiber make sense in the backplane? At present, it is extremely expensive, but with the advent of VCSEL lasers, the cost is rapidly coming down. Also, as with a virtual midplane, the interconnect becomes essentially non-existent and, therefore, no cost. Do wave division multiplexing (WDM) and packet transmission make sense in the backplane environment? Often packets are transmitted across a backplane, but the packet destination is not the card. A card-to-card protocol of some form is sometimes overlaid on the packet.
Proposed Solution
One concept that has been investigated is the idea of completely, physically separating out the control and addressing from the data on a channel-by-channel basis. Two point-to-point connections (address/control and data) for each channel would be required.
Some very powerful benefits, both from simplification and efficiency, result if the data is separated from control and addressing. This may be economically justifiable in a backplane environment. Some of these advantages include:
- The ability to maximize throughput and port utilization;
- Ongoing communication between each card and the switch card(s);
- Minimization of both setup and transmission latency.
Through the proposed architecture, it is possible to reduce transmission latency—defined as the time a bit leaves the serializer to the time it arrives at the deserializer input, typically after being routed through some form of switch—to tens of nanoseconds, excluding trace delays. Short multicasts, messaging, and doorbelling may also be more efficiently provisioned in this manner. Sending short transmissions like these over the control/address channel frees the data channel of these cumbersome operations. Both the computing environment and the multi-protocol environment should be able to benefit from this architectural change.
Wrap Up
Backplane design has recently evolved into a complex issue requiring thorough analysis and understanding of a system's particular architecture and traffic pattern requirements. As a new tool for the systems developer, point-to-point card interconnects offer tremendous potential benefits—along with demanding new challenges. In this article, we described the backplane terminology, applications, and important issues relating to traffic pattern and flow in actual systems. We also provided card designers with a brief survey of the general types of backplane solutions now available and being proposed.
About the Author
Mike Fowler is the principal product strategist in Fairchild Semiconductor's Interface Group. Mike holds a bachelors degree in Electrical Engineering from the University of Louisiana and can be reached at michael.fowler@fairchildsemi.com.
