The face of the metropolitan area networking (MAN) market is dramatically changing. With carriers looking to save on investment, equipment developers are being forced to move away from developing single-function systems toward building multi-service provisioning platform (MSPP) boxes that can handle frame, cell, and packet traffic.
However, the move to an MSPP provides strains on the switch fabric architectures housed in these designs. In the past, fabric architectures were optimized for a specific traffic pattern. However, like the boxes they are housed in, fabrics are also being pushed to support a host of traffic schemes.
Fortunately, new switch fabric components are on the way to answer this challenge. But to be successful, these fabric architectures must support new interfaces, enhanced queuing engines, and more. With that in mind, let's take a look at the key requirements engineers must consider when choosing or building a switch fabric component for an MSPP design.
The Current Landscape
Figure 1 shows a high-level architecture of a typical system targeting applications such as multi-protocol switches/routers, ATM/IP/MPLS/Ethernet switches and other types of broadband switching/access equipment.
Figure 1: Typical architecture for a multi-protocol switch/router.
As shown in Figure 1, current line card designs include a packet/cell processing block that passes a set of parameters as well as the packet or cell to a fabric plane. The specific parameters passed off include packet type, destination address, and priority parameters. The fabric plane uses these parameters to determine the necessary QoS level, including throughput (rate guarantees) and latency (delay guarantees) requirements.
In general, design engineers have had to live with proprietary interfaces when linking the processing block to the switch fabric block. However, over the past few years, chip vendors have started to shift away from proprietary solutions toward the open Common Switch Interface (CSIX), thus providing more flexibility to networking equipment designers when connecting switch fabric to other networking ICs.
Developed by the Network Processing Forum (www.npforum.org), the CSIX interface helps identify the QoS requirements of a particular packet or cell. The CSIX specification defines fields in the header of each frame being transmitted. These fields describe the information being carried in the payload of that particular frame. Specifically, these fields include frame type (unicast, multicast, flow control), class (priority), and destination address (egress fabric port number). This information is used by the fabric queuing engine to determine which fabric queue the frame payload will be stored in.
The CSIX specification also specifies segmentation of variable-size packets into cells for efficienct processing of multiple flows, which has advantages in multi-service applications. A cell-based fabric allows high- and low-priority traffic to be interleaved on a cell-by-cell basis. This reduces the time it takes for a high priority cell to be scheduled for transmission across the fabric. For packet traffic, segmenting the packets into fixed size cells, and interleaving the high- and low-priority cells, reduces the jitter associated with sending large low priority packets across a fabric. These issues determine why most fabric architectures with advanced QoS capabilities use cell-based switching.
The Speedup Factor
One problem that arises from segmenting variable-size packets into fixed-size cells is a reduction in the utilization of raw fabric backplane bandwidth. This is due to the fact that packets that are not multiples of the fabric cell size do not fit evenly into the fixed-size fabric cells that are sent across the backplane. This results in a certain portion of the raw fabric bandwidth being used to transmit the unused portion of the fabric cell (padding).
Figure 2 shows the segmentation of variable-length packets into fixed-length cells with padding added.
Figure 2: CSIX allows for the segmentation of a variable-length packets into fixed-length cells.
With the above situation in mind, designers need to increase the raw fabric backplane bandwidth in order to account for the segmentation of packets into cells. This increase is also referred to as "speedup".
Segmentation inefficiencies and in-band flow control overhead have to be accounted for when defining the bandwidth of a backplane. Thus, designers should look to define their fabric bandwidth as 1.6X to 2X higher than the traffic bandwidth defined at the ports. A speedup factor of 1.6X is acceptable, however, designers should consider a 2X factor to allow room for growth.
Figure 3 below shows the sawtooth effective bandwidth waveform that results from segmenting variable sized packets into fixed size cells. The drops in bandwidth occur at multiples of the fabric cell size, with the worst case being the fabric cell size+one byte. As the diagram illustrates, fabric backplane speedup allows the effective fabric bandwidth to be line rate for all packet sizes.
Figurer 3: Speedup allows designers to account for inefficiencies in the use of fabric bandwidth.
One of the key questions that a designer needs to ask is how to determine the effective bandwidth for their individual fabric backplane designs. This calculation can be achieved using the following equation:
raw fabric bandwidth*(packet size)/(number of cells required*payload bytes per cell)
Queuing issues
The switch fabric is tasked with scheduling queues based on QoS requirements, To do this, the fabric sorts incoming cells arriving on the ingress CSIX port of the intelligent queuing engine on a per destination port basis. In this process, separate queue structures will be used for each destination port. This input queuing technique, called virtual output queuing (VOQ), assures that no traffic will be blocked because an earlier arrived cell is waiting for a congested destination port.
Cells going to the same destination port are also sorted by priority guaranteeing high-priority traffic precedence over low-priority traffic. To make sure that this precedence is accomplished, output queues on the egress side of the intelligent queuing engine are also structured on a per-priority basis.
To meet the QoS requirements of applications such as ATM or Diffserv, each fabric queue will have a scheduling priority associated with it, which will determine how cells are transmitted across the fabric. Based on the ATM forum framework, the architecture for services provided at the ATM layer consists of constant bit rate (CBR), real-time variable bit rate (rt-VBR), non-real time VBR (nrt-VBR), undefined bit rate (UBR), available bit rate (ABR), and guaranteed frame rate (GFR).
To support these ATM services, at least two fabric priorities are required for all of the flows in the fabric, but 4 priorities, plus a priority for control traffic is preferred. Multiple levels of scheduling of the different priority queues give the fabric the flexibility to meet the service requirements of different ATM applications.
Figure 4 shows an example of how ATM class scheduling can be implemented in a fabric with four internal priorities per flow. Note that ATM services are shown since ATM seems to be the industry benchmark for QoS, but the requirements for other standards, such as Diffserv, are very similar.
Figure 4: Implementation of ATM class scheduling in a fabric architecture
Preventing Congestion
In addition to scheduling queues based on QoS parameters, the fabric must ensure that the queues never overflow when congestion occurs. To assure loss-less fabric operation, a set of sophisticated flow control features must be present in the switch chip set.
Fabric link-level flow control monitors the near-full and near-empty conditions in the queues and conveys STOP and GO messages to the previous stages. The granularity of this flow control type is per fabric class and it is most useful during unusually bursty traffic patterns.
End-to-end flow control is the highest level of flow control defined within the fabric plane. Its primary purpose is to enhance the granularity of the fabric link level flow control by operating at the full granularity of the switch interface (CSIX defines 15 traffic classes). Under the normal traffic conditions end-to-end flow control is the primary mechanism in the fabric plane.
The switch interface flow control is the final mechanism that a fabric must support. This flow control method provides a STOP and GO mechanism for all types of traffic (unicast, multicast, broadcast) down to per-port, per-class granularity.
While these three types of flow control mechanisms deliver loss-less fabric performance, the system designers may choose to further enhance performance by adding higher layer of flow control. Packet/cell processing blocks can exchange control messages through the fabric to provide finer levels of granularity and prevent unfair congestion conditions. A high-priority control queue would be used to enable flow control communication between the egress and the ingress packet/cell processors.
Scheduling Decisions
Cell scheduling (i.e. the "which cell depart next" decision) takes place on each cell for transmission through the fabric. An architectural choice needs to be made as to where the best place to make this decision. The choices are:
- On ingress queuing engine
- Centralized at the crossbar
- A combination of both
A cell-based fabric architecture that distributes the queuing and scheduling functions in both the queuing engine on the port card, and the fabric switch matrix on the switch card, is the best option. Let's explore why.
The queues in the switch fabric matrix provide for independent priority-based buffering of multicast, unicast, and control traffic. This allows for an independent scheduling in each of the switch matrix as well as a highly efficient multicast traffic handling. Only a single copy of the multicast packet needs to be sent from the queuing engine to the switch fabric matrix on its way to multiple destination ports.
Cross-point queues in the switching matrix will also allow traffic coming from the ingress ports to be fairly scheduled to the egress ports, which results in deterministic latency for high priority cells being sent across the fabric. Thus, the distributed architecture is well suited to meet the QoS requirements of multi-service applications.
Rate, Delay Guarantees
To support existing TDM services, the fabric must appropriately set parameters. Priority type (strict vs. weighted round robin), priority level, and flow control threshold parameters are needed to define throughput and latency for a given flow. Additionally, designers looking to support TDM traffic should consider implementing buffered crosspoint or shared memory switch implementations because they provide low latency throughout the fabric
Practical Considerations
When considering using any semiconductor solutions the usual practical hardware considerations come into play — integration, performance and cost. For fabrics systems, the highest integration can be achieved by providing a two-piece chipset — one device for each side of the backplane (port card and a switch card). For high reliability and ease of design it is desirable to integrate all queuing logic & buffers, scheduling logic as well as the high speed serializer/deserializer links" in both types of devices.
In real world carrier-class systems, semiconductor device failure is a reality. Thus recovery from device or board level failure is a must. To make this happen, fabric architectures should offer built-in serial link diagnostic and monitoring hardware for detecting faulty connections. Additionally, fabric components must provide automatic/manual traffic rerouting to a redundant path. The transmit drive and receive sensitivity of the serdes must also be variable and link alignment architecture must be flexible to allow migration from single-shelf systems to multi-chassis topologies.
About the Authors
Shawn Collier is a strategic field applications engineer at Vitesse Semiconductor. Shawn supports Vitesse's advanced networking products, which includes the company's switch fabric family. He can be reached at scollier@vitesse.com
Mike Grudsky is a strategic field applications engineer at Vitesse Semiconductor. Mike supports Vitesse's advanced networking products, which includes the company's switch fabric family. Mike can be reached at mgrudsky@vitesse.com.
