VCAT can be incorporated into an existing Sonet/SDH network by adding technology at the end points. The existing infrastructure can carry the data end to end. Equipment vendors are adding that capability by offering multiservice provisioning platforms, which in the Sonet/SDH space look like born-again add/drop multiplexers (ADMs).

Another technology crucial to offering metro Ethernet services is the generic framing procedure. GFP is standardized by the ITU as G.7041 and describes the encapsulation and data-rate adaptation techniques for transporting various protocols over Sonet/SDH networks.

GFP offers two categories of service: framed and transparent. Framed GFP packages a complete Ethernet (or other) frame into a GFP header. It is important to realize that the frame is carried in its entirety; thus, to the end user, it appears that the Ethernet network is expanded and can be managed like a large enterprise network.

Transparent GFP creates a data pipe that moves 8B/10B encoded data from end to end in a streaming fashion (Figure 2). Streams of 8B/10B traffic are encoded into 64B/65B superblocks for transport over the Sonet/SDH network. Rate adaptation is achieved by inserting and removing idle characters.

Today it is possible to transport Gigabit Ethernet, 1- and 2-Gbit Fibre Channel, Ficon, Escon/Sbcon and DVB-ASI over transparent GFP. By including a multirate serializer/deserializer into a transparent-GFP media access control (MAC), it is possible to offer a multiport, multirate, multiprotocol "universal line card."

Data aggregation
Today, framed GFP is standardized for Gigabit Ethernet. Other protocols, including Fibre Channel and lower-speed Ethernets, will be standardized, but it may be economical to do some aggregation of customer traffic before it enters the Sonet/SDH data infrastructure. This can be done with most enterprise-class Ethernet switches available today but presents a unique challenge in metro Ethernet, which is not commonly found in the enterprise.

A typical scenario is shown in Figure 3 wherein a service provider offers 10/100 Ethernet virtual private network (VPN) services by aggregating the 10/100 Ethernet traffic over a Gigabit Ethernet connection. The aggregation can be done with an Ethernet switch, but the provider must be able to differentiate the traffic flows to and from each customer's premises (i.e., in Fig. 3, the "red" flow and the "green" flow).

Aggregated Ethernet flows can be distinguished by inspecting parts of the Ethernet frame. By examining the outer-most virtual LAN (VLAN) tag, multiprotocol label switching (MPLS) label, Internet Protocol (IP) type-of-service byte, DiffServ code point or Ether-net source address (or combinations of these), we can use a simple table lookup to determine the Sonet/SDH VCAT group into which the flow should be encapsulated.

Using a simple tag (VLAN or MPLS) to identify a flow is convenient but poses some data coordination difficulties. We could not use a VLAN tag on one end of the network that would collide with the use of a VLAN tag on the other end of the network. Instead, we can a add label "push, pop or swap" feature. This can be achieved by enhancing the table lookup slightly. Upon entry into the Sonet/SDH network, a table lookup is performed, providing the interface number and VLAN tag as input. The table lookup provides the following:

• Sonet/SDH VCAT group channel. This could be an internal channel number or a channel number for a channelized data interface (such as SPI-4.2). Either way, the channel number maps the data flow into a Sonet/SDH VCAT group.
• New label (a label that will identify the data flow to the network egress point.
• Label action (push or swap).

The lookup results are examined to determine how to manipulate the data frame. The "label action" field tells us either to "push" the "new label" field onto a label stack or to swap the existing label with the new-label field.

Ethernet VLANs can be stacked using the relatively new "Q in Q" label-stacking scheme, so named for VLAN standard IEEE 802.1Q. MPLS labels have had stacking capabilities from the start. A bit in the MPLS field identifies a label as the outermost label.

On egress from the Sonet/SDH network, a similar lookup must be done. This time, we provide the label content and the Sonet/SDH VCAT group on which it arrived. Lookup results provide:

• Egress interface (port).
• New label (for swap operation).
• Label action (pop or swap).

Again, lookup results are examined to determine whether we swap the outmost label with the label field or just pop the label off the stack. The egress port is also returned by this lookup function, providing enough information to send the frame to the customer.

Lossless flow control
Another challenge that will be encountered in moving Ethernet to the metro is the need for lossless flow control. Ethernet comes equipped with the ability to send pause frames once a watermark is tripped, but now our reach must extend beyond the data center. The reach for a metro Ethernet network really needs to be in the 5- to 10-kilometer range. Jumbo frames must also be supported.

To understand the requirements, let's examine what takes place in the event of flow control.

Buffers are managed by setting watermarks. Tripping a watermark is an indication that congestion is occurring and must be addressed if the lossless environment is to be preserved. With Ethernet, congestion is managed by sending pause frames toward the traffic stream that is causing the congestion (Figure 4). Again, measurable latency exists that needs to be addressed in order to preserve the loss-free network.

Once a watermark is signaled, the following affect the latency in which the downstream device will react:

• The interface is already transmitting a packet downstream. That packet may be a jumbo frame, incurring up to 9,600 bytes of latency.
• The transmit time of the Pause frame itself (64 byte times)
• The latency characteristics of the fiber itself.
• The receive time of the Pause frame (64 byte times).
• The downstream device is allowed up to 1,024 bit times to react to the Pause frame.
• The downstream device may already have a frame in transmission. That frame may be a jumbo frame.

To preserve a loss-free environment in metro applications, there needs to be enough buffering to hold up to three jumbo frames (9,600 bytes) per interface to accommodate a span of 10 km.

Traditionally, Ethernet throughput is offered in increments of an order of magnitude. From the customer's point of view, that may present a less-than-desirable situation. A customer may wish to have access to a 3-Mbit service or a 50-Mbit service. In the Sonet/SDH world, 3-Mbit service can be realized by concatenating 2 VT1.5 tributaries, while 50-Mbit service fits nicely into a single STS-1.

Quality-of-service
In the Ethernet world, 3-Mbit/s service implies a 10-Mbit Ethernet connection that is policed down to 3 Mbits/s. A 50-Mbit service implies 100-Mbit Ethernet, policed down to 50 Mbits/s. Ethernet has become so economical that using a link to less than its throughput capacity has a very minor impact on the total cost of the solution. The bulk of the cost is still in the Sonet/SDH part of the network.

Provisioning is always provided by, at a minimum, specification of a committed information rate (CIR). Traffic that complies with its CIR is always delivered. Failure to do so may result in the service provider's paying a financial penalty, since these parameters are written into the service-level agreement.

Optionally, a burst information rate can also be specified. Such traffic is delivered on a "bandwidth available" basis.

Provisioning simply on a CIR can be done with a single token bucket. In the case of aggregated traffic, a token bucket per flow needs to be employed. If the amount of traffic exceeds the CIR, then either the traffic is dropped or the link is flow-controlled. Ethernet asserts flow control on an entire link, so in the case of aggregated traffic, traffic that exceeds the CIR must be dropped.

Provisioning can also be offered using a burst information rate (BIR). Here, traffic that exceeds the CIR parameter but is less than the BIR parameter is delivered on a "best effort" basis.

To support BIR, a dual token bucket mechanism is implemented for each flow-one for the CIR parameter and the other for the BIR parameter. Traffic is then marked as "green," "yellow" or "red" using the following criteria. Traffic that is compliant with its CIR, as measured by its token bucket, is marked green and is always transmitted. Traffic that exceeds the CIR but is less than the BIR is marked yellow and is transmitted on a best-effort basis. Traffic that exceeds the BIR, as measured by its token bucket, can be dropped or delayed (shaped), or flow control may be asserted.

Handling short bursts of traffic that exceed CIRs is valuable but relatively difficult. As already stated, such traffic will be marked yellow and will be handled if capacity allows. The available buffering space needs to be monitored to ensure we do not prevent green traffic from being properly handled. This is also done with a configurable watermark; once the watermark is tripped, yellow packets will be dropped.

There is some debate as to whether it is advantageous to drop from the head of the queue (i.e. packets that have already been buffered) or from the tail (before a packet is buffered). Dropping from the head allows buffers clogged with yellow packet to drain faster, which will be important if there are a large number of green packets behind the yellow ones. But dropping at the tail is simpler from an implementation perspective.

Finally, we need to consider the priority of one traffic stream relative to the other. Assuming our philosophy is "all green packets get through" and "all red packets are dropped," this again is most important in considering yellow packets. Here, several watermarks will be implemented to monitor buffer space and demark "drop yellow packets of priority 'n' and lower." The number of watermarks provided will determine the number of priorities offered.

It is instructive to think of this network as connecting the enterprise (where Ethernet is the dominant technology) to the Sonet/SDH carrier. Given that, it seems logical to pick either Sonet/SDH or Ethernet for the task, thereby simplifying management, training and integration.

With recent innovations in Sonet/SDH and metro Ethernet, the perfect storm of technologies has been brewed to offer Ethernet essentially over any distance.

Related Article
"Ethernet-over-Sonet Tutorial"; www.commsdesign.com/story/OEG20020418S0005

About the Authors
Mark Fauber (markf@vitesse.com) is a product line manager at Vitesse Semiconductor. He has a degree in industrial engineering from Iowa State University and an MBA from the University of Colorado.

Dave Dubois (daved@vitesse.com) is a director of product strategy at Vitesse. He has a Bachelor of Science degree in computer science and mathematics from the University of Rhode Island.