The term "Ethernet" is synonymous with local area network (LAN) in today's world. Today's business managers and CIOs, who are comfortable with using Ethernet, are expected to use the Internet as a business tool. Since enterprise customers are a major source of revenue, service providers are under pressure to provide them with a variety of services. These customers would like to have the comfort of a WAN connection that looks and feels like a LAN connection. Because of this, transparent Ethernet transport is a focus of the emerging metro architecture.
This article looks at various ways of transporting Ethernet in the metro area network (MAN). It also attempts to explain how Virtual Concatenation, using Sonet, is best suited to enable service providers to serve this market. For simplicity, only Sonet is referred to in this article. However, all the points discussed here are equally applicable to SDH.
Ethernet over Sonet: What's Required?
As per the Open System Interconnect (OSI) model, Ethernet is comprised of a physical layer and a link layer. Over the years, most Ethernet box vendors have also developed a similar control plane (or network management layer) so upper layers can use Ethernet in a standard way.
On the other hand, over the years the Ethernet physical layer has changed for each data rate hop. Category 3 (voice-grade unshielded twisted-pair) cable was used for 10 Mbps, and Category 5 (data-grade UTP) cable for 100 Mbps. For Gigabit Ethernet, multiple physical layers are available, including copper and fiber for different distances. The emerging 10 Gbps Ethernet standard proposes a variety of fiber and laser types for different distance ranges.
The physical layer has also evolved with different coding schemes in the physical coding sublayer (PCS). All of these changes have been accepted as natural requirements and adopted by the industry wholeheartedly. Throughout the evolution of the physical layer, the Ethernet frame has remained the same, enabling a consistent link layer and upper layer interface. This allows the same "look and feel" of Ethernet to be exhibited to LAN administrators and managers, despite the existence of different physical layers at different data transfer rates. Such consistency of Ethernet link layer has also enabled reuse of investment made in upper layer software and training in the Ethernet based network management.
Now that enterprise managers want Ethernet to extend beyond the boundaries of a single point of presence (POP), they want to retain the same "look and feel" -- that is, retain Ethernet as a link layer. They are really not concerned about the physical layer of Ethernet as long as it provides what it takes to support required performance -- quality and distance -- at a reasonable price. This is the reason why transporting Ethernet over Sonet is a "best of both worlds" solution, where Sonet can provide a proven physical layer for Ethernet link layer in MAN and WAN applications.
Ethernet in Metro--Driving Factors
CIOs, in today's business world, have several needs that can be fulfilled by a WAN connection.
- Connecting remote offices with a single high-speed LAN: It is any CIO's target that everyone in the company is able to connect to all the servers and people he / she needs through the LAN. Within a building or campus, Ethernet has already achieved this. Inter-city and longer distances mean the connection happens through a frame-relay link over leased lines, which has highly unacceptable quality today. CIOs want LAN-like connectivity over MANs and WANs between remote offices with flexible and guaranteed bandwidth.
- Accessing an ASP (Application Service Provider) or a remote storage site: Here again, flexible and guaranteed bandwidth is required. In this case, the protocol may be Fibre Channel or ESCON instead of Ethernet.
- Accessing a web hosting server: Guaranteed bandwidth is required to connect an enterprise website to the Internet so that many customers can access the Web site and transact business easily. If the Web site is hosted by a subcontractor, the subcontractor provides this bandwidth. In either case, flexible and guaranteed bandwidth is required.
- Internet access for employees: The bandwidth consistency in this case is less critical than the previous categories. Nevertheless, this is an important addition to the overall bandwidth needs of an enterprise.
In addition to these broad requirements, budget constraints demand a varied level of services, dependent on the application, time of the day, holiday season, and other external factors. For example, bandwidth allocated for a remote office connection and for employee Internet access can be reduced after working hours, but bandwidth for connections to a Web site host server needs to be available continuously.
Sonet--The Best Physical Layer Choice
Now that we laid out the driving factors behind pushing Ethernet into the metro, let's describes the physical layer criteria required to meet these needs.
The issues involved in high-speed (multiple Gigabits per second), long-distance (100 km and above), complex networks are quite different than those involved in LAN. Some of the critical requirements for a physical layer include:
- Reliability: Maintenance costs increase exponentially with longer distances. Therefore, the need for reliability is much higher than that of a campus LAN.
- Fault Recovery: The cost of losing business, if even one node malfunctions, increases exponentially with bandwidth in a MAN or WAN. Therefore, automatic and quick fault recovery is one of the most important parameters.
- Consistency of service: The signal level, quality of signal, bit error rate, and related parameters are far more significant in MANs and WANs than in LANs.
- Bandwidth management: Dedicated and flexible bandwidth provisioning at the lowest possible cost is very important in MANs and WANs. In LANs, most users have equal priority and equal bandwidth.
These requirements specify a very tightly-coupled control plane with a robust physical layer. That's the only way one can minimize the response time to a network problem and minimize the operational cost for efficient bandwidth provisioning. While the Ethernet physical layer does not meet these requirements, it is not surprising that all of these are supported well by Sonet, because Sonet was structurally designed as WAN physical layer.
Here is how Sonet's control plane and specific features meet these requirements.
- Sonet has overhead bytes in every frame, checking and communicating bit errors, signal health and the general health of the link every 125 microseconds throughout the network. Since this function is hierarchical, problems easily can be identified in any specific section of the network and therefore quick fault recovery.
- Sonet has a well-defined fault-protection scheme. Even in a modified system as in N:1 protection instead of 1:1 protection, Sonet's Automatic Protection Switch-APS--is the only proven fault recovery mechanism which recovers in less than 50 ms .
- Sonet's tight jitter specifications for components ensure better quality of signal over longer distances, resulting in sustained available bandwidth.
- Sonet's hierarchical TDM structure enables adding and dropping of bandwidth at any node in the network, making bandwidth provisioning and management a very simple and economical function.
The only major issue Sonet has been facing is the relative inflexibility of bandwidth provisioning through a Sonet pipe. This weakness is now overcome with an emerging technology called virtual concatenation, which will be discussed later in this article.
Packet Over Sonet (POS)
There are various ways of mapping a packet into a Sonet frame. The most popular method is known as packet over Sonet (POS), defined by ANSI RFC2615 and associated standards. POS essentially specifies the mechanism to transport all types of packets, including Ethernet frames in PPP (Point-to-Point Protocol) format embedded in an HDLC-like frame. To transport Ethernet frames in POS format, Ethernet frames are stripped of preamble and SFD before they are wrapped in PPP and HDLC-like frames. A robust and stable link is ensured by scrambling and CRC checks. Emerging protocols like Generic Framing Procedure (GFP) claim to provide more robust link than PPP/ HDLC type framing.
The success of POS can be attributed in large measure to Sonet's strengths and wide deployment. However, most POS deployment is done over a concatenated Sonet frame. A concatenated frame does not allow bandwidth management and therefore leaves room for new technology. The rest of this article is focused on issues related to dynamic and efficient bandwidth management and how is it effectively addressed by emerging technologies.
Contiguous Concatenation
Sonet has a hierarchical structure. As shown in Figure 1, three STS-1 (OC-1) streams can be multiplexed to create an STS-3 stream. Four STS-3 streams can be multiplexed into an STS12 stream and so on. This enabled easy aggregation of lower rate streams of telephony traffic. For data transport such hierarchical multiplexing is superfluous since IP traffic is statistical and "bursty" in nature.
Figure 1: A diagram of a channelized Sonet frame. In this example, three STS1s carrying different data streams get multiplexed into one STS3 stream.
To avoid hierarchical multiplexing and to make Sonet look like a flat physical layer, a Sonet payload is concatenated by creating a homogenous container carrying a single stream of packets. As shown in Figure 2, by concatenating the payload, packets can be mapped straight into a higher-rate Sonet stream directly, rather than breaking it into lower rate streams and then multiplexing them back into a higher rate stream.
Figure 2: A diagram of a concatenated Sonet frame featuring a single STS3c stream.
In line with Sonet hierarchy, these containers are:
- STS-3c -- 155 Mbps
- STS-12c -- 622 Mbps
- STS-48c -- 2448 Mbps
- STS-192c -- 9792 Mbps
Concatenated Sonet payload has become the norm in the last few years in almost all new long-distance networks. However, they generate their own set of problems:
- The contiguous concatenated payload may not compatible with the existing deployed network. All the equipment in the network must be upgraded to support concatenated Sonet frames.
- Bandwidth management at the physical layer is not possible with concatenated Sonet because the channelization in the payload does not exist. Therefore, all packets must be taken off the Sonet stream at every node and then packed again in the outgoing Sonet frame. This increases the processing burden on each node in the network.
For an Ethernet link over Sonet, these problems result in highly inefficient usage of bandwidth through the network. Table 1 demonstrates the bandwidth inefficiency of this approach.
Table 1: Bandwidth efficiency of mapping LAN rates to contiguous concatenated Sonet frame
| Service Rate |
Sonet Container |
% Bandwidth Utilization |
| Ethernet (10 Mbps) |
STS-1 = 51 Mbps |
20% |
| Fast Ethernet (100 Mbps) |
STS-3c = 155 Mbps |
42% |
| Gigabit Ethernet (1000 Mbps) |
STS-48c = 2448 Mbps |
42% |
Since transparent LAN and storage protocol transport is becoming highly important from a revenue standpoint, this kind of bandwidth wastage is highly unacceptable to the network operators.
Virtual Concatenation
Virtual concatenation solves the problem of flexible bandwidth provisioning by creating variable bandwidth virtual pipes in a Sonet stream. In contrast with contiguous concatenation of containers -where multiple containers are merged to carry a single payload - a single payload is fragmented and distributed among multiple containers in virtual concatenation (Figure 3). This way, Sonet can retain its structured, channelized frame, making it compatible with existing deployment, while at the same time carving out different size streams or pipes within the Sonet stream.
Figure 3: Virtual Concatenation enables channelized STS3 stream carrying 2 virtual channels with different bandwidths.
Virtual concatenation utilizes specific bytes in the Sonet path overhead in order to transmit the payload concatenation sequence information over the network, thus identifying each container in a Sonet frame. This method enables the receiver to reconstruct the single continuous payload even though it was fragmented at the source of transmission.
Virtual concatenation is feasible using STS-1 or STS-3c as containers for a higher rate. For a lower rate, containers such as VT1.5 can be used. Therefore, the granularity of bandwidth allocation is as fine as the granularity of a Sonet frame. Sonet's hierarchical structure offers finer granularity than any other technology, including Ethernet, and hence can offer a variety of bandwidth options.
A major advantage of virtual concatenation is that it is defined at the "path" level. Therefore, an OC-48 ring, consisting of 48 STS-1 containers, remains an OC-48 ring, even though all of its bandwidth may be fragmented into virtual pipes operating at different rates. In other words, virtually concatenated containers can be transported through existing networks transparently. Only the path-terminating equipment needs to be "virtual concatenation aware."
Finally, the virtual link created by virtual concatenation is dedicated and highly secure, because it operates directly at the TDM layer. No packet is looked into, broken or controlled for bandwidth allocation issues. Except at the path terminating nodes, packets are not "touched" over the transport network at all.
Bandwidth Efficiency
With virtual concatenation, bandwidth utilization can be improved to over 90% in all the cases mentioned in Table 2--without byte-by-byte control of bandwidth.
Table 2: Bandwidth efficiency of mapping LAN rates to virtually concatenated Sonet frame
| Service Rate |
Sonet Container |
% Bandwidth Utilization |
| Ethernet (10 Mbps) |
VT1.5-7v = 1-.5 Mbps |
95% |
| Fast Ethernet (100 Mbps) |
STS-1-2v = 103 Mbps |
96% |
| Gigabit Ethernet (1000 Mbps) |
STS-3c-7v = 1085 Mbps |
92% |
In addition to such bandwidth utilization at the originating node, virtual concatenation also enables the maximum utilization of bandwidth in the network. Since the containers are virtually bonded, they are actually free to traverse different paths through the network to arrive at their destination -- similar to how IP packets traverse different paths in a pure packet network. However, in contrast to IP packets, the Sonet container provides a pre-provisioned stream and therefore, guaranteed and dedicated bandwidth.
An example is illustrated in Figure 4. Customer XYZ wants 200 Mbps bandwidth to connect its Headquarters at node A with a remote office at node B. For some reason, there is no single route in the network connecting these two nodes with 200 Mbps bandwidth available when demanded by Customer XYZ. However, two different routes can support 100 Mbps each. Without virtual concatenation, Sonet could not utilize this split bandwidth. With virtual concatenation, four STS-1 containers can be virtually concatenated at the originating node A to create a 200 Mbps connection. At appropriate junctions (DCS), they can be split such that two STS-1 streams can traverse the lower ring and other two can traverse the upper ring as shown in the figure. They can be recombined at the destination node and a continuous stream of 200 Mbps can be recreated. This way, even fragmented bandwidth in the network can be utilized.
Figure 4: Virtual concatenation enables utilization of bandwidth fragmented over different routes.
Since each container can travel separately through the network, differential delay among containers needs to be compensated for at the receiving end. Sonet/SDH is a pre-provisioned & switched network, differential delay is easy to estimate, detect and compensate reliably.
Bandwidth Flexibility
Once a virtually concatenated channel is established, its capacity can be modified provided the network can accept such a change. Bandwidth modification is achieved by changing the number of virtual tributaries bonded to create the virtual channel. Two different methods are evolving for dynamic reallocation of bandwidth of a virtual channel. They are (1) Link Capacity Adjustment Scheme (LCAS) and (2) Generalized Multi-Protocol Label Switching (GMPLS).
LCAS is a protocol for communication between two nodes and any intermediary nodes about required change in a virtual channel's bandwidth. It has been proposed to the International Telecommunications Union (ITU) and ANSI standards committees. LCAS uses Sonet path overhead bytes for communication between remote nodes.
GMPLS utilizes the popular MPLS concept. Each container can be treated as a separate label and hence, MPLS concepts can be used to dynamically create, modify and tear-off label switched routes (LSRs) for each of the containers. GMPLS enables usage of proven and popular IP routing protocols like OSPF and RSVP for this purpose.
Ethernet access link
Since Gigabit Ethernet is expected to provide the least expensive link to access networks, customers requiring bandwidth from 100 Mbps to 1000 Mbps will find it better to use Gigabit Ethernet as the link to a WAN box. Service providers will allocate only the required number of STS containers for each such link, depending on how much bandwidth the customer has actually purchased.
To illustrate this point, as demonstrated in Figure 4, Customer XYZ can actually use Gigabit Ethernet links at both ends to connect its headquarters to its remote office while using only partial bandwidth on such link. To prevent Node A from dropping packets, the Customer Premise Equipment (CPE) can be equipped to control the actual bandwidth demand on such an access link. At a later point in time, if required, the customer can subscribe to additional bandwidth between headquarters and the remote office or some other point in the network, and use the same Gigabit Ethernet access link.
Wrap Up
Ethernet in the MAN is a demand-at-link layer. Over metro- and wide-area networks, physical layer specs need to be more stringent, and a tightly-coupled control plane is essential for the lowest cost and maximum reliability. Sonet as a physical layer meets all of these requirements.
Virtual concatenation of Sonet containers enables multiple, variable bandwidth links for transparent Ethernet transport over Sonet. Therefore, Ethernet over Sonet using virtual concatenation is the optimum solution in the MAN.
About the Author
Nilam Ruparelia is a product manager at Cypress Semiconductor. He holds a BE in Electronics and Communication from the Gujarat University, India. Nilam can be reached at UNR@cypress.com.
