When the Storage Area Network (SAN) first burst onto the IT Manager's radar, the 'spin' promised much but often failed to live up to its marketing hype. Great fanfares heralded the concept, explaining that users would be able to centrally pool all their data storage and allocate it to any attached server. Despite the hype, the reality remained the same. At that stage, the technology was essentially just Fibre Channel fabrics. While they offered fast, flexible connectivity they still provided only the most basic tool sets, and still suffered from many of the limitations more traditionally associated with Direct Attached Storage. True virtualization of data storage, which allowed physical data storage to be carved up into logical units of capacity, still relied, in the main, on server-based virtualization managers, able to allocate storage capacity to individual servers on a stand-alone basis.

In order to deliver on the much-vaunted promise of the SAN, the industry underwent a huge drive to deliver SAN-based virtualization—which would allow users to pool and allocate attached storage to any attached server on an 'as and when' basis. Suddenly, the rush was on and the first virtualization engines were born.

This first wave of technology took the swiftest route, delivering virtualization 'in a box', in a format largely on NT-based servers running specialized software. In essence, this meant splitting the SAN in two with the virtualization engine sitting between the servers and the data storage devices. All data requests and data transfers had to go through the virtualization engine.

Despite delivering the functionality required, this approach also created a new set of problems for users. Primarily, there was the issue of the virtualization engine being a single point of failure for the SAN and secondly, but no less problematic, was the fact that the virtualization engine represented a potential bottleneck for the movement of data between servers and their data storage.

The proposed response to both of these problems was to install a second virtualization engine. While this certainly worked in terms of giving more data throughput capability and providing failover where needed, it did, of course, drive up costs.

The last, but certainly not the least, of the problems with this technology only became apparent as users started to try to scale their SANs. It swiftly became obvious that in order to maintain data throughput, users had to invest in yet more virtualization engines. Expensive enough in its own right, there was the additional but less obvious cost of adding high-priced Fibre Channel connections consumed as each new virtualization engine was integrated into the SAN. Having said this, the technology did provide a stop-gap answer to users' requirements—until now.

The second and current wave of virtualization technology is what is known as 'off the data-path' virtualization. This works by connecting a Virtualization Manager to the SAN by a single (or double for high availability) link with a set of minimal software agents on each of the servers. The SAN Manager acts like the conductor of an orchestra communicating with the agents on the servers but not interfering in the actual data transfers between servers and data storage—an approach which immediately deals with many of the problems experienced with the earlier technologies. It consumes only a small number of Fibre Channel connections, avoids the problem of bottlenecks with data transfers and is able to scale quickly and easily by leveraging its distributed architecture.

Such an approach also offers additional benefits, which includes a single point of management for data storage regardless of Storage Vendor or Operating System—able to scale to meet future as well as current needs. In addition, the server based agent allows load balancing across a number of Fibre Channel host bus adapters, regardless of vendor—a feature which today costs users a small fortune if they want to buy this functionality from some of the established vendors. This is the way forward that was pioneered by manufacturers such as StoreAge, who have successfully deployed this technology and continue to add new functionality to enhance the basic offering.

So where are we going next? Well, nothing is perfect and even second wave technology has a drawback—what if a user wants the flexibility of the second wave technology such as StoreAge but doesn't want to have any new software installed on their servers? This concern could be due to a number of reasons including high security concerns in environments where there is a fear that any 'alien' software may allow a third party to view what is going on—or the fact that a supplier may refuse to maintain a server if any external software is added.

This is where the Networked Storage Application Platform comes into its own—the platform is designed to work as an element of the SAN and can be programmed to undertake a number of different SAN applications. These could include Virtualization, Information Lifecycle Management (ILM), and SAN based backup. The first implementations we have seen here at Zycko are leveraging this second wave of virtualization technology through application platforms such as that available from Troika. With the Application Platform acting as an enabler for the StoreAge virtualization technology, it is able to manage the virtualization agent on behalf of the attached servers, thus eliminating the problem for users who cannot allow 'foreign' software to be installed on particular servers.

This approach could be seen as introducing a new device in the data path—and, thereby, re-introducing some of the problems encountered by the first wave of virtualization technology—but the architecture of the Troika application platform deals with this. By building on some of the low latency techniques traditionally employed by Fibre Channel switches and adding a few tricks of its own, the Troika Application Platform offers no more of a bottleneck than a Fibre Channel Switch, a major step away from the much larger overhead offered by the first wave virtualization where a full NT server was placed in the data path.

So, where do we go beyond the Application Platform? The future's hazy but there are some shapes beginning to emerge. EMC's acquisition of VMWare and Microsoft's acquisition of Connectix and their subsequent announcement of Virtual Server 2005 (in beta) points to scalable virtual servers being an imminent move. If and when this happens, Application Platforms will be well placed to provide a means of integrating storage and server virtualization into one, single 'system virtualization', providing a single point of management by which the IT manager will be able to regain control of the whole IT system.

Whether this distant future is ever fully realized, the reality is that today the second wave of virtualization is taking what has been the cacophony of networked storage and orchestrating it into the symphony of the SAN.