IBM SAN Volume Controller: Optimizing Storage Utilization and Availability

Introduction

IBM SAN Volume Controller (SVC) is a network-based storage virtualization solution with over 13,000 appliances shipped running in more than 4,300 systems worldwide. This report documents the results of ESG Lab testing of IBM SVC performed in 2006 with a focus on the fundamentals of network based storage virtualization including ease of use, non-disruptive virtualization, data mobility and copy services. The enhanced storage utilization and availability capabilities of SVC 4.3 as validated by ESG Lab in 2008 are also presented.

Background

ESG surveyed 210 IT decision-makers who evaluate and purchase storage networking technologies to see who is considering an intelligent storage network solution and why.   For the purposes of this study, ESG defined an intelligent storage network solution as one "that provides storage services like storage virtualization, volume management, data migration, and data replication that execute directly on network-resident devices such as SAN switches or purpose-built appliances as opposed to hosts or storage subsystems."[1] As shown in Figure 1, simplified management, improved utilization and reduced costs topped the list of potential benefits that non-adopters were most interested in. A strikingly similar list of realized benefits was reported by early adopters.  As a matter of fact, for the top two benefits on both lists, the actual benefits achieved by early adopters exceeded the expected benefits of planned adopters.

The benefits shown above should look strikingly familiar to IT professionals who have experience with server virtualization. Like server virtualization, storage virtualization enables consolidation, which reduces complexity and cost. Storage and server virtualization are also similar in their ability to provide increased availability, fault tolerance and mobility. Given the similarities, it's not surprising that, in a more recent study, ESG learned that an increasing number of organizations are deploying server and storage virtualization together. Twenty-four percent have already deployed storage virtualization in conjunction with server virtualization and 33% plan on doing so within the next 24 months.[2]

 

One of the most significant findings-if not the most significant-is the substantial economic benefit of deploying an intelligent storage network solution. For example, 79% of early adopters with a large SAN believe they have reduced their annual storage hardware spending to some degree, with these users reporting a mean average annual savings of 19.7%.  As shown in Figure 2, storage hardware savings were the most significant due to the ability to reclaim and re-use existing storage (the least expensive storage is the storage you already have).  Consolidating storage software that previously ran on multiple servers or storage systems onto a centrally managed infrastructure also helped reduce storage software spending and storage administration costs.

The IBM SAN Volume Controller

SVC is an appliance-based storage virtualization solution that is deployed in a storage area network. Up to four pairs of appliances, deployed in a fault tolerant cluster and managed as a single solution from a centralized console, can be deployed in a data center to provide:

• Optimized capacity utilization and availability
• A centralized platform for the management of SAN-attached storage capacity
• Heterogeneous interoperability between a wide variety of hosts, operating systems and storage systems
• Online data migration services
• Remote replication and disaster avoidance services
• Space efficient provisioning and copy services

SVC may be used to provide these functions to enhance existing installed storage or as part of a new storage deployment.

 

The goal of ESG Lab testing of the IBM SVC was to validate the fundamentals of network-based storage virtualization including ease of use and non-disruptive virtualization, data mobility and copy services.  The performance impact of SVC virtualization services and the capacity and availability enhancements introduced in SVC version 4.3 were also explored. ESG Lab found that the IBM SVC is a rock-solid, feature-rich platform that delivers on the promise of network based storage virtualization by reducing the complexity and cost of managing SAN attached storage.

ESG Lab Validation

ESG Lab validation of the SVC virtualization platform was performed at an IBM facility in Gaithersburg, Maryland.  The test bed used during the validation is shown in Figure 3 and is documented in the Appendix. Separate test beds used to test Metro and Global Mirroring are presented later in this report.

SVC was used to provide storage virtualization services on each of the operating systems and storage systems shown here during the ESG Lab Validation.   Host Bus Adapters from QLogic and MDS switches from Cisco were used for SAN server connectivity and switching.  The IBM SDD multi-path driver, which is provided at no extra charge with SVC, was used for a fault tolerant connection between servers and the SAN fabric for all of the operating systems tested, except for VMware, where the native multi-path driver built into VMware ESX was used.

 

Since SVC was first announced in 2003, IBM has been committed to expanding the interoperability of SVC.  The breadth and depth of the test bed used during the ESG Lab Validation represents a fraction of the servers, operating systems, host bus adapters, switches and storage systems supported by SVC.  A complete list of the hardware, firmware and driver levels supported by IBM is available on the IBM support website.[3]

Getting Started

SVC appliances are logically configured in-band between servers and SAN attached storage. Physically introducing SVC into an existing SAN environment is performed in two steps:

1. Run FC cables between the SVC appliances and a SAN switch.
2. Add the SVC to the server and storage to the Fibre Channel zones that you would like to manage.

Once cabled, the SVC console is used to configure virtual storage in three easy steps:

1. Run a discovery process (or wait for the next automatic discovery to occur).
2. Configure a logical unit (LUN) presented by a storage array as a managed disk (MDisk).
3. Create a virtual disk (VDisk) to be presented to a server using one or more MDisks.

SVC can be configured to work with volumes that are already in use by an existing application. After the three steps above have been performed and the server has performed a re-scan or reboot, applications can be restarted. When used in this manner, SVC operates in a mode referred to as Image Mode.  Image Mode sets up a one-to-one mapping of a physical MDisk to a host-accessible VDisk.   Because the VDisk has exactly the same mapping as the underlying MDisk, any data already on the disk is still accessible when migrated to an SVC environment. Image Mode VDisk support is a powerful concept that can be used for initial testing of SVC connectivity.  A storage administrator can optionally de-configure or "undo" an Image Mode VDisk during initial SVC testing.

 

After an Image Mode VDisk has been configured and tested, it is typically migrated to a MDisk Group (MDG) to gain the full benefits of SVC virtualization and copy services.  An MDisk Group is a collection of managed disks presented by one or more storage arrays.  MDisk Groups can be configured in two modes: striped or concatenated.  Host-accessible VDisks, which have been built using capacity from an MDisk Group, can be expanded, moved and copied while applications remain online and available.

ESG Lab Testing

ESG Lab validation testing began by moving through a three step process that is typically performed by customers getting started with SVC. As shown in Figure 4, a LUN residing within an IBM disk array and accessed from a Windows server as drive letter F: was tested in three modes:

1. Un-managed This is the state before SVC was introduced. A LUN within the disk array is being accessed directly by a SAN connected Windows server as driver letter F:.
2. Managed Image Mode VDisk This is the state after SVC was first cabled into the SAN and configured. The LUN is now being managed by SVC and accessed by the Windows server as an Image Mode VDisk.  Data written to the F: drive before SVC was introduced remains accessible and unchanged.
3. A  built over a Managed Disk Group This is the more typical usage of an SVC managed disk group. In this case, the F: drive being used by the Windows server is composed of capacity from four managed disks within a storage array.  This can now be migrated, copied within an array (FlashCopy), and replicated between arrays (Metro or Global Mirror), and expanded-all while applications continue to access the F: drive without interruption.

ESG Lab testing began with an un-managed LUN residing on an IBM storage array being used by an NTFS file system on a Windows 2003 cluster.  The NTFS file system was accessed as drive letter F (as shown above in step 1).  The contents of the program file directory on the C: drive were copied to the F drive before it was configured as an Image Mode VDisk using SVC (as shown above in step 2).  After a Windows disk administrator utility rescan, the drive was again accessed as drive letter F: and the copy of the program files directory was inspected. From a user standpoint, there was no indication that anything had changed at all.

 

The SVC Validation continued with a quick tour of the basic capabilities of the SVC. ESG Lab found the SVC management console to be intuitive and straightforward.  Commonly performed tasks are wizard-driven as shown in the screen shot in Figure 5, which was used to create a managed disk group.

Managed disk groups are used to create a pool of storage from which virtual disks are carved.  Managed disk groups can be striped or concatenated.  A 200 MB striped mode managed disk was configured using the intuitive wizard shown in Figure 6.

 

ESG was impressed by how easy it was to get up and running with SVC.  The console application has a very consistent look and feel, includes wizards for commonly performed tasks, and has excellent online help. With virtually no supervision, the following tasks were completed in just under an hour using multiple storage arrays from multiple vendors, connected to a variety of operating systems,  created managed disk groups (striped and concatenated), created a VDisk, expanded a VDisk, deleted a VDisk, migrated a VDisk, created a flash copy of a VDisk , monitored task progress and viewed logs.

The IBM System Storage Multipath Subsystem Device Driver (SDD) was pre-installed on the Windows server for this phase of testing.  SDD is a multipath driver that is installed on servers to provide a fault tolerant path to shared network attached storage.  IBM provides SDD at no additional charge for SVC customers, and is committed to the support of emerging native multi-path drivers built into operating systems (e.g., the native VMware multipath driver used later during ESG Lab testing).  For customers with many servers, this can significantly reduce the total cost of ownership of a shared fault-tolerant SAN storage solution, especially when compared to competitive virtualization solutions that mandate the use of for-fee multipath software.  As a matter of fact, the savings in multi-path software licensing alone may in some cases provide cost justification for the initial purchase of SVC.

Why This Matters ESG Lab found that getting started with SVC virtualization was intuitive and straightforward.   An existing volume being used by an application can be changed into an SVC Image Mode VDisk (and back again if desired) in a matter of minutes. Once under SVC control, volumes can be expanded, migrated, copied and replicated to a remote site-all while applications remain online and available.

Space Efficient Virtual Disk

Space Efficient Virtual Disk, a feature introduced in SVC version 4.3, can be used to present a virtual pool of shared capacity that is bigger than the actual amount of physical storage available. This valuable capability, often referred to in the industry as thin provisioning, was designed to maximize capacity utilization and simplify storage capacity allocation.

When provisioning storage using traditional methods, system administrators allocate a fixed amount of capacity that is dedicated to an application. To avoid the potentially devastating impact of an application running out of storage capacity, storage administrators typically allocated more capacity than was needed to accommodate growth.  As shown in Figure 7, on top of this allocated, but unused pool of capacity, a second pool of capacity that has not been allocated is typically set aside as a reserve for applications that have exceed their pre-allocated capacity.  Provisioning storage capacity from a centrally managed pool of just-in-time capacity, the IBM SVC Space-Efficient Virtual Disks function simplifies the cumbersome task of storage provisioning while providing significant capacity savings.

As an example of what happens during traditional storage provisioning, consider a 500 GB volume that has been allocated to an application with only 100 GB of actual data being created. In this example, the other 400 GB has no data stored on it. That allocated and unused capacity belongs to that application-no other application can use it. This means that 400 GB of the allocated 500 GB is wasted storage, which means that it also wastes money and energy. And even though all of the storage capacity may eventually be used, it may take years to do so, which is a waste of storage capacity and money.

 

With IBM SVC space-efficient virtual disks, the storage provisioning process starts out the same. The storage administrator configures a virtual disk with 500 GB of capacity to be presented to a server. Later, when only   100 GB of actual data has been written, the unused 400 GB is still available for other applications. This approach allows the application to grow transparently while simultaneously ensuring that capacity is not wasted. In other words, SVC delivers just in time storage capacity. The application thinks it has 500 GB of storage, but SVC allocates capacity as needed. The rest of the capacity stays in a shared managed disk pool. Administrators can set thresholds to be alerted when capacity needs to be added.

ESG Lab Testing

ESG Lab configured a space-efficient virtual disk for use on a Windows 2003 server.  ESG Lab noted that this valuable new capability, introduced in SVC version 4.3, was added as a checkbox to the virtual disk creation wizard as shown in Figure 8.  After selecting the space-efficient checkbox, a dialogue box was used to define the size of the volume presented to the Windows server (10 GB), enable a warning when capacity utilization reached a defined threshold (50%), and enable automatic capacity expansion.  ESG Lab noted that capacity threshold warnings can be set not only at the virtual disk level, but also for a managed disk pool shared by a number of virtual disks.

The 10 GB space efficient disk was presented to the server and discovered using the Windows disk administrator utility. An NTFS file system was created and presented as a G: drive. The Microsoft Windows Explorer utility was used to monitor used and free capacity as files were copied from the C: drive on the Windows server to the space efficient G: drive.

 

The SVC management console was used to monitor the actual physical capacity used. Automatic expansion was observed as physical capacity was allocated as needed. When the used capacity as reported by the Windows Explorer utility exceeded 50%, an SVC threshold warning appeared in the SVC management console log as expected.

At the point in time shown in Figure 9, a pair of SAN-attached IBM SVC appliances was presenting a 10 GB G: drive using only 4 GB of actual disk capacity.  With traditional provisioning methods, the 6 GB shown as free space by the Windows Explorer utility would be stranded and wasted.   In contrast, a quick look at the IBM SVC console confirmed that only 4 GB of capacity had been consumed. In other words, 6 GB of capacity savings was saved as SVC delivered just-in-time capacity from a shared pool of managed disk.

Why This Matters ESG conducted a telephone survey of enterprise-class storage administrators with a focus on the limitations of traditional storage provisioning methods. [4] Fifty-four percent were aware that they had stranded and unused storage capacity due to inefficient provisioning methods. Fifty-five percent reported that between 31 and 50 percent of purchased capacity was stranded and unused. Eighty percent felt that storage provisioning was a significant time and resource drain. Thin provisioning simplifies the cumbersome task of storage provisioning and improves capacity utilization. ESG believes that thin provisioning is one of the most useful functions of a storage system and yet few storage systems currently support this capability. Space-efficient virtual disks provide the benefits of thin provisioning for all SVC-managed disk arrays even if they do not support thin provisioning themselves. With a wizard-driven configuration process and advanced management capabilities including alert thresholds at the virtual disk and pool level, ESG Lab testing has confirmed that SVC space-efficient virtual disks can be used to reduce the cost and complexity of storage provisioning while providing significant capacity savings.

Data Migration

Data migration is the movement of data that is currently in use by an application to a new set of storage devices, often on a different storage array. Data migrations occur most often when an organization buys new storage. They also occur to provide more capacity or performance for an application to meet the needs of new applications or to move information assets between tiers of storage with different performance and cost attributes according to the needs of the business.

Traditional data migration methods are complicated and disruptive. Applications must typically be shut down for the duration of the migration. Data is moved from one system to another using backup tapes or software, which moves the data over a LAN. Regardless of how the data is moved, traditional data migrations cause extensive downtime, consume significant resources and are prone to error.

The SVC platform performs data migrations without causing application downtime. Migrations can be performed within and between disk arrays from multiple vendors.  Instead of consuming server, backup, or LAN resources to move the data, SVC appliances are used as the engine which drives data migrations over a high speed SAN.    SVC migrations can be started and monitored from the SVC console or scripted using the SVC CLI.  To minimize any potential performance impact on running applications, the priority/speed of SVC data migrations is user selectable.

ESG Lab Testing

ESG Lab performed a migration from an EMC Symmetrix to an IBM DS8300 disk array, as shown in Figure 10. The volume was being used as a file system (/home/esg) on a server running the IBM AIX operating system.  A script was started on the AIX server to exercise the volume during the migration.   The script created a file every two seconds and ran for the duration of the migration.  No performance impact was noticed during the migration.

The migration was initiated using an intuitive wizard from the SVC console.  Migration status was monitored from the SVC console, as shown in the screen shot depicted in Figure 11 that was taken when the migration was 20% complete.   A 6 GB volume took approximately one minute to migrate, yielding an effective single volume migration rate of 100 MB/sec.

ESG Lab spoke with a customer who is a "...huge fan of SVC. With our four node SVC cluster, we were able to migrate over 50 TB of data in less than four months.  The first couple migrations were performed during maintenance windows, but we didn't have to shut down applications. Now that we are comfortable with the system, migrations are routinely performed outside of maintenance windows with applications up and running."  In addition, the customer said, "SVC is now a core part of our infrastructure.  We first embraced virtualization in our server infrastructure with VMware. Now that we've virtualized our storage infrastructure with SVC, we can quickly deploy new applications without worrying about over provisioning. If an application developer comes to me with a request for a hundred user application that grows to a thousand users overnight, I can move the application to a new server or storage pool on demand."

Why This Matters Data migrations using traditional methods cause application downtime and can have significant business impact. ESG has found that most customers can tolerate very little downtime without impacting revenue.  ESG research indicates that the majority of respondents could not tolerate more than one to four hours of downtime before experiencing a significant impact to their businesses (23%).  Nearly as many said that they could not tolerate more than an hour (22%) and many couldn't tolerate any downtime at all (14%).  ESG Lab has verified that SVC performs data migrations with no application downtime.

Scalability and Performance

When considering the performance impact of a virtualization layer inserted between applications and SAN attached storage, the first issue to consider is the delay caused by such a layer during normal processing.  ESG Lab's experience with a variety of storage virtualization solutions has shown that while these delays deserve scrutiny, they tend to be so small in comparison to the length of a storage operation that they won't be noticed.  The potential performance benefits that can be gained using caching and drive striping within a virtualization layer and the performance scalability of the virtualization layer are just as important, if not more so.

ESG Lab Latency Analysis

ESG Lab audited the results of IBM tests, which measured the latency inserted by SVC during a 4 KB sequential read from disk.  A single logical drive was tested. The elapsed time needed to read 6 GB of data was measured before and after the insertion of SVC between the server and a storage array.  The sequential pre-stage capability of the SVC was disabled to negate the performance benefits of caching.   The latency added by SVC to each I/O operation was 60 microseconds.

To put a 60 microsecond delay into perspective, consider the anatomy of a 4 KB disk read request as shown in Figure 12.  The timeline for the I/O starts with a read/write command being sent to the drive from an application. This is followed by long, mechanical access times waiting for the drive to move the actuator (seek) and spin the actuator over the data requested (latency). Next, the data is transferred from the drive to the server and a status handshake is performed to terminate the request. The timeline shown here uses the average seek and latency numbers for a popular high-performing Fibre Channel drive and command and status overhead measured using a bus analyzer.[5] Note that the access time associated with the mechanical disk drive (seek + latency) is clearly responsible for the majority of the "wait time."

A 60 microsecond delay is very small in comparison to the drive wait time, which is typically measured in milliseconds.  As a matter of fact, it is so small that it can't be drawn to scale on the diagram. If it could be drawn, the bulk of the delay would be inserted at the very beginning of the I/O on the far left.  To put the delay into perspective, consider a typical disk I/O that takes 5 milliseconds. Sixty microseconds represents only 1.2% of five milliseconds.  In other words, at the application level where performance delays matter most, a microscopic delay of this magnitude cannot be detected by users.  Moreover, this impact only occurs during the minority of I/O requests that are reads serviced directly from disk instead of SVC cache.

ESG Lab Caching and Striping Analysis

The micro-level latency impact of SVC is dwarfed by the macro-level performance benefits of SVC caching and striping. Experienced storage professionals are well aware of the performance benefits of caching. SVC uses advanced algorithms, similar to those in caching disk arrays, to perform writes at memory speed, pre-stage sequentially accessed read data and deliver frequently accessed reads from memory instead of disk.

SVC has eight gigabytes of cache per node (and thus a minimum of 16 GB per cluster), which is larger than many midrange disk arrays. This large centralized cache provides dramatic performance benefits. Cache hits, which use memory to perform data transfers in microseconds, are dramatically faster than cache misses, which take milliseconds because they are serviced from mechanically spinning disks. ESG Lab analyzed the cache pre-stage benefits for a 64 KB sequential read. When compared to a read miss with cache disabled, ESG Lab noted that SVC caching improved performance by 25%. The test bed and methodology for this result was the same as that presented above during the ESG Lab Latency Analysis.

Experienced storage professionals should also be aware of the performance advantages that can be realized by striping application data over multiple disk drives.   The more drives per application volume, the more work that can be done in parallel.  This is a simple matter of physics. A mechanically spinning disk drive can only find and transfer one request at a time.  For applications like databases where multiple users tend to access data on the same application volume, striping over many physical drives can provide dramatic performance benefits.  Striping has traditionally been implemented in host based volume manager software and within RAID groups inside storage arrays.  One of the key benefits of storage virtualization is the ability to deliver this performance capability as a centrally managed service.  SVC can be used to stripe application volumes over as many drives as desired.  Drives can be striped within and between arrays.  Drives can be added to an existing volume to increase performance on the fly.

Why This Matters SAN attached storage is often deployed for performance sensitive applications including databases and e-mail.   Customers therefore need to be assured that a virtualization layer inserted between applications and SAN attached storage doesn't negatively impact application performance.  ESG has confirmed that in the worst case of a cache read miss, the SVC adds a very short delay (60 microseconds), which is invisible to users and applications.   In the more typical case, the caching and wide striping capabilities of SVC actually improves application performance.

ESG Lab Performance Scalability Analysis

The modular clustered design of SVC was originally conceived at the IBM Almaden Research Center with infinite scalability in mind.  Industry stan