HP X9000 Network Storage Systems Scale-Out NAS Performance: Extremely Scalable Performance for Large and Small File Workloads

Background

File data growth has outstripped e-mail- and database-driven growth for some time now.  The reasons are many: richer file formats, ubiquitous photo and video, online communities, collaboration tools, 3-D modeling, and 4-D imaging to name a few. But exotic file types are only part of the challenge.  Organizations dealing with extensive and ever-growing archives of general purpose office data struggle with the same problems; they need to accommodate growth efficiently, manage it simply, and access it quickly.

Simultaneously, economic conditions have lead businesses to emphasize first CAPEX, and now OPEX, reductions more than ever.  File growth has resulted in higher costs in terms of additional storage, complex management, and data center floor space and energy.  As a result, the ability to scale out-that is, independently scale and tune bandwidth, processing, and storage capacity on the fly while managing a single, global namespace-is extremely popular for increasing efficiency and saving money.

Adoption of scale-out NAS solutions is driven by its ability to address multiple challenges. ESG survey respondents report selecting scale-out NAS to achieve faster storage provisioning, better scalability with easier management, improved performance of both I/O and throughput, and higher data availability (see Figure 1).[1]

Introducing HP StorageWorks X9000 Network Storage Systems Scale-Out NAS

The X9000 Network Storage Solutions are an integrated appliance series consisting of industry-standard HP servers and storage with the IBRIX Fusion segmented file system. Three models are available: one built to optimize performance or capacity, one focused on low cost and high density, and one gateway front end for use with existing SANs.  X9000 couplets can be combined into a single file system with nearly linear capacity and performance scalability.

The X9000 is designed to enable extreme scaling, up to and beyond the currently tested 16PB configuration, in a single namespace. As a result, large file systems can be accessed, managed, and protected from a single interface. Unlike other clustered file system solutions, the X9000 was designed for multi-tenant environments which often have a wide variety of workloads, including response-time sensitive, small random workloads (such as general purpose file sharing) and bandwidth-intensive, large sequential workloads (such as high definition video editing). Features include:

• Non-disruptive growth. The X9000's modular structure accommodates multi-petabyte growth without downtime. Built-in policy management and automated file migration capabilities let IT optimize for performance, capacity, or retention.
• Non-disruptive workload re-balancing. Workloads are spread across storage resources to avoid overworking individual drives, reduce hotspots, and enhance performance.
• Online migration between storage tiers. X9000 systems help reduce costs by enabling customers to move the most active data to the highest performance drives and less active data to more cost-efficient drives.
• Automated administration. Managing a single file system with an intuitive graphical interface and rich set of command line interfaces reduces administrative cost and complexity.
• Enterprise-class software features. The X9000 supports valuable enterprise-class storage software capabilities including continuous remote replication, snapshots, and automatic online migration between storage tiers.
• Predictable performance at scale. Fully distributed metadata versus a centralized lock manager file system enables predictable, near-linear performance across the file system namespace.

ESG Lab Validation

ESG Lab evaluated the results of three sets of X9000 performance tests that were run in an HP lab in Fort Collins Colorado.  The first tested large block sequential throughput using three HP StorageWorks X9320 Network Storage System (X9320) couplets, the second tested large block sequential write performance using three HP StorageWorks X9300 Network Storage Gateway (X9300) couplets, and the third tested small file performance using a single X9320 couplet.

Fusion File System

The Fusion file system at the heart of an X9000 series scale-out NAS solution was designed to provide near linear performance scalability for large and small files, regardless of the size of the file system. Fusion technology   was designed by IBRIX, which was acquired by HP in 2009. The Fusion file system uses a segmented architecture that provides scalable performance for large and small file workloads.

The Fusion file system distributes file data and metadata over multiple servers that work together in a fault tolerant, scalable cluster. Each segment server in a Fusion cluster has a map to determine which segment holds a particular file. As a result, regardless of which node accepts the data request, the server can quickly identify the location and deliver the data.  This approach minimizes performance bottlenecks as metadata and file access traffic is distributed evenly over multiple segment servers.

Test Bed

Performance testing began with three X9320 couplets as shown in Figure 3. A "couplet" is a term HP uses to describe a pre-configured, fault tolerant pair of HP servers running the Fusion file system software attached to a pre-configured HP disk array.  In this example, three couplets used 240 15K RPM disk drives to present a single 73 TB Fusion file system  to 24 NFS v3 clients.  The clients were attached to an Ethernet LAN using four bonded GigE ports. The X9320 couplet was attached to the LAN using a pair of 10 GigE connections.[2]

Performance Methodology

The HP SNPT performance utility was used to generate workloads and measure performance during each of the tests presented in this report. The SNPT utility, which stands for "simple NAS performance toolkit," was used as an alternative to industry-standard tools including iozone, Iometer, bonnie, dd, and countless other file system performance tools. SNPT was originally developed by IBRIX to overcome the limitations of industry-standard file system benchmarking tools. In particular, it was first developed for testing small file performance over an extremely large file system. Since then, the SNPT utility has been used to measure large file performance as well.

SNPT supports multiple streams running over multiple clients writing over many files with a variety of directory structures. SNPT collects fine grain performance data, down to the millisecond level for each file operation if needed. It collects performance data captured at the OS or network levels as well.

ESG Lab audited the source code of the SNPT utility and verified that it uses standard file system calls to open, write, read, and close files. Scripts used to generate SNPT calls and logs created by the SNPT utility were audited by ESG Lab. Performance data collected independently at the NFS client level to verify the authenticity of performance results reported by SNPT was also examined.

X9320 Large File Throughput Scalability

The first set of tests was designed to measure the performance of X9320 couplets for throughput-intensive write and read workloads. Sequential transfers to large files were used during this phase of testing. The aggregate throughput of the system was recorded as 24 Linux clients accessed a single Fusion file system in parallel. Tests were run on a Fusion file system deployed over one, two, and three couplets. The first two couplets were configured with 96 disk drives each. The third couplet was configured with 48 drives. The results are shown in Figure 4 and Table 1.

What the Numbers Mean

• A single X9320 couplet with the Fusion file system running on two industry-standard HP servers delivered 772 MB/sec of sustained aggregate throughput for large file writes and 1,483 MB/sec (1.448 GB/sec) for large file reads.
• Based on ESG Lab's experience testing traditional and scale-out NAS solutions, 1.448 GB/sec of aggregate throughput is an excellent result for a single file system running on a pair of industry-standard servers.
• Performance scaled in a nearly perfect linear fashion as the solution was expanded from one to two couplets.  The three-couplet results scaled well, but not perfectly, due to the fact that the first two nodes were configured with 96 drives and the third had only 48. ESG Lab is confident that a three-couplet system with each couplet configured with 96 drives would yield approximately three times the performance of a single node test:  2.26 GB/sec for writes and 4.34 GB/sec for reads.

X9300 Large File Write Throughput Scalability

The X9300 is the scale-out NAS gateway version of the X9000 series product line. Using the same hardware and software as the X9320 product line, an X9300 can be configured to work with a variety of disk arrays, including those arrays an organization may already own.

ESG Lab audited the results of throughput testing performed on a large three-couplet X9300 test bed.  This particular test bed was created to validate the write performance of an X9000 Series solution that had been ordered by a customer for a high performance computing (HPC) application with extreme bandwidth requirements.

As shown in Figure 5, the X9300 couplets were tested with six HP StorageWorks MSA disk arrays. The disk arrays, with a total of 432 15K RPM disk drives, were attached to the couplets via 4 Gbps FC connections. Ninety-six clients accessed a single 43 TB Fusion file system through a Gigabit Ethernet interface.  Each of the couplet servers was attached to the LAN via a single 10 Gigabit Ethernet connection.

The Fusion file system was accessed with IBRIX client software. The IBRIX client provides more performance compared to the industry-standard NFS and CIFS clients supported by X9000 Series scale-out NAS solutions. During read operations, the IBRIX client uses a map to direct host file requests to the segment server that holds those files.  During writes, the IBRIX client uses load balancing algorithms to distribute file data evenly over segment servers.  During this test, the IBRIX client was used to direct each client to a specific segment server.  For this particular HPC workload, this provided the best performance.

Each client wrote a single large file.  Aggregate throughput was calculated as the total amount of data written by all clients divided by the elapsed time of the clients. Elapsed time included the time to flush write data from cache to disk and close files. The results are shown in Figure 6.

Performance was measured using the open source collectl and colmux utilities.[3] The performance trace data is shown in Figure 7.  Note that three five-minute test runs correspond to the results shown above in Figure 6.

What the Numbers Mean

• A single X9300 couplet delivered an excellent sustained aggregate write throughput of 896 MB/sec.
• The three-couplet solution delivered 2.58 GB/sec of throughput.
• Write throughput of 2.58 GB/sec is an extremely high rate, beyond what most commercial NAS system can deliver from a single file system, yet is often required to meet the demands of HPC applications.
• Nearly perfect linear scalability was achieved as the system grew from one to three couplets.
• ESG Lab is confident that performance will continue to scale in a nearly linear fashion as more couplets, drives, and network connections are added to the system.

Interpreting Throughput Scalability

Content delivery is a good example of an application that benefits from a storage solution with predictably scalable large file throughput.  Content delivery is used to define a broad category of systems which deliver digital video, image, and audio files over a delivery medium such as a broadcasting network, intranet, or the Internet. The delivery of large files over a content delivery medium requires a storage solution with lots of capacity and aggregate read throughput. To put the aggregate throughput results recorded by ESG Lab into perspective, the maximum aggregate read throughput that was recorded for the three-couplet X9320 system (3.70 GB/sec) was used to calculate the number of streams that could be delivered for a number of well known content types. The results are summarized in Figure 8 and Table 3.

What the Numbers Mean

• The aggregate read throughput of the storage repository used to deliver large file digital media is one of the key factors to consider when architecting and sizing a content delivery system.
• ESG Lab used the maximum aggregate read throughput of a three-node X9320 storage system and the bit rates of well known content types to estimate the maximum number of large files that can be streamed in parallel by the three-node X9320 tested by ESG Lab.
• For example, a bit stream rate of 128 Kbps for MP3 files was used to determine that a three-couplet X9320 has the bandwidth required to support up to 253,312 song downloads running in parallel (3.70 GB/sec divided by 128 Kbps).[4]

X9320 Small File Performance Scalability

General purpose business applications that store data on shared file systems typically create and access lots of small files.  As the number of users sharing a single file system increases, small file performance needs to scale to meet the demands of the business.

High performance and rich media applications also often require good small file performance.  Applications that store lot of large files often create small files to keep track of metadata (data about the files).  A website that stores uploaded photos on a file share is a good example: besides the large photo files, the application often creates small thumbnail images and may also use small files to store metadata (e.g., titles, tags).

The Fusion file system at the heart of an X9000 scale-out NAS solution is designed to perform well for large and small file workloads.

The test bed used to analyze small file performance is shown in Figure 9.  A single X9320 couplet with 96 15K RPM drives was used during this phase of testing.  Each of twelve NFS V3 clients accessed the Fusion file system through a single Gigabit Ethernet connection.  Both of the X9320 couplet servers were attached to the LAN via dual 10 Gigabit Ethernet interfaces.

The HP SNPT performance utility and a script running on each of the 12 NFS clients was used to create as many 4 KB files as possible over the course of an hour. A master script launched synchronized copies of the script on each of the clients and waited for them to finish. The total number of files created was recorded. The results are presented in Figure 10 and Table 4.

What the Numbers Mean

• HP claims that a single NFS stream can create hundreds of files per second and multiple streams can create more than 1,000 files per second on a single X9320 couplet. In this example, multiple NFS streams running in multiple directories on a single X9320 couplet wrote 4 KB files at a sustained rate of 1,425 per second, which is more than adequate for most applications.[5]
• Due to the segmented architecture of the Fusion file system, ESG Lab is confident that adding couplets to an X9000 scale-out NAS solution can be used to meet the small file performance requirements of ANY application.
• For applications requiring even higher levels of small file performance, IBRIX clients can be used instead of NFS to create more than 10,000 files per second on a single X9320 couplet.

ESG Lab Validation Highlights

• ESG Lab confirmed that the segmented Fusion file system running on industry-standard HP hardware performs and scales extremely well when processing large and small files.
• Large file performance scaled in a nearly perfect linear fashion as X9000 series scale-out NAS couplets were added to a single Fusion file system.
• A single X9320 couplet processing large files achieved an aggregate throughput rate of 772 MB/sec for writes and 1,483 MB/sec for reads.
• A three-couplet X9320 system scaled to deliver 2.10 GB/sec of write throughput and 3.70 GB/sec of read throughput.
• An X9300 gateway configuration SAN-attached to six HP MSA disk arrays wrote very large files at a sustained rate of 2.58 GB/sec with the IBRIX client.
• A single X9320 couplet accessing multiple directories with the NFS protocol wrote small 4 KB files at a sustained rate of 1,416 files per second.
• A single X9320 couplet created 5.1 million 4 KB files over the course of an hour.

Issues to Consider

• While the IBRIX client can be used to squeeze the maximum amount of performance out of an X9000 series scale-out NAS solution, it adds deployment and management complexity compared to industry-standard NFS and CIFS, which are built into most operating systems.
• The file allocation algorithms supported by the IBRIX client (e.g., Round Robin) that can be used to squeeze even more performance out of an X9000 series scale-out NAS solution are beyond the scope of this report.     If you'd like to learn more, contact your HP representative.
• The performance results presented in this report are based on benchmarks deployed in a controlled lab environment. Due to the many variables within each production data center, capacity planning and testing in your own environment is needed to see exactly how the X9000 series scale-out NAS will perform with your applications.

The Bigger Truth

In case you haven't heard, the ever-expanding volume of unstructured data is expensive to store and retain and hard to manage.  While it is difficult to make this oft-repeated statement sound fresh and interesting, the simple fact is that file data growth drives continuing customer challenges and storage industry innovation. Today, all you have to say is "file growth cost and complexity" and your audience fills in the back-story automatically, conjuring images of staffing, energy, data access, retention, and protection challenges.

Many vendors, over many years, have tried to solve the problems of predictable cost effective scale-out NAS performance. A number of well-funded companies came and went without gaining any real traction and hundreds of millions of dollars have gone up in smoke.  Scaling has turned out to be a very difficult problem to solve. Most solutions work well for large or small files, but not for both-leading many companies to deploy a scale-out NAS solution alongside a commodity, entry-level NAS solution to deal with metadata. The difficulty is further demonstrated by the fact that storage industry behemoths, while making progress, have not been able to get the job done either by acquisition or internal development.

ESG has been following IBRIX for several years, since well before its acquisition by HP. We've written about it, spoken with customers, and performed several Lab Validations. IBRIX was born in the high-performance computing arena and early animation successes with Disney and Pixar put it on the map. To its credit, while the file system started out highly specialized, IBRIX expanded the product to benefit not just niche markets like medical imaging, scientific research, and animation, but general purpose file serving and file sharing as well. Its segmented architecture makes it work for both large files and large numbers of small files, expanding its usage and target market.

What impressed ESG Lab was not just the technology and virtually unlimited scalability, but IBRIX's commitment to effective partnering, go-to-market, and messaging strategies. IBRIX was smart to keep a narrow focus on scale-out NAS and to concentrate on OEM solutions with Dell and HP, which led to interest from EMC and IBM. Over time, the configuration and GUI were dramatically simplified and IBRIX focused on improving serviceability, usability, and manufacturability. All of this hard work resulted in its acquisition by HP in 2009.

In this most recent Lab Validation, ESG Lab confirmed that the Fusion segmented file system running on industry-standard servers and storage from HP delivers outstanding levels of performance and scalability for small and large files. An excellent large file throughput rate of 1.48 GB/sec can be achieved by a single HP StorageWorks X9320 scaling in a near-linear fashion as couplets are added to a single Fusion file system. Small file performance was outstanding as a single X9320 cluster created 5.1 million 4 KB files in an hour.

Who would have guessed that companies overwhelmed by Word and PowerPoint archives could benefit from the same solution as those burdened by 100-TB annual growth of genome sequencing data? Who knew that a NAS file system developed for high-performance computing could evolve into a graceful, cost-effective scale-out solution with predictable and near-linear performance for small and large files and exotic and everyday applications? The challenges that scale-out NAS solves are much more "everyday" than "lunatic fringe," and the X9000 makes it consumable by almost anyone. If you are facing file system growth and complexity challenges, you should consider the X9000. It's affordable, includes commercial features like snapshots and replication, and lets NFS and CIFS work on the same file system. You can buy a scale-out architecture that will grow with you and meet the needs of your business without interruption. The Fusion segmented file system, combined with HP's servers and storage (not to mention HP's buying power and supply-chain advantage), brings what started as a niche solution to the masses.

Appendix

[1] Source: ESG Research Brief, Scale-Out NAS Adoption & Market Drivers, February 2009.

[2] See the Appendix for more configuration details.

[3] collectl.sourceforge.net

[4] The bit rates for each content type are listed in Table 5, which can be found in the Appendix.

[5] 5,131,623 files divided by 3,600 seconds equals 1,425 files per second.