Validation

Co-Author(s): Brian Garrett

Introduction

This report documents performance testing of the NVMe over FC (NVMe/FC) feature of the IBM FlashSystem 9200 array in a VMware and Oracle Database environment. This testing includes a comparison of NVMe/FC and SCSI over FC (SCSI/FC) protocols using Emulex Gen 7 32G Fibre Channel (FC) Host Bus Adapters (HBAs) in the server and storage array along with the Gen 7 Brocade (IBM b-type) storage network switch.

Background

Organizations are deriving more value from their data as analytics workloads become more pervasive. In an ESG research survey, nearly half of respondent organizations told ESG that data is their business to some degree, and 93% of storage decision makers indicated that they had some level of success in deriving incremental revenue from their data.¹

Clearly, data is driving business processes, making speed of access a priority. Organizations are executing transactions and analyzing trends in real time to make better business decisions. Artificial intelligence (AI)- and machine learning (ML)-led innovations are also delivering business insights. These capabilities depend on fast access to data.

The advent of solid-state drives improved storage performance by orders of magnitude, trading hard drive spinning platters for flash. However, SAS and SATA interfaces to solid-state drives have become bottlenecks. Drives that connect via Non-Volatile Memory Express (NVMe) reduce this connectivity bottleneck, enabling lower latency and faster data access directly to flash memory. Delivering higher performance in a smaller footprint also enables infrastructure consolidation for cost efficiency. ESG research reveals that organizations are selecting NVMe not just to increase performance of existing and future applications, but also to optimize costs (see Figure 2).² However, NVMe drives alone cannot solve all the potential I/O bottlenecks. Enabling end-to-end NVMe connectivity—from the host through the switch to the flash array—with NVMe over Fabrics (NVMe-oF) can eliminate the bottlenecks and improve performance.

End-to-end NVMe over FC with IBM FlashSystem Storage, Gen 7 b-type Storage Networking, and Emulex Gen 7 Fibre Channel Server and Storage Adapters Using Oracle in a VMware Environment

The IBM FlashSystem family of storage platforms simplifies storage for hybrid multi-cloud environments. With a unified set of software, tools and APIs, the FlashSystem family of storage solutions addresses the entire range of storage needs from one data platform.³ NVMe-based members of the FlashSystem family are built to deliver fast data access and storage efficiency for secure, highly available environments. End-to-end NVMe is offered via Gen 7 Emulex LPe3500x 32G FC host bus adapters and IBM Gen 7 b-type FC switches with full support for VMware vSphere 7.0. Among their newest features are:

• NVMe over Fabrics (NVMe-oF), which provides end-to-end NVMe with compatible hosts and includes native multi-pathing.
• IBM Gen 7 b-type 32G FC storage network switching, based on Brocade technology, to increase performance and extend bandwidth.
• Emulex Gen 7 32G FC HBAs to improve performance, reliability, and management.
• IBM FlashCore Modules. These are NVMe SSDs with IBM FlashCore innovations for consistent microsecond latency, extreme reliability, and operational/cost efficiencies. Supporting inline performance neutral hardware compression, these drives can compress data by more than 2:1, with a goal of improving storage density and cost per TB. Having only been available previously at the high-end, IBM now ships these drives across the entire IBM FlashSystem family, including entry-level enterprise boxes.

Figure 2 shows the advantages of end-to-end NVMe in virtualized environments using IBM FlashSystem family with IBM b-type 32G storage networking. It provides frictionless data movement throughout the data path, from the host through the switch to the storage and back-end drives. This increased performance lets organizations accelerate the execution of business processes to drive revenue. It also enables more efficient use of storage with AI-powered storage insights; more efficient server usage, including the opportunity to reduce CPU and memory requirements; and future-proof ROI, offering a rich set of data services across all storage systems with IBM Spectrum Virtualize.

With the Emulex FC HBA’s dual protocol support, organizations can simply present NVMe/FC storage via VMware vCenter 7.0 to existing VMware ESXi hosts and migrate virtualized enterprise applications in existing VMware environments from SCSI/FC to NVMe/FC-based storage to improve performance.

VMware vSphere 7.0 I/O Vendor Partner (IOVP) support enables organizations to use their existing VMware environments so that they don’t have to learn or implement new platforms or procedures to optimize mission-critical application performance.

ESG Technical Validation

.ESG audited the results of testing conducted at an IBM data center in the Washington Systems Center (WSC) in Herndon, VA. Typical Oracle application workloads were tested in a VMware ESXi environment with a goal of quantifying the performance acceleration and server efficiency benefits of end-to-end NVMe over Gen 7 Fibre Channel powered by IBM FlashSystem 9200 with IBM b-type storage networking and Emulex Fibre Channel technology.

Oracle Database Performance Testing

A typical database workload was tested with HammerDB, an industry-standard, open source database load testing and benchmarking tool. As shown in Figure 3, Oracle 19c applications were tested in a VMware vSphere 7 environment running on a 56-core Intel Xeon server with an Emulex LPe35002 FC host bus adapter connected through an IBM SAN64B-7 storage network switch to an IBM FlashSystem 9200. The storage network was configured with two 32G Fibre Channel connections between the FlashSystem 9200 and the storage network switch. This configuration was zoned to make it easy to compare the performance of applications that were deployed on different storage networks (e.g., the latest 32G end-to-end NVMe over Fabrics versus SCSI over 32G Fibre Channel).

The ESG validation test bed also demonstrates the future-proof benefits of leveraging existing Fibre Channel infrastructure while enabling the performance and efficiency benefits of NVMe over Fabrics. Two types of NVMe fabrics are emerging: NVMe over RDMA (e.g., RoCE, iWarp, and Infiniband) and NVMe over Fibre Channel (NVMe/FC). In this case, testing was performed with a dual-protocol storage network that supports traditional SCSI/FC and next-generation NVMe/FC. This dual-protocol storage network approach reduces risk in a variety of ways:

• Uses existing Fibre Channel infrastructure (host bus adapters, switches, and storage arrays).
• Utilizes common names services, discovery, zoning, and flow control.
• Provides a cost-effective and future-proof upgrade path.
• Leverages familiar tools and team expertise.

Online Transaction Processing (OLTP) Performance

The ESG validation included a series of performance tests that quantified the benefits associated with the lower latency of IBM Gen 7 b-type storage networking and Emulex Fibre Channel technology and the efficiency of an end-to-end NVMe over Fabrics storage network. Response-time-sensitive workloads benefit the most from the lower latency and improved efficiency of end-to-end NVMe. Examples of multi-user application workloads that benefit from lower I/O latency include e-commerce, software-as-a-service (SaaS), and online transaction processing (OLTP) systems.

An OLTP workload was tested with an Oracle 19c database and HammerDB. The HammerDB workload generator emulates the activity of users in a typical online brokerage firm as they generate trades, perform account inquiries, and execute market research. The Oracle 19c database was configured with an 8KB database block size.⁴ The results are summarized in Figure 4 and Table 1. (Note: This test was designed with a goal of quantifying protocol performance benefits, not the maximum performance capabilities of the FlashSystem 9200.)

Oracle was able to sustain 83% more transactions per minute when running NVME/FC as compared to traditional SCSI/FC. Throughput tells a similar story. As seen in Figure 5, throughput over NVMe/FC was 79% higher than traditional SCSI/FC.

Detailed results are shown in Table 1.

What the Numbers Mean

• Transactions per minute (TPM) is the number of Oracle 19c database transactions recorded by the HammerDB utility.
• New-order transactions per minute (NOPM) is a count of the new warehouse orders that were processed as the system executed four other transactions types (payment, order-status, delivery, and stock-level).
• Higher TPM and NOPM results indicate the database infrastructure can support a larger number of concurrent users who are getting more work done and processing more orders.
• Transaction results captured by the HammerDB utility quantify the benefits of the improved efficiency of Emulex FC technology and end-to-end NVMe:
• 83% improvement in TPM and NOPM after upgrading from 32G SCSI/FC to 32G NVMe/FC
• 79% improvement in throughput after upgrading from 32G SCSI/FC to 32G NVMe/FC

Efficiency Analysis

Next, ESG looked at how FlashSystem 9200, IBM Gen 7 b-type Fibre Channel hardware, Gen 7 Emulex FC HBAs, and the NVMe/FC protocol improves the efficiency of CPU and I/O overhead of application servers. Performance statistics were captured with the Linux iostat utility during the Oracle OLTP performance test with a goal of quantifying the improved server efficiency. The number of transactions per minute per CPU core is shown in Figure 6.

Table 2 shows the detailed results.

What the Numbers Mean

• The CPU core utilization statistics quantify the increased CPU utilization when we tested with NVMe (from 26% with 32G SCSI/FC to 35% with 32G NVMe/FC)
• The small increase (+9%) in system CPU utilization is more than offset by the increased transaction processing (+31%)
• Correlating the application-level transactions per minute with CPU core utilization provides a useful measure of the I/O CPU efficiency benefits of end-to-end NVMe for Oracle applications, in this case up to 31% more TPM per core.
• Kernel CPU utilization reduction is only one of the potential efficiency benefits of an IBM end-to-end NVMe storage infrastructure. Additional cost-savings and return-on-investment benefits include:
• Leveraging existing Fibre Channel infrastructure, processes, tools, expertise, and vendor relationships.
• Reducing capital equipment and licensing costs using a future-proof upgrade path from SCSI/FC to NVMe/FC instead of building a new Ethernet storage network with NVMe over Fabrics (e.g., RoCE).
• Increasing the consolidation and utilization of storage and server infrastructure.
• Lowering the costs of power, cooling, and floor space in the data center.
• Reducing the number of server CPU cores to meet the needs of business-critical applications and the software licensing costs that are tied to CPU core counts (e.g., Oracle database licenses).

Ease of Application Migration with VMware vCenter

For organizations that wish to take advantage of the benefits of NVMe, the ability to integrate NVMe into existing virtualized application environments cannot be taken for granted. When using the Broadcom NVMe over Fibre Channel solution with VMware vSphere 7.0 Support, administrators can easily migrate applications from SCSI/FC to NVMe/FC-based storage to increase performance without investing in additional storage infrastructure or modifying operational processes.

Our test bed is shown in Figure 7. Using VMware vCenter 7.0, we connected an ESXi host containing multiple virtual machines (VMs) to datastores loaded onto the IBM FlashSystem 9200 via an NVMe fabric supported by an IBM SAN64B-7 FC Switch. The ESXi host connected to the fabric via a Broadcom Emulex LP-35002 Gen 7 FC HBA, while the IBM FlashSystem connected via a Broadcom Emulex LP-35004 Gen 7 FC HBA.

The FlashSystem 9200 contained two volumes—a 10GB SCSI/FC disk (containing the “FCP200” datastore) and a 10GB NVMe/FC disk (containing the “NVMe200” datastore).

ESG Testing

ESG began with viewing the VMware ESXi 7.0 host through vCenter 7.0. Using industry-standard tools, we generated an OLTP-like workload on “VM21” (see Figure 4) with a 70/30 read/write ratio, 32 threads, queue depth of two, and 4KB I/O size. We used this simulated workload to characterize performance before and after application migration. VM21 was initially connected to the datastore on “FCP200.” Average application I/O response time was recorded slightly above .001 seconds (1 ms), and average throughput was measured around 250 MB/sec.

To migrate the application, we initiated a storage vMotion by navigating to the “Migrate VM21” window and choosing to change the storage type (see Figure 9). We selected “FCP200” and chose “Browse…” from the drop-down menu. We then selected the “NVMe200” datastore. Once the system automatically performed compatibility checks, the final window popped up and prompted us to complete the migration by clicking “Finish.”

After completing the storage vMotion, ESG observed the change in both the average application I/O response time and MB/s (see Figure 6). We noted a brief dip in performance during the vMotion to NVMe, but performance quickly increased to just under 600 MB/s, more than double what was observed on the SCSI/FC disk. We also observed that the average application I/O response time dropped from 0.001 seconds to approximately 0.0007 seconds (.7 ms).

ESG noted that the storage vMotion was completed with the application running and no downtime. Migrating applications typically requires IT to involve the application owners, VMware administrators, SAN administrators, and possibly end-users to coordinate when to stop, move, and bring the application back online. This approach consumes time and resources and is therefore costly to the business. With Emulex’s concurrent support for SCSI and NVMe along with VMware vSphere 7.0, application migration is easier and non-disruptive.

Managing and Monitoring a Hybrid Infrastructure

In an era of transition with new and existing Fibre Channel speeds and function, there needs to be more intuitive and self-correcting capabilities within the infrastructure to reduce administrative overhead. During the ESG validation of NVMe/FC performance improvements, we observed the use of new management and monitoring capabilities within the IBM b-type FC fabrics along with “built-in” features that reduce administrative tasks.

IBM recently released the IBM SANnav b-type management tool that provides an intuitive view of the fabric for both NVMe/FC and SCSI/FC data flows. In addition, the IBM SANnav tool, along with the Emulex SAN Manager (ESM) management platform, which provides insight to the host IO connectivity and performance characteristics, work together to identify and self-correct SAN fabric congestion issues.

The following ESG observations are being highlighted for their innovative approach to managing and monitoring a hybrid infrastructure.

IBM SANnav

IBM’s autonomous SAN self-healing capability—based on Brocade technology—detects and automatically fixes SAN congestion problems. Autonomous SAN technology is implemented in b-type Fabric OS 9.x software and b-type Gen 7 hardware that are equipped with the latest ASIC technology. Autonomous SAN technology is also implemented in Emulex Gen 7 FC HBAs. A b-type switch that has detected a SAN congestion issue uses the recently approved INCITS/T11 specification updates to include Fabric Performance Impact Notification (FPIN)⁵ to tell Emulex FC HBAs which paths are congested and need to be remediated. IBM b-type switches and Emulex FC HBAs work together to detect, diagnose, and fix SAN congestion problems with FPINs.

Broadcom’s Brocade Storage Networking (BSN) and Emulex Connectivity Division (ECD) business units played a key role in the development of the FPIN specification. The motivation for this effort was simple: Broadcom customers consistently reported that finding and fixing SAN congestion problems was a top SAN management challenge.

ESG tested IBM SANnav’s ability to detect and autonomously recover from SAN congestion by simulating the performance impact of an I/O-intensive workload running on the same SAN as a business-critical data warehouse application. The testing was performed on IBM Gen 7 b-type FC switches, storage arrays, and Emulex Gen 7 FC Host Bus Adapters. A deliberate speed mismatch between the flash array and server HBAs was used to simulate a congestion slowdown for a higher priority data warehouse application after a throughput-intensive workload was started. The industry-standard FIO benchmark utility was used to simulate the large block (1MB) sequential read traffic of the bully and victim workloads.

SAN congestion was flagged as a problem through the IBM SANnav Management Portal as an event that needed investigation. IBM SANnav made it easy to visually investigate the error with a single click to drill down to explore the cause of the alerts. As shown in Figure 12, SANnav made it easy to detect SAN congestion and investigate which HBA ports and switch paths were fighting for bandwidth through the topology view. Once the IBM b-type switches detected and identified the congestion impact, the FPIN was sent to the end device, in this case, the Emulex FC HBA.

Next, we used the Emulex SAN Manager (ESM) interface to visualize the FPIN notifications sent by b-type switches, as shown toward the left in Figure 13. ESG then turned on a moderate level of port congestion management for the lower priority application. This setting, which was previously set to “monitor only,” tells the Emulex FC HBAs to slow down the lower priority “bully” traffic with a goal of eliminating the port congestion issue.

It’s important to note that the autonomous SAN technology detected the SAN congestion issue in IBM b-type switches and used the FPIN protocol to notify Emulex FC HBAs about the congestion and to start using an adaptive congestion management algorithm to fix the problem.

The Bigger Truth

Improving storage performance makes a significant difference in application performance. But improvements to only one component of the data path often reveal bottlenecks in other areas. Moving from disk to flash provided about a 10x performance improvement, and the advent of NVMe drives increased performance again—but then host connectivity became a bottleneck. The data path prevented applications from achieving the full performance potential of NVMe drives. This problem is more pronounced in highly virtualized environments where many applications and workloads share resources across the whole stack.

NVMe/FC with IOVP support for VMware provides end-to-end NVMe that brings another 10x improvement. This speeds connectivity from the host through the switch to the flash array, enabling another performance improvement of an order of magnitude.

This solution—the IBM FlashSystem with IBM Gen 7 b-type 32G storage networking and Emulex 32G FC HBA technology—demonstrates numerous benefits of end-to-end NVMe. Higher performance is the most obvious benefit, but in addition, the solution enables workload consolidation, which leads to lower costs. The NVMe protocol is more efficient than traditional SCSI, and NVMe/FC allows NVMe-native devices to operate on existing Fibre Channel infrastructure, making it simple and cost-effective to add into a current FC environment. And, by optimizing the CPU utilization on the host, it can significantly reduce the processing and memory needs of servers, freeing up resources to run applications, and saving on power, cooling, and licensing costs for applications such as Oracle.

ESG validated that in a virtualized VMware vSphere 7 environment, an IBM FlashSystem 9200 end-to-end 32G NVMe solution with IBM Gen 7 b-type storage networking and Gen 7 Emulex FC HBAs:

• Delivered 83% more Oracle OLTP transactions per minute and 79% higher throughput in a VMware ESXi 7.0 environment compared with 32G SCSI/FC technology.
• Provided up to 31% more performance per CPU core for Oracle databases.
• Can help organizations to perform application migrations from SCSI/FC to NVMe/FC-based storage easily and non-disruptively.
• Have leveraged the FPIN specification to automatically find and fix SAN congestion problems. Detecting and troubleshooting SAN congestion with IBM SANnav Management Portal and displaying FPIN notifications with Emulex SAN Manager (ESM) was intuitive and easy.

IT must always make a tradeoff between performance and cost, even with technology innovations like NVMe. The solution we tested diminishes these tradeoffs, delivering high performance and low latency with frictionless data movement from the host through the network, all the way to back-end storage.

While the IBM and Broadcom solution does not require organizations to make huge changes to either the existing SAN infrastructure or current IT operating procedures, it is always a good idea to gauge exactly what changes may need to occur to achieve the benefits of simplicity in IT daily activities and increased performance simultaneously.

If your organization is looking to accelerate access to data throughout your storage infrastructure and capitalize on its potential to increase value and revenue, then ESG believes that you should consider the potential advantages of leveraging IBM and Broadcom’s 32G NVMe/FC solution with VMware vSphere 7.0 support in your FC SANs to maximize the efficiency and performance of your organization’s virtualized mission- and business-critical applications, including Oracle databases.

Appendix

The configuration of the test bed that was used for the ESG Validation testing is summarized in the table below.