Validation

Introduction

This report documents ESG’s validation of Dell Technologies performance testing of the VxRail Hyperconverged Infrastructure (HCI) configured with Intel Optane persistent memory (PMem). The high performance that PMem adds enables organizations to bring mission-critical, high-performance workloads to VxRail, increasing the opportunities to gain from VxRail’s management, efficiency, and flexibility benefits.

Background

Across all industries, organizations have recognized the benefits of HCI as they modernize data centers. The ability to consolidate multiple virtual workloads on a single platform brings agility and scalability to match the speed of business today. Instead of individually managed silos of infrastructure (compute, storage, and network) that IT must separately evaluate, procure, test, deploy, tune, and manage, HCI solutions like Dell EMC VxRail provide a centrally managed, single solution with software-defined resources that are easy to deploy, scale, and manage. These features also endow HCI solutions with a lower cost of ownership advantage. It is no surprise, then, that ESG research shows a considerable increase in HCI usage over the past few years. While in 2017 22% of respondents were employing HCI on-premises, the responses in 2019 had more than doubled to 46% (see Figure 1).¹ Organizations see HCI as a key contributor to digital transformation and are using it to power mission-critical production applications.

However, when multiple applications compete for resources, an “I/O blender effect” can occur, degrading performance. As a result, organizations have historically avoided placing applications that need consistent high performance on HCI, such as real-time analytics, online transaction processing (OLTP), content delivery, and high-performance computing (HPC). In order for these applications to run on HCI, a hyperconverged solution must offer high performance; this would expand the opportunity to gain from HCI’s efficiency, flexibility, and scalability.

Solution: Dell EMC VxRail Hyperconverged Infrastructure with Intel Optane PMem

VxRail is an HCI solution that delivers agility, scalability, simplicity, and operational efficiency. Jointly developed by Dell EMC, VMware, and Intel, VxRail now offers configurations using Intel Optane persistent memory, increasing both performance and memory capacity while maintaining data persistence, at a significantly lower cost than DRAM.

Dell EMC VxRail

VxRail provides turnkey infrastructure for a variety of modern, virtualized applications including databases, virtual desktop infrastructure (VDI), data analytics, and artificial intelligence (AI) at the core, edge, or cloud. It offers the fastest, simplest VMware environment, including management using the familiar vCenter interface. VxRail includes virtualized compute, networking, and storage; VxRail’s foundation is Dell PowerEdge servers with Second Generation Intel Xeon Scalable Processors and VxRail HCI System Software.

Three VxRail models (E560F/N, P570F/N, and P580N) now offer configurations that include Intel Optane PMem, adding the performance necessary to support performance-intensive workloads such as OLTP, real-time analytics, in-memory databases, and HPC. Intel Optane PMem is available for VxRail in 128GB, 256GB, or 512GB configurations up to 12TiB. Two operational modes can be utilized: Memory Mode, which provides additional volatile memory that has a clear cost advantage over DRAM, and App Direct Mode, which can be used as low-latency, persistent storage.

Using modular building blocks, organizations can non-disruptively scale capacity and performance from three to 64 nodes in a single cluster. VxRail includes the VxRail HCI System Software that keeps infrastructure in a continuously validated state. It provides intelligent lifecycle management, including automated, non-disruptive upgrades, patches, node additions, and node retirement. It also provides artificial intelligence- and machine learning (AI/ML)-driven health reporting, designed to help customers maintain optimal performance and availability. Options include:

• 4-112 Second Generation Intel Xeon Scalable Processor cores; single, dual, or quad socket.
• 128GB, 256GB, or 512GB Intel Optane PMem modules, up to 12TiB.
• NVMe, SAS, or SATA drives, hybrid or all flash, up to 7.68 TB.
• Intel Optane SSDs.

Intel Optane Persistent Memory

At a high level, Intel’s Optane technology allows memory cells to be individually addressed in a dense, transistor-less, stackable design. Intel offers Intel Optane SSD and persistent memory, both providing faster storage than traditional NAND SSD. Intel Optane PMem sits between memory and storage, creating an additional performance tier between DRAM and SSD that can be non-volatile. Supported operating systems include MS Windows Server 2019, VMware ESXi, RHEL, Ubuntu, Oracle, CentOS, and SLES Linux.

Intel Optane PMem modules are installed in the memory channel in the DDR4 DRAM form factor and can be combined with conventional DDR4 DIMMs. Each socket supports one to six modules, for up to 3 TiB per socket, double the maximum density of current DRAM DIMMs. Intel Optane PMem DIMMs also include 256-AES hardware encryption. Different from traditional DRAM, Intel Optane PMem offers both data persistence (so that data is retained during power loss or restart), and high density, with a lower cost per GB. It offers high performance, low latency, and better endurance for write-intensive workloads. By delivering more performance in a small footprint, Intel Optane PMem can improve total cost of ownership, by enabling higher workload consolidation rates and minimizing software licensing, node deployment, and power/cooling/data center costs.

Intel Optane PMem supports two addressing modes:

Memory Mode. This mode functions as expanded, volatile system memory, enabling organizations to increase memory capacity in a denser footprint and at a lower cost than DRAM. It requires no changes to application software. In this mode, DRAM serves as a cache for the most frequently accessed data, while PMem provides expanded main memory capacity. Cache management is done by the Intel Xeon Scalable Processor memory controller. When data is requested, DRAM cache is checked first; if data is not there, it is read from Intel Optane PMem.

App Direct Mode. App Direct Mode provides data persistence during a VM failure, eliminating the need to recover data during application restart. Applications are responsible for protecting data from a host failure using replication and other methods. In App Direct Mode, both Intel Optane PMem and DRAM are independent memory resources under direct load/store control of the application. It can be consumed as byte-addressable memory or as block storage and can be directly accessed by PMem-aware applications.

Because App Direct Mode expands the capacity of persistent memory, applications and operating systems recognize two types of direct-load memory—DRAM and PMem—and can send writes to the preferred one. Both are counted as total platform memory. Data that needs the lowest latency and does not require persistence can be sent to DRAM, while data that needs persistence and can support slightly higher latency can be directed to PMem. Applications that are optimized for Intel Optane PMem gain faster memory access with lower software I/O overhead. Byte-addressable App Direct Mode requires support for a persistent, memory-aware file system; supporting operating systems include vSphere 6.7U3, Windows Server 2019 NTFS DAX, Red Hat Enterprise Linux 7.6 or later, SuSe Enterprise Linux 12 SP4 or later, and Ubuntu 18.04 or later. Database support includes SAP HANA 2 SPS 4+, Redis, Aerospike, SQL Server 2019, and Apache Cassandra.

Configuring Intel Optane PMem

Intel Optane PMem is automatically recognized by the OS, and administrators configure the PMem operating mode using BIOS memory management tools. It can be designated as either App Direct Mode or Memory Mode. In App Direct Mode, regions (defined as groups of PMem modules) can be configured to operate as interleaved (in which all PMem modules are seen as a single memory space with I/O striped across them) or non-interleaved (in which each PMem module is seen as a separate entity, with I/O filling one module before going to the next).

ESG Technical Validation

ESG audited results of performance testing by the VxRail team using Intel Optane PMem in both Memory Mode and App Direct Mode.

Memory Mode Testing

Using VxRail with Intel Optane PMem in Memory Mode increases the volatile system memory capacity at a lower cost than DRAM, enabling greater virtual machine density without any changes to existing applications. ESG audited Memory Mode testing that was designed to evaluate the in-memory database performance difference between DRAM only and DRAM + PMem. We also reviewed VxRail pricing as reported by Dell.

The test bed (a configuration which is available to customers) used a single VxRail E560N with VxRail 4.7.510 and vSAN 6.7U3. Testing used 8 Redis in-memory database VMs; each VM included 2 vCPUs and 64GB of RAM allocated. The Redis-developed memtier_benchmark was used for load generation and was executed on a dedicated host with eight connections and two threads; tests were run six times, and the average results reported. Three workload profiles were tested: Set (100% write), Get (100% read), and Mixed (66% read, 33% write). Each workload was run in two VxRail configurations: 1) with 1.5TB DRAM only; 2) with 384GB DRAM + 1.5TB PMem.

As Figure 6 shows, testing revealed that write and read workloads were minimally affected by PMem when compared to DRAM only, while mixed workloads were slightly more impacted. When DRAM was combined with PMem (the navy-blue bars in Figure 6), write performance dropped only 6.4% with 2.6% cache misses, and read performance dropped only 6.2% with 3.6% cache misses. Mixed workload performance dropped only 10% with 2.3% cache misses. Note that the cache hits would be handled by DRAM, and the misses by PMem. All workloads performed with less than 1.5ms latency.

As previously mentioned, the key benefit of Memory Mode is to deliver a larger memory footprint more affordably than a DRAM-only system. In this testing, since the DRAM cache hit rate was greater than 96%, Redis delivered high performance along with a lower total cost of ownership. Using Dell’s reported street pricing (which includes typical discounts) for the VxRail models used in the Redis testing, the cost of a VxRail E560N with 384GB DRAM + 1.5TB PMem was 14% lower than the cost of a VxRail E560N with 1.5 TB DRAM.

ESG also reviewed testing with traditional RDBMS that demonstrated the ability to increase VM consolidation by 25%, at similar price points, using a combination of DRAM and PMem with minimal performance impact.

App Direct Mode Performance and Scalability Testing

Testing was designed to evaluate IOPS, throughput, scalability, and latency. First, we viewed a baseline of VxRail performance with all-flash disk groups using NVMe cache and SSDs. This was done using four VxRail P570Fs with two disk groups, each having 800GB NVMe cache and two 7.68TB SSD. Each VxRail host attained 1) 152K RR 4K IOPS and 40K RW 4K IOPS; and 2) 2.4 GB/s RR 64K throughput and 1.6 GB/s RW 64K throughput. These demonstrate impressive performance for an all-flash system running many workloads.

Next, we viewed VxRail testing with Intel Optane PMem. The first set of tests was conducted using a four-node VxRail model P570F configured with dual Intel Xeon Gold 6254 CPUs, 800GB NVMe cache, and 23.8TB SAS SSD. We evaluated the performance of the system in two ways: 1) using a single 128GB PMem module, and 2) scaling the number of 128GB PMem modules from one to 12. PMem was configured as byte-addressable NVDIMM using a PMem-aware DAX filesystem. The FIO test harness was used to generate workloads using one FIO VM per socket, with one disk per VM, and 80% working set size. I/O characterizations included 4KB and 64KB, 100% random read and 100% random write. It should be noted that VxRail created and assigned non-uniform memory access (NUMA) nodes to tie together specific CPU and memory; in this testing, we confined each FIO NVDIMM to a single NUMA node. This ensures that VMs access PMem on a local socket for fast access.²

We tested a single PMem module to find the “four corners” raw performance, a common assessment of basic horsepower of the system. As shown in Figure 9, the system supported high performance for both small block IOPS (more than 1.3 million read IOPS and more than 400K write IOPS) and larger block throughput (more than 5 GB/s for reads and more than 1.5 GB/s for writes).

The ability to support more than a million IOPS and multiple GB/s throughput with a single PMem module demonstrates that PMem can add significant performance to an HCI deployment at low cost.

Next, we looked at the ability of the system to scale performance as PMem capacity increased. Tested configurations included one, two, four, eight, and twelve 128GB PMem modules, configured with interleaving. With multiple PMem modules configured on both sockets, VxRail created and assigned four NUMA nodes per socket. Figures 10 and 11 show near-linear scalability of random read and write IOPS and throughput as the amount of PMem capacity increases.

Figure 12 shows the “four corners” performance using 12 x 128GB PMem modules.

Simulated Application Testing

Next, ESG reviewed testing designed to simulate real-world workloads using vPMem disk. In this method, the vDisk was created on the local PMem datastore of the host and associated with a storage policy. This mode is completely transparent to applications and operating systems, requiring no changes.

We compared results of VxRail configured with vPMem disks using a virtual NVMe controller to VxRail using vSAN configured with all-flash storage. The PMem tests were run using four VxRail P570F nodes configured with Intel Xeon Gold 6254 CPUs, 12 x 32GB DRAM, and 12 x 128GB PMem; it supported two VMs per host with 8 x 25GB vPMem disk per VM. The comparison environment was an all-NVMe vSAN environment with four quad-socket VxRail P580Ns with two VMs per host, each configured with 8 x 200GB vmdk. The VDBench utility was used to generate a workload.

First, we looked at an OLTP workload consisting of 4K random I/O, 70% read/30% write. This simulates typical transactions such as retail sales, web commerce, ATM transactions, and similar tasks that involve inserting, deleting, and updating small amounts of data in a database. After an initial test run to determine the maximum IOPS and throughput, the test was run in 5-minute intervals starting at 10% of max IOPS and incrementing by 10% with each new run.

In this test, the PMem-configured VxRail delivered more than 1.8M IOPS, 5.3x more than the all-flash VxRail system. In addition, response time for the PMem VxRail remained under 0.2 ms for most of the test, while the all-flash system response time began at 0.37 ms and quickly slowed to 1.28 ms. At 10% of max IOPS, vPMem was driving 184K+ IOPS with 0.04 ms response time; vSAN achieved comparable IOPS at 40% of max, and response time was 0.57ms. The vPMem configuration was 93% faster at the beginning and averaged 72% faster response time throughout the test (see Figure 14).

Finally, we looked at results of a TPC-C-like benchmark⁴ that demonstrates database log acceleration, comparing VxRail configured with vPMem disk to a VxRail all-flash vSAN. For the vSAN configuration, all data and log files resided on vSAN; the PMem configuration stored only the 64GB log file on PMem. The test bed was four VxRail P570Fs, each with Intel Xeon Gold 6254 CPUs, 392 GB of RAM, 25Gb Mellanox networking, 745GB SSD cache, and 3 x 1.75GB SSD disks. The PMem system included 2 x 128GB modules with RHEL VMs on VMware ESXi.

MariaDB was used to create a MySQL database configured to load data into 100 data warehouses with 32 concurrent connections and an initial database size of 20 GB. The test simulated multiple transaction types and concurrent transactions of different types and complexity. It was run for one hour, tracking the number of transactions per minute. PMem VMs were built with the XFS filesystem; all VMs included 24 vCPUs and 64GB RAM.

As Figure 15 shows, the vPMem configuration provided 45% more transactions per minute. This demonstrates that for some use cases, only a small amount of PMem capacity in a host can add significant value.

The Bigger Truth

In recent years, organizations have focused on digital transformation, working hard to make optimal use of digital assets to better understand customer needs, inform strategy, and improve operations. According to ESG research, the most common objectives for digital transformation are to become more operationally efficient, and to provide better and more differentiated customer experience.⁵

To accomplish these goals, IT organizations employ a wide range of applications with varying needs. Many have realized that consolidating diverse applications on HCI can greatly improve efficiency of infrastructure and management, but only if performance demands are met. Application performance requirements have prevented many organizations from using HCI for workloads such as real-time analytics, OLTP, and high-performance computing.

The Dell EMC VxRail HCI solution now offers the option of adding Intel Optane persistent memory to increase performance without high cost. Memory-constrained virtual infrastructure often also underutilizes the compute capacity that VxRail offers. Intel Optane PMem in Memory Mode provides a new memory caching tier that can reduce I/O bottlenecks, speeding application performance and data recovery more affordably. As a new performance tier between DRAM and SSD, Intel Optane PMem offers three key benefits: larger capacity than DRAM at a fraction of the cost, high-performance storage, and data persistence after power-off.

ESG validated cost, performance, and scalability advantages for VxRail with Intel Optane PMem, including:

• In Memory Mode:
• 33% lower memory cost and 14% lower cost per VxRail node while maintaining at least 90% performance.
• 25% increase in VM consolidation, at similar price, with minimal performance impact.
• In App Direct Mode:
• More than 1M IOPS and 5GB/s throughput with a single 128GB PMem module.
• Near-linear IOPS and throughput scalability as PMem capacity increases.
• 5.3x more IOPS and 93% faster OLTP response time than VxRail with all-flash vSAN.
• 45% faster database log acceleration than all-flash vSAN versus a single PMem module.

The results presented in this report are based on testing in a controlled environment. Due to the many variables in each production data center, it is important to perform planning and testing in your own environment to validate the viability and efficacy of any solution.

While most business applications don’t need millions of IOPS and GB/s of throughput individually, when they are consolidated, the performance is divided up among all applications. VxRail with Intel Optane PMem enables organizations to consolidate more applications, even those with high-performance needs, and gain deployment, management efficiency, and cost benefits. It also expands the range of VxRail-applicable workloads at the core, edge, and cloud. In addition, the ability to start with a small amount of Intel Optane PMem and grow with scalable performance fits with the HCI profile of flexibility and agility.

Most important, with VxRail with Intel Optane PMem, organizations gain more transactions, faster insight, and better business decisions while also gaining infrastructure and management cost efficiency. Enterprise-class performance that once required expensive, dedicated storage solutions is now available to any organization. For any organization looking to consolidate applications for cost efficiency while delivering high application performance, ESG recommends evaluating Dell EMC VxRail with Intel Optane PMem.

Appendix