HP StorageWorks Secure Key Manager: Hardened, Clustered Key Management

Introduction

The HP StorageWorks Secure Key Manager (SKM) is a hardened security appliance providing centralized key management operations for HP StorageWorks Enterprise Storage Library (ESL) E-Series Tape Libraries and HP StorageWorks Enterprise Modular Library (EML) E-Series Tape Libraries.  This ESG Lab report documents the results of hands-on testing of HP SKM solutions with a focus on ease of deployment, transparent integration with existing backup operations, clustered fault tolerance and strong security.

Background

ESG began tracking backup encryption requirements in 2004 as part of a project that resulted in the ESG Research Report, Storage Security Perspectives.  At that time, ESG discovered that only 7 percent of large organizations regularly encrypted their data as it was backed up, while 60 percent claimed that they never performed any type of backup encryption.  This situation has changed substantially in recent years.  Driven by a combination of regulatory compliance requirements, sophisticated security threats and corporate/IT governance initiatives, ESG estimates that around 40% of large organizations employ some form of backup encryption today.  ESG expects that governance, risk management and compliance requirements will accelerate data encryption adoption over the next few years.

As encryption technologies proliferate, Chief Information Security Officers (CISOs) have found they need a new set of technologies, skills and processes to create, distribute, store and manage an increasing number of encryption keys.   Centralized key management services are designed to address these needs.

The need for encryption and the benefits of centralized key management services are well understood within large organizations. In fact, according to ESG Research, 19% of large organizations have already deployed a centralized key management solution and more than half are extremely interested or somewhat interested in doing so.[1]

Any problems around key management could lead to devastating consequences.  Vulnerabilities could lead to a system compromise where an intruder copies or alters encryption keys.  This could result in a variety of crimes: A hacker could use duplicate encryption keys to steal critical data or threaten to destroy the keys unless he or she receives a large extortion payment.  Aside from a criminal attack, sloppy key management could also be extremely damaging.  If encryption keys are lost as a result of a human, process, or technology issue, the result could be ruinous-no encryption keys, no data recovery.

As encryption adoption continues, key management services must offer enterprise-class scale and interoperability without compromising security, availability or operational flexibility.  Security professionals are already considering these requirements, as illustrated by ESG research.  When asked to define critical key management requirements, security professionals identified things like industry standards support, role-based access control, key backup and recovery, and strong auditing and reporting tools (see Figure 2).

Hardened, clustered Secure Key Management appliances (SKM) from HP are designed to meet the requirements shown above while catering to the demands of disparate IT groups.  For the security professional, SKM provides centralized key management services with strong security, high availability, fine-grained access control and secure logging.  For the storage professional, SKM-with an encryption capable tape library-provides secure backup privacy that does not interfere with existing processes and schedules.  Acting as a proxy for IT security and storage professionals, this ESG Lab Validation was designed to examine enterprise requirements and concerns from each of these perspectives.

This report is meant to complement the necessary due diligence performed by seasoned security professionals.   While this report provides a synopsis of ESG Lab Validation testing with a focus on ease of deployment, transparent integration with existing backup operations, clustered fault tolerance and strong security, it does not cover security penetration testing or other formal security validations.[2]

ESG Lab Validation

ESG Lab performed hands-on evaluation and testing of SKM appliances at HP's facility in Fort Collins, Colorado.  Testing was designed to validate the ease of use and security of clustered, hardened SKM appliances working in concert with HP tape libraries supporting encrypting LTO-4 tape drives.

Getting Started

ESG Lab testing began with a typical backup environment before encryption has been implemented (see Figure 3). Acting as a backup administrator, backup software running on a LAN-connected backup server was used to perform non-encrypted backups to LTO-2 tape libraries within an HP ESL tape library.  Backup data on tape cartridges sent offsite for long term archival was stored in clear text form.

Hardened and Secure

HP provides centralized encryption key management services running on one or more hardened SKM appliances as shown in Figure 4.  A hardened appliance is an industry standard term for a server-based solution that, through a variety of hardware and software modifications, is designed for maximum security.

ESG Lab Testing

ESG Lab testing began with an out-of-the-box physical inspection of the SKM appliance. The first and most obvious physical security feature was the double-locked front panel. Two different physical keys were needed to open the front panel. This access is provided to service the system, including a pair of mirrored hard drives. The numbered keys marked Do Not Copy cannot be found at a typical hardware store.  Moving to the back of the appliance, the USB slot had been plugged and neutered.  Tamper evident labels were noted on the top and bottom of the appliance at each potentially vulnerable sheet metal joint.

Realizing that the only detection-free avenue into the appliance was through the external Ethernet interface on the back, ESG Lab powered on the system and audited the ports and services available for configuration and management.   For security purposes, HP locks down "out-of-box" configuration options.  During initial setup, SKM appliances can only be configured using a serial port and CLI.  In other words, an intruder attempting to configure the server through the Ethernet port would be unable to establish a connection using secure or insecure communication protocols such as HTTP, FTP, Telnet, SSL, or SSH..  Root access and availability of a login prompt was also denied-even to HP personnel.

Introducing SKM Enabled Encryption

SKM appliances are designed to work with an existing tape backup environment (see Figure 5).  The backup administrator uses existing backup software and processes as the HP SKM solution and the LTO-4 tape drives transparently export tape cartridges in encrypted cipher text. LTO-4 tape drives within an HP tape library perform the encryption while a cluster of HP SKM appliances provides the keys.

This model provides a scalable and secure platform for the CISO and security administrator.  Management communication between the key management console and a two node key management cluster flows over a secure LAN connection.  AES-256 encryption keys are generated, stored and replicated within the two-node SKM cluster and fed to LTO-4 tape drives for encrypted backup and restore operations as needed.

ESG Lab Testing

A two-node SKM cluster was configured from scratch on a pre-wired test bed and encrypted backup jobs were running in 45 minutes.  At a high level, key management was introduced into an existing backup environment in three phases as shown in Figure 6:

1. Create cluster and exchange certificates to establish a trusted relationship between SKM appliances
2. Exchange certificates to establish a trusted relationship between the SKM cluster and the tape library
3. Configure an encryption policy for an LTO-4 tape drive and run an existing backup job

ESG Lab Testing

The configuration of the first SKM appliance began by powering on the appliance and connecting a PC to the appliance using the provided null modem cable.  A hyper terminal session was used to follow a series of prompt-driven configuration steps. After setting the admin password and the clock, basic IP configuration information was provided, including an HP recommended web interface port number. The balance of the configuration was performed using a web browser over a secure https connection.

A Secure Socket Layer (SSL) communication link was configured between the SKM appliance and the tape library before powering on the second SKM appliance and configuring a cluster. The screen shot shown in Figure 7 shows the interface used as the second SKM appliance joined the cluster.

With a cluster of two SKM appliances defined, the next step was to establish a trusted relationship between the two SKM appliances.    The SKM solution supports certificates created by either an SKM appliance or by a trusted authority.   ESG Lab noted that HP supports certificates created by a number of well known certificate authorities including American Express, Entrust and Equifax. ESG Lab chose to have the SKM appliance provide the certificate.  Installing and exchanging certificates was performed by creating the certificate on one appliance and then copying and pasting that certificate into the management GUI on the second appliance.

A similar process was performed to establish the trusted relationship between the SKM cluster and the HP ESL tape library. A wizard-driven process launched from the tape library console (HP Command View) was used during this phase of the configuration. ESG Lab noted that this wizard was extremely helpful as it provided hints along the way and checked to make sure that necessary prerequisites were in place.

At a high level, the same procedure used to establish a two-way trusted relationship between nodes in the SKM cluster was used to establish a relationship between the SKM cluster and the tape library. That procedure is as follows:

• Generate a certificate on the tape library
• Install that certificate on the SKM cluster
• Generate a certificate on the SKM cluster
• Install that certificate on the tape library

Exchanging certificates creates a trusted relationship so that the tape library can be sure that keys are not provided by an attacker posing as a key manager. And in the opposite direction, the key manager can be sure that keys are not provided to an imposter pretending to be a tape library.  The wizard used to enter a certificate during the ESG Lab validation is shown in Figure 9.

Backup Administrator Transparency

After the security team has configured a trusted relationship between HP SKM appliances and established a policy for how keys will be generated, backup administrators can perform encrypted backup and restore operations with no change to existing processes.

ESG Lab Testing

ESG Lab performed encrypted backup and restore operations with Veritas NetBackup and HP Data Protector.  The highlighted areas in Figure 10 show the NetBackup and HP SKM logs after a successful encrypted backup. From a backup administrator's perspective, there was no perceivable difference in the way the backup job was configured and run.

From a security administrator's perspective, the HP SKM console was used to view the log entries as shown on the right in Figure 10. ESG Lab noted that every key export operation between SKM and the tape library was logged.  Note the long list of letters and numbers after the word "KeyExport." That's a value that the SKM appliances use to track keys internally-it is not the key itself.

The encrypted backup operation was followed by a restore operation using Veritas NetBackup.  A comparison of redirected restore data confirmed that encrypted backups and restores worked flawlessly.

A similar test was performed using HP Data Protector.  HP Data Protector backup and restore jobs were directed at an encrypting partition within the HP ESL tape library.  A partition is a group of one or more tape drives grouped together.  Partitions have historically been used to divide a single tape library into a number of resource pools for a particular application, business unit or backup software package. In this case, partitions were used to create pools of encrypted LT0-4 tape drives and cartridges.

Encrypted HP Data Protector backup and restore jobs completed without error. From a backup administrator's perspective, there was no change in the way that operations were configured, scheduled, executed and monitored.

An attempted restore of an encrypted backup set from a partition that was not configured to receive keys from the HP SKM cluster was tested.  This test was performed to see what it looks like from a backup administrator's perspective if a restore of an encrypted tape were attempted without the key.  This also shows what would happen if someone attempted to restore an encrypted tape that was lost or stolen.

Flexible Key Enrollment

An HP SKM cluster can be configured to exchange keys with an HP ETLA tape library using one of three enrollment policies (note: this part of the configuration process is configured and controlled on the HP tape library and not the SKM) :

• Don't generate keys
• Generate a unique key for every tape cartridge
• Generate a key shared by one or more tape drives in a partition

An encryption policy which shares one key for all of the tape drives and cartridges in a tape library can be configured using the last of the options listed above and specifying all of the partitions within the library.

ESG Lab Testing

ESG Lab configured the SKM to provide a single key for all LTO-4 tape drives and cartridges in a Veritas NetBackup partition (see Figure 11). The LTO-2 tape drive partition was configured for no encryption.  Later, during ESG Lab testing, a more advanced and secure key encryption policy was tested with HP Data Protector as each tape cartridge was assigned a different key.

Clustered Fault Tolerance

Two or more HP SKM appliances can be configured in a cluster for fault tolerance.  Appliances may be located in the same facility connected over a local area network, in different facilities connected over a wide area network or a combination of both.  Keys and policies are replicated automatically for fault tolerance.

ESG Lab Testing

ESG Lab began testing using a two node cluster in the Fort Collins facility.  A third SKM appliance located in Colorado Springs was configured for increased fault tolerance as shown in Figure 13.

A policy change was made using the HP SKM console accessed through one of the appliances in Fort Collins. The appliance at the remote site in Colorado Springs was checked two minutes later. ESG confirmed that keys and policies had been automatically replicated on all appliances in the cluster.

One of the appliances was turned off to simulate a hardware failure.  Encrypted backup and restore operations completed without error. The second appliance was turned off to simulate a site failure in the Fort Collins facility.  After the simulated disaster in Fort Collins, encrypted tapes were restored using a copy of the keys in Colorado Springs.

Role Based Administration and Secure Logging

The HP SKM console can be configured to support a number of administrative accounts at different levels of authority. Security professionals can use this role-based administrative capability to create a separation of roles or a division of roles.

ESG Lab Testing

ESG Lab tested role based administration after configuring three users on the SKM console:

1. A security officer with all rights
2. A security administrator with all rights except the ability to backup keys and certificates

ESG Lab logged on as the security administrator and tried to backup keys. As expected, the request failed with insufficient rights.   Key and authorization policy changes were configured to require multiple credentials from a security officer and a security administrator.  An attempt to delete a key named aes256key presented in the screen shown in Figure 14. The key deletion completed as expected after the second set of credentials had been provided.

ESG Lab Validation Highlights

• ESG Lab has confirmed that the SKM is a hardened appliance with tamper evident packaging and two physical keys on the front panel.
• A pair of pre-wired SKM appliances was configured from scratch and providing keys for encrypting LTO-4 drives alongside legacy LTO-2 drives in an HP ESL tape library in 45 minutes.
• Encrypted backup and restore operations were performed with Veritas NetBackup and HP Data Protector. In each case, there was no change to existing processes and procedures from a backup administrator's perspective.
• An attempted restore of an encrypted LTO-4 drive without a key failed as expected.
• A key enrollment policy-which provided a single key for all of the tapes in a partition-and policy which assigned a unique key for every tape cartridge were tested.
• Clustered fault tolerance was tested by simulating a key manager appliance failure and a simulated primary data center failure.  Encrypted backup and restore operations continued to work with the alternate HP SKM providing the keys.
• Role-based administration was used to grant backup and export key authority to a security officer and deny it for a security administrator.

Issues to Consider

• Scalability.  ESG Research indicates that early adopters of tape encryption tend to start with a few encrypting devices and a simple key policy (e.g., one key per tape library). As organizations grow comfortable with encryption and key management technology, ESG recommends a stronger key allocation policy (e.g., one per tape cartridge) as a best practice for increased depth of security.  This greatly increases the number of keys under management.  For organizations looking to grow into these best practices, a key management solution must not only be as easy and transparent for one key as it is for thousands of keys, it must also be able to scale.  The first version of the HP SKM appliance can scale to handle up to 100,000 keys.  Up to 20 appliances can be configured in a fault tolerant cluster with each appliance adding support for up to five tape libraries-with each library configured to hold up to the maximum number of supported tape drives.
• Support for heterogeneous encrypting devices.  Centralized HP key management is currently limited to tape libraries and LTO-4 tape drives from HP.  Large organizations with a mix of tape libraries from multiple vendors or aggressive plans to add encryption elsewhere in the IT infrastructure should work with HP to understand its product support roadmap and custom integration capabilities.  It should be noted that HP's long term plan is to add support for additional encryption clients to its SKM appliance over time-future integration requirements may overlap with the
  Tags: encryption HP key management SKM tape

  Comments (0)

  New Comment

  Post Comment

  Edit Comment

  Reply to Comment

  • Name
    
  • Email
    
  • Comment
    
  • Leave this field empty