Security Data Explosion Is Driving Big Data Security Analytics

In my last blog, I mentioned that 44% of security professionals working at enterprise organizations (i.e., more than 1,000 employees) believe that security data collection and analysis could be considered “big data” today while another 44% believe that security data collection will qualify as “big data” within 2 years.

What makes security data “big data?“ One place to start is simply security data volume. According to recent ESG Research, 47% of enterprises collect more than 6TB of security data on a monthly basis to support cybersecurity analytics. Security data volumes are growing as well: 43% of enterprises are collecting “substantially more” security data than they did 2 years ago while another 43% of enterprises are collecting “somewhat more” security data than they did 2 years ago.

The volume and growth of security data collection is driven by 3 primary factors:

  1. The increasingly dangerous threat landscape. Stealthy malware and targeted attacks are designed to circumvent traditional security controls and appear like normal behavior to legacy security analytics systems. This is driving the need to collect and analyze more security data.
  2. The need for greater visibility up and down the stack. When large organizations first connected to the Internet, security data collection was really focused on the network perimeter and two primary security devices – firewalls and IDS. Fast forward to 2012 and most malware attacks target applications rather than networks. As a result, CISOs want to monitor the entire technology stack – networks, operating systems, applications, databases, and storage devices.
  3. New IT initiatives. IT technologies like cloud computing, host virtualization, and mobile computing introduce new technologies, threats, and vulnerabilities. Given the relative technology immaturity here, enterprises believe it is worthwhile to collect and analyze security data with greater frequency and diligence.

All three of these trends are also driving new sources of data collection and security analysis. Four years ago, large organizations did IP packet capture on a sampling or as-needed basis. Now many firms want to collect and analyze all IP packets across the enterprise network. In the past, endpoint forensics was done after a security incident occurred. Now organizations want to capture endpoint forensic data proactively.

So data collection for security analytics continues to grow from numerous sources for a multitude of analysis activities. This trend is driving new requirements for security analytics scale, intelligence, automation, and ease-of-use. ESG expects massive innovation in this area over the next few years. See our complete research report on the emerging intersection between big data and security analytics.

Topics: Storage Cybersecurity Networking Enterprise Mobility