Big Data Security Challenges

According to ESG Research, 47% of enterprise organizations collect 6TB of security data or more on a monthly basis to support their cybersecurity analysis requirements.  Furthermore, 43% of enterprise organizations collect “substantially more” security data then they did 2 years ago while an additional 43% of enterprise organizations collect “somewhat more” security data then they did 2 years ago.

Just what types of data are they collecting? Everything. User activities, firewall logs, asset data, vulnerability scans, DNS logs, etc. Most enterprises aren’t collecting, storing, and analyzing large volumes of network packets (i.e., Full-packet capture or PCAP) today but they will increasingly do so in the future. Once this happens, security data volume collection will take another quantum leap.

If this activity doesn’t signal the need for big data security analytics then nothing does. Nevertheless, CISOs’ need go beyond dumping a bunch of unstructured data in a Hadoop cluster. 

So what’s required? To find out, ESG recently surveyed 257 security professionals working at North American-based enterprise organizations (i.e., more than 1,000 employees) and asked them a series of questions about security data collection, processing, and analysis. As part of this project, security professionals were asked to identify specific difficulties around security data collection and analysis. The top 2 problems revealed were:

• 62% of enterprise organizations have “significant difficulties “ or “some difficulties” with security data visualization
• 53% of enterprise organizations have “significant difficulties “ or “some difficulties” with security data analysis

Existing security analytics tools tend to catch obvious attacks or provide a 50,000 foot perspective of the network. Security analysts and CISOs need an atomic view of packets, protocols, payloads, and behavior over various timeframes—seconds, minutes, days, weeks, months, etc. They need visualization tools that provide context of what’s normal, what’s anomalous, and what’s extremely dangerous. Finally, they need security technology to do more of the heavy lifting analysis. Forget big data technology buzz words like NoSQL, Cassandra, and MapReduce. CISOs need data analysis and visualization not just a bigger file system for unstructured data. 

Lock down your network all you can but you will still need continuous monitoring and big data tools to analyze and visualize the billions of IT activities that happen each day to attain situational awareness and make tactical security adjustments. 

This is the near future of enterprise security analytics. The vendor that provides big data backend technologies along with superior analytics intelligence and visualization will win big. 

*All views and opinions expressed in ESG blog posts are intended to be those of the post's author and do not necessarily reflect the views of Enterprise Strategy Group, Inc., or its clients. ESG bloggers do not and will not engage in any form of paid-for blogging. Click to see our complete Disclosure Policy.