Big Data Security Analytics or Big Data IT Analytics?

According to ESG research, 44% of enterprise organizations believe that their security data collection, processing, and analysis qualifies as “big data” today while another 44% believe that their security data collection, processing and analysis will qualify as “big data” within the next two years.  Given this requirement and market opportunity, it is not surprise that IBM and RSA announced Hadoop-based big data security analytics platforms this week.

Yup, big data security analytics will become increasingly pervasive over the next few years but what about the rest of IT?  Certainly IT operations teams could benefit from big data analytics for performance tuning, capacity planning, and SLA management.  In fact, vendors like LogRhythm, Splunk, and Sumo Logic either market directly to IT operations or find that IT operations jumps onboard quickly once the security team deploys their analytics tools.

Okay so this begs an obvious question:  Should large organizations focus their big data technology investments on security or leverage them across IT?  ESG posed this question to 225 security professionals working at North American-based enterprise organizations (i.e., more than 1,000 users).  Interestingly, 61% of organizations say that a big data project would “encompass many aspects of IT including security,” while 38% say that a security-focused big data project would be implemented independently of any other IT projects/architectures.” 

I understand this sentiment; the majority of CIOs want to create a common IT big data architecture rather than purchase, deploy, and operate a bunch of IT big data silos.  Makes sense but this could easily turn into a multi-year IT mega-project if organizations aren’t careful.  What’s more, CISOs have a pressing need for big data security analytics because traditional security controls are unable to detect and prevent advanced malware and sophisticated hacking techniques.

There is a way for CIOs/CISOs to get their big data cake and eat it too.  ESG suggests that enterprise organizations:

• Build common infrastructure for data collection, processing, and storage.  Do your research and figure out what types of IT plumbing you will need as a first step.  This will depend upon what types of data you want to collect, the volume of data, your skill sets, your network, and your objectives.  This should help you design a distributed big data architecture and sort through the myriad of big data technologies like Hadoop, Hive, MapReduce, NoSQL, Pig, and Zookeeper. 
• Separate big data infrastructure from analytics.  Yes, there are a bunch of query and visualization tools that can be used for heterogeneous data analysis, but this is likely where market specialists will start to differentiate themselves with fixed-function analysis, algorithms, and visualization tools. 

To be clear, these recommendations don’t necessarily mean two separate projects.  As I mentioned, the new IBM and RSA offerings are based upon Hadoop so the infrastructure could easily extend beyond security analytics alone.

Big data security analytics isn’t merely industry hype – CISOs really need to collect and analyze more data for continuous monitoring, situational awareness, and tactical security controls adjustments.  Massive plans for big data across IT shouldn’t delay enterprises from addressing this pressing requirement. 

Read the ESG Research Report, The Emerging Intersection Between Big Data and Security Analytics for more information.

*All views and opinions expressed in ESG blog posts are intended to be those of the post's author and do not necessarily reflect the views of Enterprise Strategy Group, Inc., or its clients. ESG bloggers do not and will not engage in any form of paid-for blogging. Click to see our complete Disclosure Policy.