Big Data Security Is Inevitable

There's been a fair amount of discussion about the fact that security analytics is becoming a big data problem. I participated on a big data security panel at RSA and I believe there were a few others on this topic as well.

Based on some recent ESG Research, I'm now convinced that it's no longer a question of whether security analytics are classified as big data but merely when this trend will become mainstream. While I've analyzed a lot of data that leads me to this conclusion, there are two enterprise information security issues driving the big data trend:

  1. Constant changes to the IT infrastructure. There is an increasing amount of network variability happening all the time: Virtual machine provisioning/movement, mobile devices, DNS look-ups, traffic patterns, application objects, etc. While leading enterprises harden systems and monitor network traffic, most have little understanding about the security implications of these cumulative and ever-changing patterns. Preventing bad things from happening depends upon an omniscient purview of assets, configurations, threats, and vulnerabilities but many organizations are "flying blind."
  2. Keeping up with "normal" behavior in light of new threats. Closely related to #1, most large organizations now realize that sound risk management processes and tight security controls alone aren't enough. In today's threat landscape, you have to assume you'll be attacked and take the necessary steps to bolster your incident detection capabilities. One of the ramifications here is that enterprises need a much more thorough understanding of "normal" behavior as a baseline for judging anomalous activity--at the network, system, application, and user level.

Each of these requirements means collecting, normalizing, storing, and analyzing more data, more often, for longer periods of time.

If you think that enterprises recognize these trends, boning up on Hadoop, Cassandra, and NoSQL, and hiring data scientists to tag along with security analysts, think again. There's a growing security skills shortage that will preclude these activities before they even start. In my mind, big data security analytics are entirely dependent upon:

  • Leading security software vendors that can build scalable intelligent automated systems. The key here is creating security software that is capable of a lot of the background heavy lifting tasks -- event correlation, incident detection, automated response, etc. IBM, McAfee, and RSA seem best positioned for this. I also consider HP, Symantec, and Tibco as wild cards. I'm not at all convinced that any startup has the resources, software chops, and enterprise account management skills to play here.
  • Managed Security Service Providers. Yes, large financial institutions and Federal agencies will invest in big data skills and technology but I believe this will also be a tipping point where many enterprises punt. Some will wash their hands of the whole enchilada while others will want support services. Good news for BT, Dell, Symantec, Unisys, and Verizon.
Topics: Cybersecurity Data Platforms, Analytics, & AI