I have no crystal ball, so I can’t peer into the swirling fog and read the future. This is a constant disappointment, but explains my fantasy football results. What I do have is better though: The crowd-sourced wisdom of many, many leaders in the world of big data. Based on all the candid conversations with customers and vendors, I can predict with some confidence the following trends for 2015…
SPOILER ALERT for those who prefer to preserve the surprises.
1. More real-time analytics. Ok, an easy one to start. An unbelievable number of folks tell me they’re doing this today. Yet when you dig into their activities, it turns out their idea of real-time is once a week, or after the data loads, or every five minutes. So this may be more aspirational than factual, but the interest will drive the industry. Spark, in-memory, and other “go fast” technologies will become easier to implement and more common in production. Startups such as Databricks, MemSQL, and Interana are pushing to find ways to find the answers, like right now, just as the big boys (and big girls) at SAP, Microsoft, Oracle, and IBM have built their performance capabilities. No one likes waiting.
2. More machine learning. I love the concept of pattern recognition. Take a pile of seemingly random input, turn it 90 degrees, squint, and suddenly it all becomes clear and obvious. Humans are remarkably good at this, but also tend to fall for confirmation bias and sensory overload. Machines are better at this stuff, particularly as you increase the variables and scope. Expect organizations to invest in big data tools that spot the most subtle connections and present them for review. Machine learning sounds difficult, but it’s getting more approachable, thanks to groups like Skytree, Microsoft Azure, and IBM’s Watson.
3. More diversity of deployment models. Big data means Hadoop, and Hadoop means cheap-as-chips commodity servers right? Well, sometimes. But we’re now seeing a lot of variation on the theme. Many are choosing big data appliances engineered systems, like those offered by Oracle, Teradata, Dell, and Cray (yes, that Cray). Others are looking for cloud to auto-magically give them nigh infinite resources on demand. This market is being satisfied by the likes of Amazon Web Services, Microsoft Azure, Google, and boutique players like Altiscale and Treasure Data. If you'd like a private cloud, you must check out BlueData. Why fuss with the hardware integration, when you could be fussing with the data?
Wait, do you see that? It’s the new year dawning. Perhaps I’ve said enough for today. Join us next time for more prognostications.