Live from New York, if just a bit Off Broadway, were the O'Reilly AI conference and Strata & Hadoop World shows last week. These events were fascinating as character studies of overlapping technology stars. AI is now the fresh faced celebrity, if a bit immature, raring to get a big break, while Hadoop has become recognized as always delivering a solid performance, but may be fading a bit in glamour.
Starting with the AI conference then. This was a stunningly cool event. The stuff being done with machine learning, deep learning, neural networks, etc., is incredible. A well-trained AI can accurately process rough inputs, analyze to build complex models, and predict all manner of outcomes. In an instant. Except it's all still rather hard to use and could benefit from some better packaging and integration with other IT technologies. If there is a shortage of general data scientists, I wonder how many people can really master machine learning as it's presented today. Can we say "user interface?" That aside, we are on the cusp of some truly remarkable things. Though AI may still struggle to do some tasks my 10-year old tackles effortlessly, at the same time it can process phenomenally more info and find hidden patterns. And it's fascinating how an AI thinks, perceiving the world in fundamentally different ways than us meat. This space is ripe for massive growth, if we can get the tech accessible to more than a handful of companies, universities, and government labs.
Now to Strata. Hadoop risks being conceptually reduced to a storage layer, which is fine and economically powerful and enables access to unbounded data lakes, but all the action is happening above this foundation. I applaud all the efforts to make it governable, secure, catalogued, and generally well managed. This is necessary stuff for enterprises, and everyone else, too. I saw many solutions for improved data integration, hardened open source, more stable pipelines, more flexible resource utilization, even a "data ops" dashboard that looks like something you'd find in a NOC. The problem is that it's all a bit dull. Managing physical and data infrastructure kind of stuff. Whither the actual analytics that make big data into insights? The transformation of hella-bytes into profundities isn't apparently much of a thing anymore. To put it another way, operational quality is the new sexy.
I'm not hating on Hadoop. This is an important phase of a maturing industry. It just reinforces the point that the new jazz is in AI and machine learning, the new stars in the biz. If you want to hear more about specific players and their plays, drop me a line....
Footnote: it was good to see the diversity in the audience and on stage. People of all types were appreciated for their ideas, full stop. No hard data to prove this, but it was noticeable to me.