"Big data, huh, what is it good for?" at the Hadoop Summit

hadoop summitThe mood of this week’s Hadoop Summit has felt wonderfully diverse. There is a cognitive disconnect between the incremental progress of dot release feature sets and the revolutionary new business and societal applications of the technology.

In the same keynote session the topics can swerve from optimizing cluster utilization to optimizing marketing yields to finding a cure for cancer. The technical lectures were packed, while the expo floor was focused. A loud rock and roll string trio in (unnecessarily short) black dresses exits the stage to be replaced by serious talk of open-source projects and community. One day a presenter will explain how clever they are to be able to apply pervasive surveillance of drivers for more profits, the next day a keynote is focused on rallying the audience to develop their ethics and fight the forces of ignorance.

Is the Hadoop Summit about vendor announcements or business outcomes? Is it about infrastructure or applications? Is it “sex sells” or cubical contributions? Is it profit or politics? Ultimately, it seems the big data industry is struggling to define who will benefit and how from our new technologies. If I had to pick some winners and losers, here would be my list.


Cloud — Hortonworks announced their hybrid management tools, Google demonstrated “fast, easy, and cheap” big data in GCP, Microsoft Azure showed real world impact of data, and BlueData talked about hybrid fluidity. Although AWS wasn’t present in booth, they were everywhere in mind.

Developers — Zeppelin and Jupyter are gaining ground for collaboration, and IBM’s data science experience is still top of mind. Make it easy for the pointy end of the stick.

Openness — Everyone is showing interest in open-source, both consuming and contributing. The exception being when proprietary offers significantly better quality, though these examples are looking thinner. The ODPi is gaining momentum like a freight train, slow off the line, but now chugging along. Accelerating ISV releases and testing while bringing more confidence to buyers. 


Point products — The trend seems to be skewing towards “more is more” with those vendors offering broader portfolios and value gaining preference. There is obviously still room for the specialized plays, but it’s got to be really compelling and differentiated to justify another contract.

Hardware vendors — Great products on display, but a feeling that this wasn't where the real action will be going forward. The most notable exception may be "ready to run" engineered systems and appliances.

Proprietary offerings — See openness. Facilitation, specialization and supportability were held up as the key attributes of vendors' pitches, but commoditization is creeping in.

All in, this shows an industry that's moving on to the real business of accomplishing real business. Technology for technology's sake isn't the way forward. It's got to make a difference.

big data internet of things

Topics: Data Platforms, Analytics, & AI