The Data Scientist in the Room

This summer I spent a lot of time interviewing the self-proclaimed big data leaders at a wide range of midmarket and enterprise companies. These are the people charged with defining their organizations' big data strategies and new initiatives. First, let me say these are not dummies. No one gets put in the driver's seat for these projects without knowing a lot about technology and business both.

What was a little surprising was the spectrum of views on big data. Many saw it as a chance to upgrade and modernize their server, network, and storage infrastructure. Some were focused on integration of new internal and external data sources to traditional analytics activities. A few were putting a new label of "big data machine" on their biggest, shiniest rack in the data center. More than a couple wanted to wholescale adopt new data platforms such as Hadoop and NoSQL databases. One sheepishly suggested he'd do whatever his data warehouse vendor recommended, figuring they must be the experts. Most were on a headhunt for the perfect unicorn, err, data scientist.

What most of these clever people had in common was a shared view that big data is far more than the sum of its (many) moving parts. To get their companies really on board and really benefitting from the efforts, almost all felt they had to show the value. While there is an intrinsic, almost fanatic belief that big data will pay off, it has to pay every time for every player. From the board room to the meeting room, everyone wanted more of their employees to embrace the data in making business decisions. The scientific method of hypothesis, collect data, test assumptions, calculate results, interpret, and report findings has never been more popular.

Yet this can contradict the "cult of personality" in many companies. A lot of institutions are still top-down, vision-led hierachies. Big data can and should influence the executive's strategic plans, but it will also usher in a golden data-driven meritocracy (datatocracy!) where the plan is fact-based, not employee title-based. This will be a big change for many corporate cultures, and should be as much of the initiative as any piece of technology. I predict that the vendors that help their customers succeed in transformation of business behavior will also prevail in market share.

Topics: Analytics Big Data Data Management & Analytics Enterprise Software