Predicting the unthinkable

charts.jpgThe UK has voted to leave the European Union. The Republican party has voted Donald Trump as their presidential nominee. Few predicted these things would happen. Why not?

We at ESG do a lot of surveys and interviews in our research. Sometimes this is a broad view of market sentiment, sometimes it's a very specific question that needs answering. Technology vendors and buyers alike rely on our work to guide their most strategic decisions. We pride ourselves on careful survey design, controlled selection of qualified respondents, and thoughtful analysis of the findings. Our methodology isn't just SurveyMonkey and $5 Starbucks gift cards going out to 1,000 Shakespearean monkeys with 1,000 keyboards. Yet there are a couple of inherent limitations in asking humans their opinions.

Surveys aren't a big data or advanced analytics method. By definition, a survey has a relatively small number of responses to a relatively small number of questions. It isn't a complete view of the particular environment, concerns, goals, and factors that guided past decisions. Forward-looking statements are aspirational and by definition reflect ideas of things that haven't happened yet. At the same time, more information and understanding is always better. Getting into behavioral analytics can definitely help guide business decisions, with a lot of data on circumstances and outcomes used to determine meaningful correlations, if not causations.

Even machine learning with careful training on massive event-series data sets, over long periods of time, and with refined models, doesn't get the predictive side right in all circumstances. You might be able to show very high accuracy of pattern spotting when the same outcomes are repeated many times under the same circumstances, thereby supporting a hypothesis. This is the very fundamental definition of empirical science, done at enormous scale. The biggest blind spot? When something has never happened before.

There may be faint echoes of similar scenarios, and those who cannot remember the past are condemned... Yet maybe the most frustrating part of recent world events to the analytically-minded is that there is a clear information asymmetry in how voting decisions are made. Not everyone has the same information to make an informed decision, and expertise from painstaking study and long experience is too often being dismissed as elitism. Fact-based, logical rationale is falling out of favor, and we're defaulting back to evolutionary instincts of fear and desire to belong to a dominant group for survival. People don't like change. There is a lot of psychology, sociology, and organizational behavior theory in play now. Competition for scarce resources is damping humanity's better judgment, and the prisoner's dilemma is falling back to its default "bad for everyone" state.

At the same time there are definitely lessons for businesses on how to build a data-driven culture, balanced by human wisdom, and we see real signs of progress everywhere, all of which is encouraging. I strongly disagree with other analyst firms that say most data lakes are doomed to failure, they just need refinement and attachment to important questions or processes. We must strive for knowledge and wisdom, and technology can certainly help us get smarter. As far as the world geo-political landscape, well, we're in uncharted territory here, folks. Chin up and fight for insights, not just instincts. big data internet of things

Topics: Data Platforms, Analytics, & AI