...and what it means for big data and analytics.
As a data-driven person, I was a bit surprised at the outcome of the election yesterday. I had closely watched the top sites tracking up-to-date information about the moods, intentions, and opinions of the American populace. Most of my focus was on:
1. Nate Silver's FiveThirtyEight (which aggregates and analyzes others polls)
2. Politico (which had a nice visualization and drill down interface to live results)
3. Votecastr (which, along with Slate and Vice, published actual polling place observations)
As you know by now, they all had their issues. Margin of error is part of the story, but I think it was more a failure to employ modern big data approaches, relying instead on the traditional method of just asking people and then providing arbitrary weights to their responses. Human behavior tricked us in the form of confirmation bias, shyness about speaking unpopular beliefs, not reaching the right samples, and basic emotions like hopes and fears.
If you want to know more about how we were misled, and how big data could help, not hurt read more in CIO...