The Big Data Market - We've Seen This Movie Before

Nothing is ever truly new in IT, and Big Data is no exception.

Big Data in 2013 is SAP(ERP) twenty years ago. Or Siebel (CRM) 12 years ago. It's rolling out the exact same way.

1. Users use all sorts of disparate, loosely connected applications to piece meal a solution together. In the pre-SAP days, it was a warehousing system, an inventory system, a manufacturing/flow system, an ordering system, an accounting system, etc. etc.

2. Someone develops an integrated, single stack application that can throw out all the custom home grown, disparate junk systems and provide a single unified, connected pane of glass. In the SAP case, we called it ERP.

3. Advanced IT shops, typically in very large, well-funded companies with strategic vision, jump on the opportunity. They become the early adopters because they know that a successful implementation will provide a distinct competitive advantage to their organizations. These are few and far between.

4. The buzz created by the next great thing gets the mainstream market going. These are the everyday folk who's bosses boss hears about what's happening out there in the world and asks their IT department to get on the train—no matter what that train is, or how capable their organization is at getting on. (Cloud, anyone?)

5. Companies drink the kool-aid and start spending like drunken sailors. SAP becomes HUGE. Implementations go from dozens to thousands in short order. Each implementation has a massive up front cost (which SAP loves, as well as the ecosystem providers like EMC or HP in this case), and requires an army of services to make stuff remotely work. It takes years to customize the system.

6. 80% or more of all those really expensive implementations fail spectacularly—because the issue was never really about the technology (10%), it is always about the business processes—or lack thereof.

In the case of SAP's rise to glory, the issue was "now that you have spent 20 million bucks over 5 years to get this implemented, you need to change the way you do business to fit our software." That's why these implementations died a painful death.

Only after SAP was able to help customers (via partners) to understand that this wasn't a panacea platform that does all the work for you—but a panacea platform to capture and create efficiencies ONCE you re-architected your business outcomes/processes did it gain huge momentum again. And, it never looked back.

In Big Data, I see the same thing evolving. First it's all about the exciting "Gizmo's"—the industry gets fired up and buys Netezza, Greenplumb, Vertica, etc. because they do SOME function wicked fast—but mostly without context of the bigger issues (like "what the hell is big data?"). Then we move on to create a class of super geniuses—in this case "Data Scientists" who can manipulate data so that we can find needles in our proverbial haystacks. You can't find them right now—they are super rare geeks. (DBA's were super rare geeks, as were networking guys, as were security guys, as were storage guys....) so you pay a fortune or you wait—for a services company to come to you to charge you a fortune.

The same thing happened in the SAP days. A few big integrators made a zillion bucks in the gold rush because they had all the talent that could "DESIGN" the system to do stuff. They were SAP's data scientists.

Where you are going to fail is not in the infrastructure or the design—you will fail in the outcome because like then, you haven't thought through the business process transformation necessary to TAKE ADVANTAGE OF WHAT YOU DISCOVER within your pile of data. It's like rushing to hire a the best prospector, arming him with the best sifting equipment, pointing him to the place where all the gold is, finding all the gold, and having no means of getting the gold to the marketplace. Your are left with a nice shiny pile of rocks.

In order to make this all work, the buying community needs to realize that they will need to act on the information they mine—or it too will be a big pile of shiny rocks. The infrastructure team will do it's job, the data scientists will do their job, and the business will have wasted a truckload of time and money because nothing will come of it.

The evolution will be great for services companies, super for infrastructure companies, wonderful for big data/analytics software companies, and won't do crap for the mass market until someone teaches them how to effectively change the way they conduct business. Other than that, I'm all for it.

Topics: Data Platforms, Analytics, & AI