An End to "End to End" Big Data Solutions?

JigsawSome people like putting together jigsaw puzzles, and some people like buying paintings. Both approaches are pretty popular in the world of big data, though perhaps the nature of analytics in business tends to skew things heavily toward the "some assembly required" side of the spectrum. This is not a trivial problem, but it's a problem that does need to be solved. Connecting data sources, preparing data, developing analytical models, sharing any normal workflow, there are a number of steps to be taken, and a much larger number of technologies that will come into play.

So why are tech marketers so fond of the term "end-to-end" solution? Well, complexity is daunting. Most business users just want to get access to relevant information and formulate some credible answers. How it all works isn't so important to them. Data scientists and IT architects get stuck with stitching together a Frankenstein's monster of scavenged software packages and hardware components, and no one else really wants to hear about it.

Yet, both business and IT leaders want to know that any investment they make in a new technology will 1) add some actual value and 2) work with everything else they already have lying around. An appealing fantasy is that buying everything from a single vendor makes this easy. We buy pre-assembled cars and refrigerators and houses, why can't I just buy a pre-assembled big data and analytics technology stack? The short answer is that you can, but it may not quite do what you want. If every car is a Model-T, then it's not hard to choose a black one and drive it off the lot, but what if what you need is a red firetruck, or a yellow sportscar? The Model-T ain't gonna get 'er done. That chassis won't hold water and those skinny old tires aren't going to give you much traction.

Even more confusing is that in the big data market, dozens of totally different kits are described as "end to end" platforms. Some are "complete" engines, some are "complete" dashboards, some are "complete" pipes. None are complete cars. What smart technology vendors should be doing isn't selling the fantasy of all-encompassing, one-size-fits-all solutions—they should be giving you a checklist and highlighting exactly which pieces they provide and which you need to source elsewhere. A slide with a dozen logos of so-called partners is not nearly as useful as a blueprint with the specific components highlighted and annotated.

Here's a good example from Redhat and Hortonworks (click to see bigger on their website):


Now this doesn't claim to include everything—it kind of leaves out the essential fact that some hardware will be needed somewhere below, and leaves the analytics and BI tools as just icons above—but it does show you what each company is actually bringing to the party. Is that so hard?

Don't be yet another "end to end" solution, just say what it is you do!

big data analysis

Topics: Data Platforms, Analytics, & AI