The Future of Data Storage and Management Is Nigh!

tunnelYou may have heard that Gene Amdahl recently died. I was reading some of the surrounding commentary and analysis, but—I am not sure whether this was cathartic, reassuring, or horrifying!—many of the things he predicted back in the 70s and 80s are only just happening now. And many of his predictions for what's still to be expected seem valid. I am not commentating as much on the fact that things take a long time to happen...more on the fact that our imagineering (as Disney would put it) isn't perhaps always as dramatic as we need in order to make big leaps forward.


I think a lot of this stems from our tendency (I am talking general human nature as much as specific IT technologies) to improve rather than to reinvent things. We like to iron out problems and make things incrementally better. That's laudable of course, but it is very different from foundational change, where more often you ignore the standard modus operandi, rip up many of the "rules," and start to try to address the underlying problem rather than simply improve the existing solution(s).


And, further, it seems to me that's exactly where we are with much of the storage industry today. There's a whole tranche of “evolutionaries,” but only—although satisfyingly—a small, though increasing, band of (let's use the word) “revolutionaries.” The challenges of the old ways (which have been largely unchanged for a generation) are large and well know…and so the opportunity for those who can address them successfully are enormous—even as the inevitable naysayers and unavoidable demographic-anchors might slow things down a bit. Change is clearly expected, even as we largely carry on much as normal today.


What it all amounts to is an impending and dramatic change in the way data storage and management gets done. We simply won't have a storage business (like we've known it for decades) in a relatively few years from now. I am not just talking about cloud and convergence; they are clearly very important contributors and enablers but they are essentially variations to the consumption model. Instead I am talking about a "take" on what we currently call software-defined storage (which is way too big of a descriptor and is suffering from some of the semantic uncertainty/generalization that plagued “cloud” early on) that I will for the sake of this blog call software-converged-data-management. Please note that's all lower case as I am really not trying to name the category just yet!


What are the key elements of this new approach? To keep it simple there are two:

  • a focus on data rather than storage
  • a heterogeneous and agnostic approach whether looking "up" or "down" the IT "stack" (up at applications and user needs, and down at the infrastructure).


That list of two may read as simple but it encapsulates a revolutionary move away from storage being proprietary, closed, and hence often the tail wagging the dog! At ESG, we’ve been commenting on this for some time now. Check out this ESG Video Capsule that Steve Duplessie did last November and a more complete article he wrote just recently. Our latest 2015 Data Storage Market Trends research report (subscription required) also clearly demonstrates that users are expecting genuine paradigm shifts to occur. 

How and when does all this happen? There are a number of interesting approaches emerging…not all “from scratch” per se, but all sharing the necessary new thinking that leads to a very different and holistic approach to our storage and data management conundrum. Please note that I am not about to produce an inclusive list (so no postcards and e-mails please as I’ll come back to this topic with more exhaustive vendor names over the coming weeks) but rather for now I just want to give a couple of examples so that you can see that the change is being worked on from a number of angles. You certainly see some of the mega systems vendors deploying portfolio tools designed to bring all data management together—IBM’s Spectrum approach is one. There’s great vision and marked improvements from traditional storage players—such as NetApp with its Data Fabric. And then there are a number of new startups—Formation Data Systems is a good example—that are simply building what we would have had if we could only have thought of it (and had the software and processing power!) when Amdahl and his peers were really getting going.   


Why does all this matter? It’s simple: The current storage model is stretched to its limits operationally, technically, and economically. We can probably get a few more turns of the crank before the engine seizes but everyone can see it is not sustainable. Only by really rethinking how we manage everything data-wise can we continue to support the kind of digitally enhanced and enabled future that we all desire and expect. A couple of quotes probably wrap this blog better than I could: Rich Warren said that "We are products of our past but we don't have to be prisoners of it," which is certainly neat phrasing to make us think of doing things differently…but it is Lao Tzu who give us the necessary “burning platform” motivation: "If you do not change direction you may end up where you are heading."

software defined storage insight