If you're reading this blog, statistically it is highly probable that you work in IT like me, the lone anomaly being my mother (hi, mom!). And if you are like me, you probably got into IT not so much because it's a sexy topic for cocktail parties, but because of all the cool techie toys, a love of science fiction, and a keen desire to understand how things work.
So, it's a bit of a surprise to me that IT has only recently started to seriously adopt big data and analytics to explore its own foibles. Number crunching has long been aimed at better business intelligence, finding market opportunities, making products better, reducing cost inefficiencies, and such. Why haven't we done more to analyze our own machine and sensor data?
The problem is certainly not that your various hardware and software doesn't generate plenty of data about itself. Actually the problem was much the opposite. I started my career in tech support (see, glamorous already!) and remember endless hours of scrolling and searching through log files for timestamps to correlate particular daemons' activities with an incident. Sometimes this meant days onsite with a panicked customer, sometimes they left me in my cave. It was grueling, stressful, and tedious all at once, and most often circled back with another request for yet more log files.
Thankfully, there is a better way of gathering, storing, analyzing, and identifying issues with IT systems today, being the category of products for operational analytics. Some vendors have been active in this space for a while, but newer big data technologies like Hadoop and more are finally giving them the horsepower to actually deliver the goods in the form of real-time alerting and troubleshooting, historical analysis, and even predictive "what if" capabilities, most importantly at a reasonable cost. This is tremendously powerful if IT wants to keep earning the right to be the provider of choice for its own constiuents in the lines of business, who are otherwise all too ready to self-provision new resources and services from public cloud offerings or other third parties.
Without going into any details here, I'd call out some of the leaders in this area as IBM, HP, Riverbed for performance management, CloudPhysics for virtual server environments, and Splunk for just about anything.
Image of Moss from "The IT Crowd," which is so fabulous I can't believe you haven't seen it yet.