For Big Data, Does Size Matter?

Following a presidential debate in which the candidates concerned themselves with the proportions of their various body parts, I can't help but wonder why some are so obsessed with size. In the world of big data, there are a few different ways to think about this topic:

  1. big data content marketingThe most obvious referring to the huge volumes of data being stored and analyzed. Generally, "more is better" in that department, giving a richer and more complete view of a given situation. Probably the most important factor here is that the immense amount of data kept remains economical and relevant.

    A related consideration is the data lifecycle, meaning at what age does the equation change such that excess data becomes more expensive than its worth. Sorting this out isn't easy, as one can usually think of circumstances where a longer historical view is still useful. Conversely, there can be some legal exposure in having data beyond any mandated duration. Don't be afraid to set policies for housekeeping, and don't hoard data for nebulous reasons.

    Not least, there is the ongoing debate about sampling versus examining all data, and finding the optimal balance between accuracy versus resources and/or time required. This applies to both model building and production analytics. Smart vendors are exploring these different trade-offs around tremendous volumes of data.
  2. Another aspect of size in the context of the big data market place is vendor size. This is indeed worth considering. Though the tiny startup may sometimes have the coolest new innovation, the accompanying business risk is measurable.

    Ask yourself, do they have a big enough war chest to be sustainable? Do they have enough staff to support your efforts around the world? At the other end, are the incumbent IT industry titans sometimes too elephantine to be nimble? Is big data a large focus for them or just "one of many" solutions competing for attention internally? Are you a sizable enough fish to motivate them into offering good discounts or high-touch help when you need it? There may be a sweet spot in the middle where a certain class of vendors are big enough to be healthy but small enough to care!
  3. By far the least important aspect of size is branding. Some IT vendors want to be seen as "thought leaders" around hot trends like big data and IoT, yet don't really have much of substance to say. So these companies are overcompensating for small value-adds with gigantic marketing budgets. I've personally experienced this on projects where the vendor emphasizes the size of their install base (for other products), the prodigious number of partners they have (doing what exactly?), and how much they are spending on R&D (for intangible results.)

    The very worst instances come when vendors simply try to buy their way into prominence with platinum event sponsorships, mammoth booths, and gargantuan ads plastered everywhere. By all means, if you have something useful to say, spend on good content marketing to tell your story. Yet such ego around logo size and placement does nothing to help the market develop or the customer succeed. This is sheer folly on an colossal scale.

So yes, size does matter, but not always in the way you might think.

IT research firm analysis

Topics: Internet of Things Data Platforms, Analytics, & AI