Long Live Small, Fast, Actionable Insights!
Big data is everywhere. Organizations are being advised to hoard it, and to do everything they can to derive actionable insights from it. This article will argue that this approach puts the cart before the horse.
First, let’s start with a revelation. Despite the hype and despite everything you’ve been told about big data as a necessary precursor to achieving valuable business insights, it has by and large failed to deliver on its promises.
All the data in the world means nothing, if you can’t make sense of it. And that’s the hard part. Even Gartner predicts that through 2017, 60% of big data projects will fail to go beyond piloting and experimentation, and ultimately will be abandoned.
High Risk and Cost of Failure in Data Analytics
Part of the reason for this failure is that organizations rush to accumulate and analyze as much data as possible, all at once, and without much of a plan. This usually ends terribly. It’s not unusual anymore for organizations to house multiple petabytes of data. These compulsive data hoarders pathologically store any data they can get their hands on, thinking that the more data they have, the better off they will be, the more intelligent they will be, and the higher the quality and number of insights they can glean from it.
But starting big can lead to an overwhelmed team, not to mention huge costs. These organizations won’t be able to see the insights for the data, so to speak. With more data and more capabilities to rationalize data across silos, it can feel like there are more opportunities than there are resources to exploit them.
The problem is exacerbated by the fact that traditional approaches often fall short. Heavy duty enterprise business intelligence (BI) tools, central data warehouses, etc., designed to analyze big data sets, take years and cost millions to implement (something your BI service provider won’t tell you at the outset).
These heavy data warehouses can pay off. But when they don’t, the cost of failure is high. In a survey of more than 500 executives by McKinsey Analytics, 86% of respondents reported that their organizations were only somewhat effective at meeting the goals they set for their data and analytics initiatives.
The moral of the story? Hoarding big data is not a prerequisite for smart insights.
A Shorter, Nimbler Path to Insight: Follow the Breadcrumbs!
The truth is, for many organizations, starting small is a much more fruitful (and lower cost) exercise than bringing in heavyweight tools and consultants. Time and again we have seen some of the most valuable business insights derived from surprisingly small data sets. It’s time to focus, not on big data, but on the right data at the right time, and find a way to ask the right questions of that data.
It’s an approach to data analytics that we replicated from our best practices in software product development. Focus on delivering an MVP as fast as you can, and iterate from there. In this case, MVP does not stand for Minimal Viable Product, but Minimal Viable PREDICTION.
The MVP approach challenges traditional perceptions and established norms of big data and data analytics. The notion of having to assemble data, at any cost, is an extreme one. Instead of eating the entire elephant at once (your mouth is too small and you’ll choke), we recommend a low-risk approach to data analytics that a successful start-up might take. It is grounded in best-in-class software development methodologies. Here’s how it breaks down:
Start with focusing only on the #1 problem you want to solve - Instead of assembling all your data, disregard the noise and assemble only the data that correlates with your number one problem.
Once you’ve established your primary problem, work backwards from there identifying other related data, and follow the data breadcrumbs that lead to actionable outcomes. For example, you may start with understanding and predicting customer complaints - something that can have a very real impact on your revenue and profits. So, you’d assemble complaints data. Your next data breadcrumb might be customer service representative data, a potential reason for complaints. Another set of data could be transaction size by customer. Easy enough so far, and guess what? You may already be halfway there or more! Perhaps a single customer representative evokes unusually high numbers of customer complaints. But you can further query the data to understand whether transaction size shows correlation and predictability of customer complaints in association with that customer service representative.
You have just reached MVP status – you’re a Most Valuable Player having created a Minimal Viable Prediction! This sounds straightforward, but it can be tempting to jump into the data lake in front of you instead of just following small, relevant, iteratively assembled data breadcrumbs.