Big data is on the minds of every CEO, they hear about it and know that it can add value. But it is becoming more common that big data projects end up failing. There are some solid reasons for those failures and one of those reasons is the structure of the process to use big data. Big data can work and it can be a successful program for a company if it is understood and executed properly.
Big data projects often fail from a lack of proper understanding of the disciplines involved. Too often it is seen as an IT project when in fact, IT is one of 5 steps. It is seen as IT because that’s where a lot of the buzz and hype has taken place and it gets incorrectly dubbed an IT discipline. As a result, companies go out and hire a bunch of IT people and think that something will come out of it. That’s is not the first place or the main place to put your focus in big data. Big data is really all about using data to improve your business or customer experience. And because of that, it is a multi discipline activity that requires many teams.
I like to use the Quimby Wheel of Big Data to help explain this. The Quimby Wheel is a five step process that helps any company figure out how to use big data. Those steps are:
2. IT development
4. Data Visualization
Each step of the Quimby Wheel, helps a company to maximize their benefits from a big data project.
Strategy is the business arm, the part where you define what is going to happen, you need a hypothesis of the outcome and goals for what you expect and this needs to be in the form of business benefit or customer benefit. Normally a business owner or a marketing team does this.
Once you know what is the goal, then you talk to IT and look to see what tools you will need to accomplish the goal. This part has gotten far easier. With companies like Hortonworks, Cloudera and MapR all selling solutions, you can easily go to a vendor and get a solution without doing a lot of the work yourself. And frankly there are few cases now where you need to do it all yourself, two years ago, that was a different story.
Many companies actually start here or end here, that is a recipe for failure.
Once you have a system set up and data is collected, you need to analyze it. This is not an IT function, this is something for data scientists (who are not IT) or traditional BI. Analysis is about finding the gems in the data. A data scientist will spend 80% of their time on the data analysis and data cleaning. The other 20% may be spent on creating algorithms.
A mistake some companies make is putting these people in IT which then eventually only wants to focus on the algorithm coding piece. That often leads to an exit of great data science talent.
Once you have analyzed the data, you need to tell the story of what the data is telling you. This is where the designers come into play. Some data scientists can do this piece but I have found that a designer who is good at telling a story graphically, and has a passion for this, is great. Often this skill set already exists in an organization in branding or UX. The tools they use are very familiar to a designer.
The ability to tell the story about what is in the data is critical to getting the information out to the people that want to use it. They don’t care that you used hadoop or R for your analysis, they want to know what does it all mean for the bottom line. Data visualization helps to give them that story.
Once you have reach the end, you need to go back to the strategy piece and see, did you meet expectations or not? I remember once an IT executive was disappointed because he wasn’t surprised by the data. If you did your strategy piece correctly, you shouldn’t have too many surprises, your hypothesis should be fairly close to the outcome. It’s not about being entertained by the data, it’s about meeting the objectives as objectively as possible.
By following the Quimby Wheel, any company will ensure that their chances of success with a big data project, will increase. Of course there are more details than are listed here but this provides the basic frame. Adjustments made to each business and the goals of the project will need to take place but the overall framework will remain the same. Follow this and you will see your big data project be a success.