Big Data's 'Elephant in the Room': The Issue Nobody Wants to Talk About

It’s been a hushed murmur in LinkedIn discussions and blog posts. It’s a silent scream in many businesses. But until now, it hasn’t been spoken aloud. Intel changed all that recently by coming out and publicly saying, “This is the dirty little secret about big data: No one actually knows what to do with it,” stated Jason Waxman, an Intel vice president and general manager of the Intel cloud platforms team, “They think they know what to do with it, and they know they have to collect it, because you have to have a big data strategy. But deriving the insights from big data is a little harder to do,” he said.

The Claim

In fact, many companies give up on their big data initiatives before seeing any actual ROI, and still more stand on the sidelines, horrified to undertake a big data program at all. Hadoop adoption is an excellent indicator of how big data is faring in the marketplace, as this platform is more or less synonymous with big data for most organizations. Hadoop adoption across the board stands at somewhere around 10 percent, but most of this is concentrated among the Fortune 1000 companies. This means that an astoundingly few small- and mid-size businesses are actually taking a bite out of big data.

The Truth in the Claim

What’s the holdup? Well, there are three primary inhibitors to big data adoption:

• Organizations are unsure what to do with big data
• Organizations are unsure where to get the expertise to manage big data
• Organizations are reluctant to invest in the tools necessary to delve into big data

Big data tools like MapReduce have given way to next generation tools like Spark incredibly quickly. Now many organizations are stuck with a bunch of MapReduce stuff that was incredibly expensive, time consuming, and labor intensive but is now outdated. That’s just the companies that were actually able to score experts capable of handling these tools, and those that were able to figure out what they wanted and needed to do with big data.

The Untruth in the Claim

The one notable exception is in the marketing industry. Marketing professionals have been quick to realize the potential for consumer data. Marketers in all size businesses and across industries have been able to hone and refine customer profiles, measure the success of marketing campaigns, and develop more relevant campaigns using big data. Now the business side needs to play catch up.

One easy way to get into big data and learn the ropes is to begin with something simple, such as big data marketing or offloading mainframe data into Hadoop. These are both excellent ways to learn the process and get used to handling big data before taking on a more challenging function, such as using data for forecasting or operational intelligence purposes.

Other ways to make big data easier to manage are:

• Training big data scientists and analysts in house (these are rare, expensive, and hard to come by through hiring).
• Utilizing the cloud for data storage and processing, which eliminates the enormous strain of buying and maintaining IT resources in house.
• Find other businesses that have been successful in a big data initiative and recreate their processes and procedures. Case studies and white papers are an ideal resource for finding these real-world examples.

Though organizations may need some help getting started with big data, few question its business value. Big data may be slow to catch on, but that doesn’t mean it won’t. Just like previous technologies that rocked the known universe (barrels, toilets, and radios, just to name a few) big data will eventually achieve mass acceptance. Will you be among the early adopters?