The Hot and Cold Relationship Between CIOs and Hadoop

Do CIOs love Hadoop, hate it, or love to hate it? It turns out, the answer is: a little of all three. While hiring Hadoop talent at unprecedented rates and paying hefty sums to get those workers, CIOs also complain about the many problems they have implementing and using Hadoop clusters. Here is a look at both sides of the CIO-Hadoop love-hate coin.

CIOs Know Big Data is Important to the Future of Companies

Big data is slated to become one of the primary driving functions of future business models in every industry from health care to marketing to sports. CIOs can’t afford to let their companies fall behind, because as competitors get the big data advantage, it might be impossible to catch up later. Since Hadoop is fundamental to most companies leveraging big data, no matter how a CIO feels about Hadoop, they can’t allow their companies to get passed by competitors who are investing in Hadoop setup and implementation.

CIOs Can’t Allow Their Competitors to Get all the Talent

Since companies delving into big data have few choices besides Hadoop at this juncture, CIOs can’t afford to let their competitors scoop up the talent necessary to leverage big data and Hadoop resources in the future. Hiring of Hadoop talent is up more than any other type of business talent, while salaries soar to a median of $100,000 per year. Demand is up by 34 percent for Hadoop talent, and a shortage of such talent is expected to reach 140,000 to 190,000 workers by the year 2018. Workers who know how to make sound decisions based on big data analysis are already hard to come by, and smart CIOs can’t let their competitors scoop up those workers, leaving their company in the dust.

However, even while embracing Hadoop, which well over half of companies are, 72 percent of CIOs surveyed say it’s too soon to say how important Hadoop will be to their future technologies.

Hadoop is Difficult to Program

After covering the “love” side of the CIO-Hadoop coin, it’s time to examine the “hate” element. A recent survey of data scientists discovered that 76 percent of them have found Hadoop too hard to program, leading about 35 percent of these scientists to give up on it entirely. Difficulties finding and hiring Java programmers in this already difficult environment for talent dampers many CIOs’ enthusiasm for Hadoop.

CIOs Worry That Hadoop is Too Unpredictable

Another common complaint is the unpredictable and sometimes risky nature of Hadoop. Some CIOs report problems with how long it takes Hadoop to complete jobs, the inability to predict how long it will take Hadoop to complete jobs, and the resources Hadoop takes away from production tasks, sometimes causing the production jobs to fail completely. This is a definite point of contention for CIOs who have to justify the costs of this downtime and lost productivity.

Surprisingly, Security Isn’t a Major Concern for CIOs

Of all the reasons CIOs have to hate Hadoop, it would be expected that the security concerns ranked high on their list of concerns. However, only 2 percent of CIOs surveyed cited security as a Hadoop concern. Over half cited “undefined value proposition,” 21 percent said, “cost of acquiring skills” and 19 listed “integration with other infrastructure” but few worried about Hadoop’s lack of adequate security.

In the end, though CIOs might fuss and fret over the issues with Hadoop, most are sticking with it in order to assure their competitive stance heading into the future of big data.