Hadoop Used to Predict Flight Delays

Hadoop consolidates large volumes of information, stores it efficiently, processes it powerfully, and does all this inexpensively. This makes Hadoop an ideal tool for many applications, and data scientists have tapped this potential by illustrating how Hadoop can be used to predict which airline flights will be delayed or cancelled. Obviously, this is useful information for consumer apps, not to mention travel agencies and other stakeholders.

In a nutshell, data scientists compiled data on flights from 1987-2008, including the airport of origin, the destination airport, and 26 other variables to build a learning model that accurately predicts flight delays using historical flight data and weather information.

This is a nifty demonstration, but admittedly a narrow view of what Hadoop can do. After all, adopting Hadoop is not a trivial undertaking. Are there situations when Hadoop isn’t really the right hammer for the nail?

When Hadoop is the Best Tool for the Job

Hadoop does an excellent job of distributing file systems for storage, and the MapReduce function allows for powerful analysis even on unstructured data sets like graphs and charts, social media data, documents, and videos. However, when the data is structured nicely and fits easily within columns and rows, Hadoop might be overkill. Even big data is easily managed with other less cumbersome, faster tools like spreadsheets.

Hadoop also does great with smart grids, research and development data, operational data, and data on complex transactions, not to mention building indexes, recognizing patterns, and creating recommendation engines or conducting sentiment analysis. If high volumes of data are involved and it’s not necessary to query the data quickly, Hadoop works wonderfully.

When Hadoop Isn’t the Best Tool for the Job

However, Hadoop doesn’t work as well with transactional data, such as the transactions of an eCommerce site. These types of transactions involve numerous steps, all of which are performed rapidly. Hadoop also doesn’t perform well in situations that won’t tolerate latency. A smart solution is to use Hadoop for managing unstructured big data when time isn’t of the essence, and continuing to leverage well-established tools for structured data and situations that demand low latency.

To get the most out of Hadoop, it needs to be utilized on the bare metal cloud. Hadoop was designed to work in this environment, and users who switch from the virtual cloud to the Full Metal Cloud offered by Bigstep are astounded at the speed they gain when querying. Visit Bigstep today to see how the full metal cloud can boost your Hadoop experience to new heights.