What's New & Exciting for the Hadoop Ecosystem These Days?

Remember a few years ago, when Hadoop took knocks left and right for lacking usability, security, and other key features and functionality? Well, no more. A couple of weeks ago, Hortonworks revealed the latest version of Hortonworks DataFlow (HDF), its integrated system that allows for dataflow management and streaming analytics.

What’s Up at Hortonworks?

It’s called HDF version 2.0, and according to VP of Product Management for Hortonworks, Jamie Engresser, it reflects a lot more interest by customers in streaming analytics. Users are wrestling with all of the data in motion flooding in from the Internet of Things devices, clickstreams, and sentiment analysis.

“We have seen significant HDF adoption across our customer base,” Engresser said in a statement Tuesday. “Customers clearly want an integrated, real-time solution for streaming data and the new functionality of HDF 2.0 accelerates business value from data-in-motion for customers.”

HDF was officially created in August of 2015, when Hortonworks acquired Onyara, one of the key contributors to and creators of Apache NiFi. Onyara’s engineers became the core team responsible for HDF, which is meant to make it easier for users to automate and secure their data flows, as well as collect, curate, and analyze business intelligence in real time, using streaming data (data in motion).

Version 2.0 brings a new level of functionality to the user interface. It’s designed to make deployment easier and help enterprises adopt the IoT. A few of the most notable features of HDF 2.0 include:

A new and improved user experience—This version offers a brand new graphical UX, as well as integration of Apache NiFi, Kafka, and Storm directly into the Ambari management console. According to Engresser, it’s kind of like Google Docs for moving enterprise data.

Integration with Apache Ranger—This integration is going to assist the enterprise in terms of preparation and readiness for HDF. It provides centralized total security policy management for streaming data across the enterprise. Engresser said, “It’s really about the data. How do I operationalize it? How do I deploy it? How do I monitor it? How do I manage it? How do I secure it? Now all open source components get monitored and managed from one centralized console.”

HDF 2.0 includes Apache MiNiFi, which is a lightweight mini version of NiFi. According to Engresser, this allows for the delivery of NiFi in a far smaller form factor, which can then be installed on smaller IoT devices. It also delivers enterprise-level security and management at scale.

What’s Up at Cloudera?

Hortonworks isn’t the only one who’s been hard at work on their Hadoop ecosystem offerings. Cloudera hasn’t yet announced a release date for version 5.8, but has already began tempting customers by publishing what they can expect in the upcoming version.

• Active data optimization. This should help organizations, particularly larger enterprises, adopt Hadoop successfully. Cloudera Navigator Optimizer is now available, delivering an easy option for offloading certain SQL workloads into Hadoop, as well as managing data within the Hadoop ecosystem.
• High-performance analytic SQL in the cloud.
• Improved SQL development experience.

For a sneak peek at Cloudera 5.8 check the Cloudera developer’s forum.

Of course, you can always begin leveraging the best of the best of Hadoop in the Bigstep cloud. Better performance—guaranteed. Learn more about us now.