Who Are Onyara & What Is Apache NiFi?

Total Shares

http://nifi.apache.org/images/niFi-logo-horizontal.pngWhen Hortonworks announced they were going to buy Onyara I have to admit I had never heard of Onyara and had no idea what technology or value they would be bringing to the Hadoop table.  To be acquired there had to be something interesting I was missing.  So, Starting with a basis of no knowledge, my natural curiosity kicked in. For the past few days I have been trying to learn about Onyara and the technology they were involved with. This blog reflects a little of what I have learnt. I welcome any comments or additional information to help all of us learn more.

Onyara

If, like me, you had never heard of Onyara then you are probably not alone. Onyara was only founded earlier in 2015 so they are a very young company.  They are a  small privately owned organization based out of Washington featuring a number of individuals (their LinkedIn page described them as a 1-10 person organization) who are credited with being key contributors to a US government project which would eventually become Apache NiFi (more on that in a moment).

It seems clear that the team at Onyara were/are the largest group of definitive experts in Apache NiFi in terms of its original aims, internal workings, development history and, one has to assume, its usage. Given Apache NiFi is open source you cannot say that they have been bought because of their technology IP so what is it that Hortonworks is getting?

With the acquisition @Hortonworks has gained deep expertise, and a strong voice, in Apache NiFi. Click To Tweet

In short Hortonworks can now probably help to shape the product development towards what their customers are asking for and they can serve their customers with established experts especially in the Internet of Things space.

Apache NiFi

This was an Apache project I had never personally heard of which many at Onyara were helping to drive. Apache Nifi originated from the NSA and was released via the NSA Technology Transfer Program back in Autumn 2014.  It gained top-level Apache project status only in July 2015 so in that sense it is a very new Apache top-level project. Apache NiFi might be new but the development effort has been going on for sometime with 8 years of development inside the NSA.

Apache NiFi might be new but the development effort has been going on for sometime Click To Tweet

The first impressions most people will get of Apache NiFi will focus around the GUI development environment. It looks very much like many of the Data Integration/ETL tools of today. It seems to feature a decent drag and drop interface, connectors to a host of different systems, processors to transform data and more.  If something is not there it is touted as being easy to extend.

Under the covers there is of course an engine. This seems to be a lot of the secret sauce and I have much still to learn there. For now let me say that all I have seen suggests it is an advanced message orientated middleware, with a lot of additional capabilities, that can be driven by the GUI but I also assume via APIs. I could spend an age here describing it but I think the Video below will save you pages of reading.

It would appear that Apache NiFi will be able to support the more traditional use cases such as data warehouse and data mart preparation and it could also support analytic base table preparation or other types of data set preparation for BI/Analytics. Looking at the tool I am not sure it is simple enough for a non-IT person so its likely audience are IT folks looking to help support business applications or business counterparts with their data needs.

All of this is good but the real kicker is when you think ahead to the Internet of Things. Apache NiFi is touted to deal with a lot of issues such as two way data flows, acquisition of data from anywhere, storage of data anywhere and the associated security and governance that will be required.  Given it originated from the NSA this is probably not a hollow claim.

Acquisition leads to @Hortonworks DataFlow powered by Apache NiFi. Click To Tweet

As a result of the acquisition Hortonworks has announced plans to release “Hortonworks DataFlow powered by Apache NiFi”.

Want to watch a video today on Apache NiFi?

The video I found most informative I have embedded below. It was given fairly recently at OSCON 2015 by Joe Witt – a member of the Apache NiFi PMC. I think if you take the time to watch that you will get a very good overview explaining the issues they were trying to overcome, a few of the use cases, a view of the architecture and a demo of how you go about building flows. Very informative and it makes you want to learn more.

My conclusion and where to learn more?

I still have much to learn about Apache NiFi but on the surface it looks interesting. It might be disruptive in the traditional space when it comes to preparing data for marts, warehouses and analytics/BI especially when Hadoop is a core part of that effort. I also think it might become something organizations will look at before considering other options to get data in and out, and understand where it is in that process, when considering the Internet of Things space. I have to assume Hortonworks is starting to see a number of customers looking at that which is why they made the move. It makes sense given where our digital world and enterprises is moving.

So in making this move Hortonworks will have a core set of seasoned experts. While anyone can learn Apache NiFi the fact that Hortonworks has those deep experts may allow them to exploit it more quickly and with more confidence than others.

If you want to learn more then Hortonworks will host a webinar on September 23rd 2015 at 10:00 a.m. PST to provide an overview of the acquisition and discuss Hortonworks DataFlow. Click here to go register and maybe I will see you there!

2 thoughts on “Who Are Onyara & What Is Apache NiFi?”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.