Skip to main content

Posts

Showing posts from October, 2019

Apache NiFi Load Balancing via Load Balanced Connections

Modern Apache NiFi Load BalancingIn today's Apache NiFi, there is a new and improved means of load balancing data between nodes in your cluster.   With the introduction of NiFi 1.8.0, connection load balancing has been added between every processor in any connection.    You now have an easy to set option for automatically load balancing between your nodes.  The legacy days of using Remote Process
Groups to distribute load between Apache NiFi nodes is over. For maximum flexibility,
performance and ease, please make sure you upgrade your existing flows to
use the built-in Connection Load Balancing.

If you are running newer Apache NiFI or Cloudera Flow Management (CFM), you have had a
better way of distributing processing between processors and servers.   This is for
Apache NiFi 1.8.0 and higher including the newest version 1.9.2.
Note:  Remote Process Groups are no longer necessary for load balancing! Use actual load balanced connections instead!
Remote Process Groups should only …

Migrating Apache Flume Flows to Apache NiFi: JMS To X and X to JMS

Migrating Apache Flume Flows to Apache NiFi:  JMS To/From AnywhereArticle 8 - This - JMS Article 7 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_15.html
Article 6 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_9.html Article 5 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html Article 4 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html Article 3 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html Article 2 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html Source Code:  https://github.com/tspannhw/flume-to-nifi

This is a simple use case of being a gateway between JMS and other sources and sinks.   We can do a lot more than that in NiFi.  We can be a JMS Consumer or Producer.  All with No Code.  We can work…

Migrating Apache Flume Flows to Apache NiFi: Any Relational Database To/From Anywhere

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest to Parquet, Kudu, ORC, Kafka, Hive and Impala

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest
Source Code: https://github.com/tspannhw/minifi-grove-sensors

Acquiring sensor data from Grove sensors is easy using a GrovePi Hat and some compatible sensors.


Just before my talk at the Future of Data Meetup @ Bell Works in Holmdel, NJ, I thought I should ingest some data from a grove sensor interface.
It's so easy a sleeping cat could do it.



So what does this device look like?  

I have a temperature and humidity sensor on there.


The distance sonic sensor is in there too, that's for the next article.


Let's do this with minimal RAM.


That's a 64GB hard drive underneath in the white case with the RPI.



I need more data and BACON.

We design our MiNiFi Agent Flow in CEM/EFM.   Grab JSON data stream and run sensors.
Apache NiFi 1.9.2 / CFM 1.0 Received HTTPS S2S Events From MiNiFi Agent


A simple flow to query and convert our JSON data, then store it to Kudu and HDFS (ORC) as well as push it to Kafka with a sche…