Skip to main content

Posts

Showing posts from October, 2019

Apache NiFi Load Balancing via Load Balanced Connections

Modern Apache NiFi Load Balancing In today's Apache NiFi, there is a new and improved means of load balancing data between nodes in your cluster.   With the introduction of NiFi 1.8.0, connection load balancing has been added between every processor in any connection.    You now have an easy to set option for automatically load balancing between your nodes.   The legacy days of using Remote Process Groups to distribute load between Apache NiFi nodes is over. For maximum flexibility, performance and ease, please make sure you upgrade your existing flows to use the built-in Connection Load Balancing. If you are running newer Apache NiFI or Cloudera Flow Management ( CFM ), you have had a better way of distributing processing between processors and servers.   This is for Apache NiFi 1.8.0 and higher including the newest version 1.9.2. Note:  Remote Process Groups are no longer necessary for load balancing! Use actual load balanced connections instead! Re

Migrating Apache Flume Flows to Apache NiFi: JMS To X and X to JMS

Migrating Apache Flume Flows to Apache NiFi:  JMS To/From Anywhere Article 8 - This - JMS Article 7 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_15.html Article 6 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_9.html Article 5 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html Article 4 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html Article 3 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html Article 2 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html Article 1  -  https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html Source Code:   https://github.com/tspannhw/flume-to-nifi This is a simple use case of being a gateway between JMS and other sources and sinks.   We can do a lot more than that in NiFi.  We can be a JMS Consume

Migrating Apache Flume Flows to Apache NiFi: Any Relational Database To/From Anywhere

Migrating Apache Flume Flows to Apache NiFi:  Any Relational Database To/From Anywhere Article 8 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_42.html Article 7 - This Article 6 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_9.html Article 5 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html Article 4 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html Article 3 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html Article 2 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html Article 1  -  https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html Source Code:   https://github.com/tspannhw/flume-to-nifi This is a simple use case of being a gateway between Relational Databases and other sources and sinks.   We can do a lot more than that in NiFi.

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest to Parquet, Kudu, ORC, Kafka, Hive and Impala

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest Source Code:    https://github.com/tspannhw/minifi-grove-sensors Acquiring sensor data from Grove sensors is easy using a GrovePi Hat and some compatible sensors. Just before my talk at the Future of Data Meetup @ Bell Works in Holmdel, NJ, I thought I should ingest some data from a grove sensor interface. It's so easy a sleeping cat could do it. So what does this device look like?   I have a temperature and humidity sensor on there. The distance sonic sensor is in there too, that's for the next article. Let's do this with minimal RAM. That's a 64GB hard drive underneath in the white case with the RPI. I need more data and BACON. We design our MiNiFi Agent Flow in CEM/EFM.   Grab JSON data stream and run sensors. Apache NiFi 1.9.2 / CFM 1.0 Received HTTPS S2S Events From MiNiFi Agent A simple flow to query and convert our