Skip to main content

Top 25 Use Cases of Cloudera Flow Management Powered by Apache NiFi

Top 25 Use Cases of Cloudera Flow Management Powered by Apache NiFi

 Cloudera Flow Management has proven immensely popular in solving so many different use cases I thought I would make a list of the top twenty-five that I have seen recently.   

If you have never used CFM or Apache NiFi before, please checkout these two quick resources:   https://github.com/tspannhw/EverythingApacheNiFi and https://nifi.apache.org/docs/nifi-docs/.

21-25

25.   Ingesting Data into Kafka in the Public Cloud

https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-kafka-ingest/topics/cdf-datahub-fm-kafka-ingest-overview.html

24.  Cybersecurity Data Collection and Filtering

https://www.datainmotion.dev/2020/10/monitoring-mac-laptops-with-apache-nifi.html

23.  Ingesting Data into Hive in the Public Cloud

https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-hive-ingest/topics/cdf-datahub-nifi-hive-ingest.html

22. Ingesting Data into HBase in the Public Cloud

https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-hbase-ingest/topics/cdf-datahub-nifi-hbase-ingest.html

21. Ingesting Data into Kudu in the Public Cloud

https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-kudu-ingest/topics/cdf-datahub-nifi-kudu-ingest.html


16-20

20.  Ingesting Data into ADLS Storage

https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-azure-ingest/topics/cdf-datahub-fm-adls-ingest-overview.html

19.   Populate SOLR Indexes

https://www.datainmotion.dev/2020/04/building-search-indexes-with-apache.html

18.  Hadoop Data to Kafka

https://www.datainmotion.dev/2020/04/read-apache-impala-apache-kudu-tables.html

17.   Deep Learning And Machine Learning Pipelines

https://www.datainmotion.dev/2019/12/easy-deep-learning-in-apache-nifi-with.html

16.  Intercepting JMS and SOA

https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_42.html


11-15

15.    Edge ML Model Integration

14.   Migrate Data from On-Premise Private Cloud to Public Cloud

13.   Converting XML to JSON

12.   MQTT to HDFS

11.   Ingesting REST Endpoints (Bulk)

6-10

10.  Ingesting Data into AWS S3 Buckets

9.  Ingest REST Endpoints

8.  Ingesting SAAS Products Like Salesforce

7.   Automating Manual Tasks

6.  Ingesting Social Media Data

Top 5

5.  Logs, Logs, Logs

https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html

https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

4.  FLaNK Streaming Data Pipeline (Any Data to Kafka to Flink SQL)

https://www.flankstack.dev/

3.   IoT - MiNiFi Agents Ingest, Store and Forward

https://www.datainmotion.dev/2020/02/edgeai-google-coral-with-coral.html

https://community.cloudera.com/t5/Community-Articles/IoT-Series-Sensors-Utilizing-Breakout-Garden-Hat-Part-2/ta-p/249380

2. Pseudo-CDC / Database Ingest

https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_15.html

1.  Doing a 1,000 different ingest, conversion, routing and transformation flows

The most common use case is doing a lot of things with a lot of data, including things like documents, XML, JSON, AVRO, Parquet, CSV, PDF, Images, Video, Mongo documents, Logs and more.    Rarely do I ever see someone solve just one problem with NiFi and say, that was enough.   One simple use cases leads to another and another and before you know it every cron job, script, ETL, ELT and big data op is now touched by NiFi.    Keep it up, Cloudera will make it ever easier soon.   Also check out NiFi Stateless for some of those more job/event oriented things like File to Kafka, Kafka to Kafka and more.

https://community.cloudera.com/t5/Community-Articles/Scanning-Documents-into-Data-Lakes-via-Tesseract-MQTT-Python/ta-p/248492


Popular posts from this blog

Ingesting Drone Data From DJII Ryze Tello Drones Part 1 - Setup and Practice

Ingesting Drone Data From DJII Ryze Tello Drones Part 1 - Setup and Practice In Part 1, we will setup our drone, our communication environment, capture the data and do initial analysis. We will eventually grab live video stream for object detection, real-time flight control and real-time data ingest of photos, videos and sensor readings. We will have Apache NiFi react to live situations facing the drone and have it issue flight commands via UDP. In this initial section, we will control the drone with Python which can be triggered by NiFi. Apache NiFi will ingest log data that is stored as CSV files on a NiFi node connected to the drone's WiFi. This will eventually move to a dedicated embedded device running MiniFi. This is a small personal drone with less than 13 minutes of flight time per battery. This is not a commercial drone, but gives you an idea of the what you can do with drones. Drone Live Communications for Sensor Readings and Drone Control You must connect t

Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HDFS / Kudu / File / Hive

Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HDFS / Kudu / File / Hive Article 7 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_9.html Article 6 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html Article 5 -  Article 4 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html Article 3 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html Article 2 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html Article 1 -  https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html Source Code:   https://github.com/tspannhw/flume-to-nifi This is one possible simple, fast replacement for " Flafka ". Consume / Publish Kafka And Store to Files, HDFS, Hive 3.1, Kudu Consume Kafka Flow   Merge Records And Store As AVRO or ORC Consume Kafka, Upda

Advanced XML Processing with Apache NiFi 1.9.1

Advanced XML Processing with Apache NiFi 1.9.1 With the latest version of Apache NiFi, you can now directly convert XML to JSON or Apache AVRO, CSV or any other format supported by RecordWriters.   This is a great advancement.  To make it even easier, you don't even need to know the schema before hand.   There is a built-in option to Infer Schema. The results of an RSS (XML) feed converted to JSON and displayed in a slack channel. Besides just RSS feeds, we can grab regular XML data including XML data that is wrapped in a Zip file (or even in a Zipfile in an email, SFTP server or Google Docs). Get the Hourly Weather Observation for the United States Decompress That Zip  Unpack That Zip into Files One ZIP becomes many XML files of data. An example XML record from a NOAA weather station. Converted to JSON Automagically Let's Read Those Records With A Query and Convert the results to JSON Records