Skip to main content

Posts

Showing posts from August, 2019

Google Coral TPU with Edge Devices and MiNiFi

Google Coral TPU with Edge Devices and MiNiFi 
Designing Our Edge AI Flow with Cloudera Edge Flow Manager.


Configure Your Remote Process Group to Send Data to Your NiFi Cluster

Monitor Your Agents From the Events Screen

Let's grab all the new images and then delete on completion


We have Input and Output Ports to have Bidirectional communication with 0-n MiNiFi agents

Our NiFi flow to process calls from MiNiFi Agents running Coral TPUs

We run a query to check the TensorFlow Lite classification results and send out a slack message.


Let's push JSON data to a Kafka Cluster in AWS




Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi

Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi Yes, we have bidirectional communication with MiNiFi agents from Apache NiFi via Site-to-Site (S2S) over HTTPS.   This means I can push in anything I want to the agent, including commands, files and updates.
I can also transmit data to edge agents via MQTT, REST and Kafka amongst other options.

NiFi Ready To Send and Receive Messages From Other NiFi Nodes, Clusters and MiNiFi Agents

Our NiFi flow is consuming Kafka and MQTT Messages, as well as reading updated model files and generating integration test sensor data.


MiNiFi Agents Have Downloaded The Model and Anything Else We Send to It


It's Easy to Configure MQTT Message Consumption in CEM, we just need the broker (with port) and a topic to filter on if you wish.



To Listen For Files/Models You can easily add a REST End Point to Proxy in Data of Your Choice with or without SSL

Here's an example CURL script to test that REST API:
curl -d '{"key1&q…

Generating JSON Data in Apache NiFi

Example of a JSON IoT Generator
{
   "ip": ${ip()},
   "unique_id": "${UUID()}",
   "thread": "${thread()}",
   "hostname": "${hostname()}",
   "sensor_9": ${random():mod(100)},
   "sensor_id": ${random():mod(30)},
   "sensor_3": ${random():mod(50)},
   "sensor_2": ${random():mod(500)},
   "sensor_1": ${random():mod(110)},
   "sensor_0": ${random():mod(150)},
   "sensor_7": ${random():mod(255)},
   "sensor_6": ${random():mod(95)},
   "sensor_5": ${random():mod(80)},
   "sensor_ts": ${now():toNumber()},
   "sensor_8": ${random():mod(120)},
   "sensor_4": ${random():mod(60)},
   "sensor_11": ${random():mod(20)},
   "sensor_10": ${random():mod(10)}
}

Resources
https://www.datainmotion.dev/2019/04/integration-testing-for-apache-nifi.htmlhttps://medium.com/hashmapinc/its-here-an-apac…

Using Cloudera Streams Messaging Manager for Apache Kafka Monitoring, Management, Analytics and CRUD

Using Cloudera Streams Messaging Manager for Apache Kafka Monitoring, Management, Analytics and CRUD SMM is powerful tool to work with Apache Kafka and provide you with monitoring, management, analytics and creating Kafka topics.   You will be able to monitor servers, brokers, consumers, producers, topics and messages.   You will also be able to easily build alerts based on various events that can occur with those entities.
From Cloudera Manager, we can now install and manage Kafka, SMM, NiFi and Hadoop services.

Let's create a Kafka topic, no command-line!




For a simple topic, we select Low size for replication factor of one and replica count of one.  We also set a cleanup policy of delete.


Let's create an alert.

For this one if the nifi-reader consumer group has a lag then send an email to me.

Let's browse our Kafka infrastructure in our AWS Cloudera Kafka cluster, so easy to navigate.


You can dive into a topic and see individual messages, see offsets, keys, values, times…

GTFS Real-time Streaming with Apache NiFi

GTFS Real-time Streaming with Apache NiFi
To facilitate ingesting GTFS Real-Time data, I have added a processor that converts GTFS (General Transit Feed Specification) formatted ProtoBuf data into JSON.   This is using the standard Google APIs to accomplish this.      You can see the Protocol Buffers schema here

We will be able to get data on trip updates including delays, service alerts including changed routes and vehicle positions which can have location and congestion information.   This is the same information that these public transit systems feed to Google Maps.

https://developers.google.com/transit/gtfs-realtime/


An Example NiFi Flow For Accessing GTFS Data


First we add my new NiFi Processor, which you can get as source and build the NAR.   Or download one of the release builds of my new NiFi NAR archive.   This is alpha level code done very quickly.   If you find bugs please report and suggest updates.





Once it is added to your canvas, you can change the name and scheduling.



EFM Series: Using MiNiFi Agents on Raspberry Pi 4 with Intel Movidius Neural Compute Stick 2, Apache NiFi and AI

EFM Series:   Using MiNiFi Agents on Raspberry Pi 4 with Intel Movidius Neural Compute Stick 2, Apache NiFi and AI



The good news is Raspberry Pi 4 can run MiNiFi Java Agents, Intel Movidius Neural Compute Stick 2 and AI libraries.   You can now use this 4GB of RAM device to run IoT with AI on the edge.
Flow From MiNiFi Agent Running OpenVino, SysLog Tail and Grabbing WebCam Images

Configure The Execution of OpenVino Python Applications on RPI 4


Events Returning from Raspberry Pi 4


Models Used Download model using downloader.

Githubhttps://github.com/tspannhw/minifi-rpi4-ncc2

DATE=$(date +"%Y-%m-%d_%H%M")
fswebcam -q -r 1280x720 --no-banner /opt/demo/images/$DATE.jpg

python3 /opt/intel/openvino/build/test.py /opt/demo/images/$DATE.jpg
SoftwareApache NiFiApache NiFi - MiNiFi AgentsTensorFlowOpenVinoPython 3FSWEBCAMOpenCV DNNPSUTIL Python Librariespip3 install getmac pip3 install psutil pip3 install --upgrade pip
pip3 install --upgrade setuptools
Intel Movidius Neural Compute Stick…

Find cacerts from Java JRE Lib Security for NiFi SSL

Find cacerts from Java JRE Lib Security for NiFi SSL For REST Calls
On NiFi 1.9.2+

On My Mac

/Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home/jre/lib/security/cacerts

Default Password is changeit
JKS
TLS



See:
https://stackoverflow.com/questions/11936685/how-to-obtain-the-location-of-cacerts-of-the-default-java-installation

On Mac Get the Directory

/usr/libexec/java_homeOn Linux 

ps -ef | grep -i java
readlink -f /usr/java/default/bin/java

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/lib/security/cacerts
Walk back from


/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/bin/java

Migrating Apache Flume Flows to Apache NiFi: Log to HDFS

Migrating Apache Flume Flows to Apache NiFi:  Log to HDFS

Article 7 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_9.html
Article 6 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html
Article 5 -
Article 4 - This Article 3 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html Article 2 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html Source Code:  https://github.com/tspannhw/flume-to-nifi

Example NiFi Flow to Read From List of Directories to HDFS



I am moving some Apache Flume flows over to Apache NiFi, this is the first one I am doing.    I am grabbing log files from a directory and pushing over to HDFS.   We had added some features beyond what was available in the original Flume flow.

We also can visually see it running and get access to real-time metrics and stats via Apa…