Skip to main content

Posts

Showing posts from August, 2019

Google Coral TPU with Edge Devices and MiNiFi

Google Coral TPU with Edge Devices and MiNiFi  Designing Our Edge AI Flow with Cloudera Edge Flow Manager. Configure Your Remote Process Group to Send Data to Your NiFi Cluster Monitor Your Agents From the Events Screen Let's grab all the new images and then delete on completion We have Input and Output Ports to have Bidirectional communication with 0-n MiNiFi agents Our NiFi flow to process calls from MiNiFi Agents running Coral TPUs We run a query to check the TensorFlow Lite classification results and send out a slack message. Let's push JSON data to a Kafka Cluster in AWS

Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi

Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi Yes, we have bidirectional communication with MiNiFi agents from Apache NiFi via Site-to-Site (S2S) over HTTPS.   This means I can push in anything I want to the agent, including commands, files and updates. I can also transmit data to edge agents via MQTT, REST and Kafka amongst other options. NiFi Ready To Send and Receive Messages From Other NiFi Nodes, Clusters and MiNiFi Agents Our NiFi flow is consuming Kafka and MQTT Messages, as well as reading updated model files and generating integration test sensor data. MiNiFi Agents Have Downloaded The Model and Anything Else We Send to It It's Easy to Configure MQTT Message Consumption in CEM, we just need the broker (with port) and a topic to filter on if you wish. To Listen For Files/Models You can easily add a REST End Point to Proxy in Data of Your Choice with or without SSL Here's an example CU

Generating JSON Data in Apache NiFi

Example of a JSON IoT Generator {    "ip": ${ip()},    "unique_id": "${UUID()}",    "thread": "${thread()}",    "hostname": "${hostname()}",    "sensor_9": ${random():mod(100)},    "sensor_id": ${random():mod(30)},    "sensor_3": ${random():mod(50)},    "sensor_2": ${random():mod(500)},    "sensor_1": ${random():mod(110)},    "sensor_0": ${random():mod(150)},    "sensor_7": ${random():mod(255)},    "sensor_6": ${random():mod(95)},    "sensor_5": ${random():mod(80)},    "sensor_ts": ${now():toNumber()},    "sensor_8": ${random():mod(120)},    "sensor_4": ${random():mod(60)},    "sensor_11": ${random():mod(20)},    "sensor_10": ${random():mod(10)} } Resources https://www.datainmotion.dev/2019/04/integration-testing-for-apache-nifi.html https://medium

Using Cloudera Streams Messaging Manager for Apache Kafka Monitoring, Management, Analytics and CRUD

Using Cloudera Streams Messaging Manager for Apache Kafka Monitoring, Management, Analytics and CRUD SMM is powerful tool to work with Apache Kafka and provide you with monitoring, management, analytics and creating Kafka topics.   You will be able to monitor servers, brokers, consumers, producers, topics and messages.   You will also be able to easily build alerts based on various events that can occur with those entities. From Cloudera Manager, we can now install and manage Kafka, SMM, NiFi and Hadoop services. Let's create a Kafka topic, no command-line! For a simple topic, we select Low size for replication factor of one and replica count of one.  We also set a cleanup policy of delete . Let's create an alert. For this one if the nifi-reader consumer group has a lag then send an email to me. Let's browse our Kafka infrastructure in our AWS Cloudera Kafka cluster, so easy to navigate. You can dive into a t

GTFS Real-time Streaming with Apache NiFi

GTFS Real-time Streaming with Apache NiFi To facilitate ingesting GTFS Real-Time data, I have added a processor that converts GTFS (General Transit Feed Specification) formatted ProtoBuf data into JSON.   This is using the standard Google APIs to accomplish this.      You can see the Protocol Buffers schema here .  We will be able to get data on trip updates including delays, service alerts including changed routes and vehicle positions which can have location and congestion information.   This is the same information that these public transit systems feed to Google Maps. https://developers.google.com/transit/gtfs-realtime/ An Example NiFi Flow For Accessing GTFS Data First we add my new NiFi Processor, which you can get as source and build the NAR.   Or download one of the release builds of my new NiFi NAR archive.   This is alpha level code done very quickly.   If you find bugs please report and suggest updates. Once it is added to your canvas, you can change th

EFM Series: Using MiNiFi Agents on Raspberry Pi 4 with Intel Movidius Neural Compute Stick 2, Apache NiFi and AI

EFM Series:   Using MiNiFi Agents on Raspberry Pi 4 with Intel Movidius Neural Compute Stick 2, Apache NiFi and AI The good news is Raspberry Pi 4 can run MiNiFi Java Agents, Intel Movidius Neural Compute Stick 2 and AI libraries.   You can now use this 4GB of RAM device to run IoT with AI on the edge. Flow From MiNiFi Agent Running OpenVino, SysLog Tail and Grabbing WebCam Images Configure The Execution of OpenVino Python Applications on RPI 4 Events Returning from Raspberry Pi 4 Models Used Download model using downloader. Github https://github.com/tspannhw/minifi-rpi4-ncc2 DATE=$(date +"%Y-%m-%d_%H%M") fswebcam -q -r 1280x720 --no-banner /opt/demo/images/$DATE.jpg python3 /opt/intel/openvino/build/test.py /opt/demo/images/$DATE.jpg Software Apache NiFi Apache NiFi - MiNiFi Agents TensorFlow OpenVino Python 3 FSWEBCAM OpenCV DNN PSUTIL Python Libraries pip3 install getmac pip3 install psutil pip3 install --u

Find cacerts from Java JRE Lib Security for NiFi SSL

Find cacerts from Java JRE Lib Security for NiFi SSL For REST Calls On NiFi 1.9.2+ On My Mac /Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home/jre/lib/security/cacerts Default Password is changeit JKS TLS See: https://stackoverflow.com/questions/11936685/how-to-obtain-the-location-of-cacerts-of-the-default-java-installation On Mac Get the Directory / usr / libexec / java_home On Linux  ps -ef | grep -i java readlink -f /usr/java/default/bin/java /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/lib/security/cacerts   Walk back from   /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/bin/java

Migrating Apache Flume Flows to Apache NiFi: Log to HDFS

Migrating Apache Flume Flows to Apache NiFi:  Log to HDFS Article 7 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_9.html Article 6 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html Article 5 - Article 4 - This Article 3 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html Article 2 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html Article 1  -  https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html Source Code:   https://github.com/tspannhw/flume-to-nifi Example NiFi Flow to Read From List of Directories to HDFS I am moving some Apache Flume flows over to Apache NiFi, this is the first one I am doing.    I am grabbing log files from a directory and pushing over to HDFS.   We had added some features beyond what was available in the original Flume flow. We also can visually see it running and get acces