Skip to main content

Posts

Showing posts with the label minifi

IoT Series: MiNiFi Agent on Raspberry Pi 4 with Enviro+ Hat For Environmental Monitoring and Analytics

IoT Series:  MiNiFi Agent on Raspberry Pi 4 with Enviro+ Hat For Environmental Monitoring and Analytics Summary :  Our powerful edge device streams sensor readings for environmental readings while also performing edge analytics with deep learning libraries and enhanced edge VPU.   We can perform complex running calculations on sensor data locally on the box before making potentially expense network calls.  We can also decide when to send out data based on heuristics, machine learning or simple if-then logic. Use Case:    Monitor Environment.   Act Local, Report Global. Stack :   FLANK Category :   AI, IoT, Edge2AI, Real-Time Streaming, Sensors, Cameras, Telemetry. Hardware :  Intel Movidius NCC 2 VPU (Neural Computing), Pimoroni Enviro Plus pHAT, Raspberry Pi 4 (4GB Edition). Software :  Python 3 + Java + MiNiFi Agents + Cloudera Edge Flow Manager (EFM/CEM) + Apache NiFi.   Using Mm... FLaNK Stack. Keywords :  Edge2AI, CV, AI, Deep Learning, Cloudera, NiFi, Ras

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest to Parquet, Kudu, ORC, Kafka, Hive and Impala

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest Source Code:    https://github.com/tspannhw/minifi-grove-sensors Acquiring sensor data from Grove sensors is easy using a GrovePi Hat and some compatible sensors. Just before my talk at the Future of Data Meetup @ Bell Works in Holmdel, NJ, I thought I should ingest some data from a grove sensor interface. It's so easy a sleeping cat could do it. So what does this device look like?   I have a temperature and humidity sensor on there. The distance sonic sensor is in there too, that's for the next article. Let's do this with minimal RAM. That's a 64GB hard drive underneath in the white case with the RPI. I need more data and BACON. We design our MiNiFi Agent Flow in CEM/EFM.   Grab JSON data stream and run sensors. Apache NiFi 1.9.2 / CFM 1.0 Received HTTPS S2S Events From MiNiFi Agent A simple flow to query and convert our

Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi

Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi Yes, we have bidirectional communication with MiNiFi agents from Apache NiFi via Site-to-Site (S2S) over HTTPS.   This means I can push in anything I want to the agent, including commands, files and updates. I can also transmit data to edge agents via MQTT, REST and Kafka amongst other options. NiFi Ready To Send and Receive Messages From Other NiFi Nodes, Clusters and MiNiFi Agents Our NiFi flow is consuming Kafka and MQTT Messages, as well as reading updated model files and generating integration test sensor data. MiNiFi Agents Have Downloaded The Model and Anything Else We Send to It It's Easy to Configure MQTT Message Consumption in CEM, we just need the broker (with port) and a topic to filter on if you wish. To Listen For Files/Models You can easily add a REST End Point to Proxy in Data of Your Choice with or without SSL Here's an example CU