Skip to main content

Posts

Showing posts with the label s2s

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest to Parquet, Kudu, ORC, Kafka, Hive and Impala

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest Source Code:    https://github.com/tspannhw/minifi-grove-sensors Acquiring sensor data from Grove sensors is easy using a GrovePi Hat and some compatible sensors. Just before my talk at the Future of Data Meetup @ Bell Works in Holmdel, NJ, I thought I should ingest some data from a grove sensor interface. It's so easy a sleeping cat could do it. So what does this device look like?   I have a temperature and humidity sensor on there. The distance sonic sensor is in there too, that's for the next article. Let's do this with minimal RAM. That's a 64GB hard drive underneath in the white case with the RPI. I need more data and BACON. We design our MiNiFi Agent Flow in CEM/EFM.   Grab JSON data stream and run sensors. Apache NiFi 1.9.2 / CFM 1.0 Received HTTPS S2S Events From MiNiFi Agent A simple flow to query and convert our

Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi

Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi Yes, we have bidirectional communication with MiNiFi agents from Apache NiFi via Site-to-Site (S2S) over HTTPS.   This means I can push in anything I want to the agent, including commands, files and updates. I can also transmit data to edge agents via MQTT, REST and Kafka amongst other options. NiFi Ready To Send and Receive Messages From Other NiFi Nodes, Clusters and MiNiFi Agents Our NiFi flow is consuming Kafka and MQTT Messages, as well as reading updated model files and generating integration test sensor data. MiNiFi Agents Have Downloaded The Model and Anything Else We Send to It It's Easy to Configure MQTT Message Consumption in CEM, we just need the broker (with port) and a topic to filter on if you wish. To Listen For Files/Models You can easily add a REST End Point to Proxy in Data of Your Choice with or without SSL Here's an example CU