Data In Motion

Analyzing Wood Burning Stoves with FLaNK Stack Part 2 - Analytics - Part 2Part 1: https://www.datainmotion.dev/2020/01/analyzing-wood-burning-stoves-with.html

See:   https://shop.pimoroni.com/products/sgp30-air-quality-sensor-breakout
Sensiron SGP30 TVOC and eCO2 sensorTVOC sensing from 0-60,000 ppb (parts per billion)CO2 sensing from 400 to 60,000 ppm (parts per million)Running the fire I can see I am getting higher CO2 production than normal.
Since I stored my data in Kudu tables, it's easy to analyze with Impala and Hue.

select equivalentco2ppm, totalvocppb, systemtime from gassensors order by equivalentco2ppm desc


select avg( cast(  equivalentco2ppm as double) ) CO2PPM from gassensors
The average was 493.
Now that we have some time series data, I can start feeding this to some standard machine learning algorithms and have CML and a Data Scientist if me some analytics and help me determine where I a may want an alert.
When should you worry:   https://www.co2meter.com/blogs/news/5678873…

Analyzing Wood Burning Stoves with FLaNK Stack:   MiNiFi, Flink, NiFi, Kafka, Kudu (FLaNK Stack)
Winter has arrived, finally.   The 50-70 F days are over, it dropped below 30 F in Princeton, so time to light up the wood burning stove and burn some season cherry wood (We get cherry wood from a local tree service that removes dead trees for people and then season the wood.  Recycle!) .  It's great for camp fires, smoking meats and for heating up our house.  Also if you have no smelled cherry wood smoke it is amazing.   I wanted to see if having a fire that raised my houses temperature from 67 F to 87 F would produce noticeable sensor readings.   Fortunately, I have a thermal camera sensor (Pimoroni rocks! Add another thing to my list of thinks I love from Britain (Dr. Who, Jelly Babies, Pimoroni and my awesome boss Dan).  I also have Raspberry Pi sensors for temperature, humidity, light and various gas sensors.   Let's see what the numbers look like.    The temperatures and ima…

Writing a Custom Kafka Rest Proxy in 4 Hours
A custom processor for using NiFi as a REST Proxy to Kafka is very easy.  So I made one in NiFi 1.10.   It's a simple Kafka smart client that accepts POSTs, GETs or whatever HTTP request and returns a message from a Kafka topic, topic can be set via variables, HTTP request or your choice.   To get on of the query parameters, you do so like this:   ${http.query.param.topic}.    I am using the plain old KafkaConsumer class. 



If you need data, just CURL it or use your HTTP/REST client controls in Java, Python, Go, Scala, Ruby, C#, VB.NET or whatever.


curl http://localhost:9089?topic=bme680
Download a pre-built NAR and install.   Note:   this is a pre-release Alpha that I quickly built and tested on a few clusters.   This is not an official project or product.   This is a POC for myself to see how hard it could it be.   It's not!   Roll your own or join me in building out an open source project for one.   What requirements do you have?…

MiNiFi with Thermal Cameras and Air Quality Sensors Cloudera Edge2AI:  MiNiFi Java Agent with Raspberry Pi and Thermal Camera and Air Quality Sensor - Part 1Use Case / Overview:

We need to track heat signatures, web camera images, gas and other readings from a remote workers office.   This is for occupancy analytics and safety monitoring.   We can extend this to field and remote sites where levels of temperatures, movements, noxious gases and other real world items may cause risk to our staff.

For tracking room temperature and humidity we have other sensors:   https://www.datainmotion.dev/2019/12/iot-series-minifi-agent-on-raspberry-pi.html
https://www.datainmotion.dev/2019/10/using-grovepi-with-raspberry-pi-and.html
https://www.datainmotion.dev/2019/09/powering-edge-ai-for-sensor-reading.html
https://www.datainmotion.dev/2019/12/easy-deep-learning-in-apache-nifi-with.html

We can also track things like GPS, overhead plane traffic, regionalized social media reports, news, government repo…