Data In Motion

From Vasilis Vagias: evaluateJsonPath immediately after the InvokeHTTP and replace the flowfile content with the $.response then NiFi unescapes and removes the additional quotes auto magically. This is helpful for occasions when CDSW returns JSON or other REST APIs which may double encode JSON files.

QuickTip:   Ingesting Google Analytics API with Apache NiFi  Design your query / test the API here: https://ga-dev-tools.appspot.com/query-explorer/ Building this NiFi flow is trivial. Add your URL with tokens from the Query Explorer console. You will need to reference the JRE that NiFi is using and it's cacerts if you don't want to build your own trust store.   The default password for JDK 8 is changeit .   No really. Here are our results in clean JSON Here are some attributes NiFi shows. Example JSON Results {   "kind": "analytics#gaData",   "id": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:33&metrics=ga:users,ga:percentNewSessions,ga:sessions&start-date=30daysAgo&end-date=yesterday",   "query": {     "start-date": "30daysAgo",     "end-date": "yesterday",     "ids": "ga:33",     &

Analyzing Wood Burning Stoves with FLaNK Stack Part 2 - Analytics - Part 2 Part 1:    https://www.datainmotion.dev/2020/01/analyzing-wood-burning-stoves-with.html See :    https://shop.pimoroni.com/products/sgp30-air-quality-sensor-breakout Sensiron SGP30 TVOC and eCO2 sensor TVOC sensing from 0-60,000 ppb (parts per billion) CO2 sensing from 400 to 60,000 ppm (parts per million) Running the fire I can see I am getting higher CO2 production than normal. Since I stored my data in Kudu tables, it's easy to analyze with Impala and Hue. select equivalentco2ppm, totalvocppb, systemtime from gassensors order by equivalentco2ppm desc select avg( cast(  equivalentco2ppm as double) ) CO2PPM from gassensors The average was 493. Now that we have some time series data, I can start feeding this to some standard machine learning algorithms and have CML and a Data Scientist if me some analytics and help me determine where I a may want an alert.

Analyzing Wood Burning Stoves with FLaNK Stack:   MiNiFi, Flink, NiFi, Kafka, Kudu (FLaNK Stack) Winter has arrived, finally.   The 50-70 F days are over, it dropped below 30 F in Princeton, so time to light up the wood burning stove and burn some season cherry wood (We get cherry wood from a local tree service that removes dead trees for people and then season the wood.  Recycle!) .  It's great for camp fires, smoking meats and for heating up our house.  Also if you have no smelled cherry wood smoke it is amazing.   I wanted to see if having a fire that raised my houses temperature from 67 F to 87 F would produce noticeable sensor readings.   Fortunately, I have a thermal camera sensor ( Pimoroni rocks! Add another thing to my list of thinks I love from Britain (Dr. Who, Jelly Babies, Pimoroni and my awesome boss Dan).  I also have Raspberry Pi sensors for temperature, humidity, light and various gas sensors.   Let's see what the numbers look like.    The temperatures a

Writing a Custom Kafka Rest Proxy in 4 Hours A custom processor for using NiFi as a REST Proxy to Kafka is very easy.  So I made one in NiFi 1.10.   It's a simple Kafka smart client that accepts POSTs, GETs or whatever HTTP request and returns a message from a Kafka topic, topic can be set via variables, HTTP request or your choice.   To get on of the query parameters, you do so like this:    ${http.query.param.topic}.    I am using the plain old KafkaConsumer class.  If you need data, just CURL it or use your HTTP/REST client controls in Java, Python, Go, Scala, Ruby, C#, VB.NET or whatever. curl http://localhost:9089?topic=bme680 Download a pre-built NAR and install.   Note:   this is a pre-release Alpha that I quickly built and tested on a few clusters.   This is not an official project or product.   This is a POC for myself to see how hard it could it be.   It's not!   Roll your own or join me in building out an open source project for one.   What re

MiNiFi with Thermal Cameras and Air Quality Sensors Cloudera Edge2AI:  MiNiFi Java Agent with Raspberry Pi and Thermal Camera and Air Quality Sensor - Part 1 Use Case / Overview: We need to track heat signatures, web camera images, gas and other readings from a remote workers office.   This is for occupancy analytics and safety monitoring.   We can extend this to field and remote sites where levels of temperatures, movements, noxious gases and other real world items may cause risk to our staff. For tracking room temperature and humidity we have other sensors:    https://www.datainmotion.dev/2019/12/iot-series-minifi-agent-on-raspberry-pi.html https://www.datainmotion.dev/2019/10/using-grovepi-with-raspberry-pi-and.html https://www.datainmotion.dev/2019/09/powering-edge-ai-for-sensor-reading.html https://www.datainmotion.dev/2019/12/easy-deep-learning-in-apache-nifi-with.html We can also track things like GPS, overhead plane traffic, regionalized social media reports, news