Data In Motion

Using NiFi CLI to Restore NiFi Flows From Backups

Please note, Apache NiFi 1.11.4 is now available for download. 

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.4

References:

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.1https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/comment-page-1/https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI

#> registry list-buckets -u http://somesite.compute-1.amazonaws.com:18080

#   Name   Id                                     Description 
-   ----   ------------------------------------   ----------- 
1   IOT    45834964-d022-4f4c-891f-695898e1e5f0   (empty)     
2   IoT    250a5ae5-ced8-4f4e-8b3b-01eb9d47a0d9   (empty)     
3   dev    46b7bab7-400f-44ae-a0e6-7340ff19c96f   (empty)     
4   iot    c594d6bc-7413-4f6a-ba9a-50b8020eec37   (empty)     
5   prod   0bf59d2e-1dd5-4d24-8aa0-0614bf991…

v4l2-ctl --list-devices v4l2-ctl -d /dev/video0 --list-ctrls v4l2-ctl --get-ctrl=white_balance_temperature v4l2-ctl --set-ctrl=white_balance_temperature=4000 v4l2-ctl --set-ctrl=white_balance_temperature=4000 -d /dev/video0 v4l2-ctl --set-ctrl=white_balance_temperature_auto=1 v4l2-ctl --set-ctrl=white_balance_temperature_auto=0 v4l2-ctl --set-ctrl=white_balance_temperature_auto=4000 v4l2-ctl --set-ctrl=exposure_auto=3 v4l2-ctl --set-ctrl=exposure_auto_priority=0 v4l2-ctl --set-ctrl=exposure_absolute=250 v4l2-ctl --set-ctrl=exposure_absolute=0 v4l2-ctl --set-ctrl=exposure_absolute=250 v4l2-ctl --set-ctrl=gain=0 v4l2-ctl -d /dev/video0 --list-ctrls v4l2-ctl --set-ctrl=white_balance_temperature_auto=4000 v4l2-ctl --set-ctrl=white_balance_temperature_auto=0 v4l2-ctl --set-ctrl=white_balance_temperature=4000 v4l2-ctl -d /dev/video0 --list-ctrls


This article is great:   https://www.kurokesu.com/main/2016/01/16/manual-usb-camera-settings-in-linux/


v4l2-ctl -d /dev/video0 --list-ctrls brightness 0x00980900 (int)…

OpenDS4All


Start learning now:
https://github.com/odpi/OpenDS4All/tree/master/opends4all-resources/opends4all-data-wrangling-and-integration
ODPI has officially announced this recently and it looks great. https://www.odpi.org/news/2020/02/28/odpi-announces-the-opends4all-project
There is a ton of amazing materials including slides, notes, documentation, homework, exercises and Jupyter notebooks covering Data Wrangling, Data Science, the Basics and Apache Spark.   


This“starter set” of training materials can help you build a Data Science program for yourself, your company, your university or your non-profit.    I am going to bring some of these to my meetups and hopefully can help give back with new materials, updates and suggestions.
These are college level materials developed by the University of Pennsylvania and open source via the ODPI with IBM leading.   The code and slides look great.   I can see these helping to enable the world adding another million desperately needed Data Scient…

Apache NiFi 1.11.3
http://nifi.apache.org/download.html

If you have downloaded anything after NiFi 1.10, please upgrade now.   This has some major improvements and some fixes.

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.3

I am running this now in Anaheim, and it's no Mickey Mouse upgrade.   It's fast and nice.

Some of the more recent upgrades:

https://www.datainmotion.dev/2019/11/exploring-apache-nifi-110-parameters.html

For parameters and stateless, and ability to download a flow as JSON is worth the price of install.



See some more NiFi 1.11 features here:   https://www.datainmotion.dev/2020/02/edgeai-google-coral-with-coral.html

EdgeAI:   Google Coral with Coral Environmental Sensors and TPU With NiFi and MiNiFi
Building MiNiFi IoT Apps with the new Cloudera EFM 

It is very easy to build a drag and drop EdgeAI application with EFM and then push to all your MiNiFi agents.

Cloudera Edge Management CEM-1.1.1
Download the newest CEM today! https://www.cloudera.com/downloads/cdf/cem.html https://docs.cloudera.com/cem/1.1.1/release-notes/topics/cem-whats-new.html








NiFi Flow Receiving From MiNiFi Java Agent

In a cluster in my CDP-DC Cluster I consume Kafka messages sent from my remote NiFi gateway to publish alerts to Kafka and push records to Apache HBase and Apache Kudu.  We filter our data with Streaming SQL.

We can use SQL to route, create aggregates like averages, chose a subset of fields and limit data returned.   Using the power of Apache Calcite, Streaming SQL in NiFi is a game changer against Record Data Types including CSV, XML, Avro, Parquet, JSON and Grokable text.   Read and write different formats and convert …