Skip to main content

Posts

Using NiFi CLI to Restore NiFi Flows From Backups

Using NiFi CLI to Restore NiFi Flows From Backups

Please note, Apache NiFi 1.11.4 is now available for download

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.4

References:

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.1https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/comment-page-1/https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI

#> registry list-buckets -u http://somesite.compute-1.amazonaws.com:18080

#   Name   Id                                     Description 
-   ----   ------------------------------------   ----------- 
1   IOT    45834964-d022-4f4c-891f-695898e1e5f0   (empty)     
2   IoT    250a5ae5-ced8-4f4e-8b3b-01eb9d47a0d9   (empty)     
3   dev    46b7bab7-400f-44ae-a0e6-7340ff19c96f   (empty)     
4   iot    c594d6bc-7413-4f6a-ba9a-50b8020eec37   (empty)     
5   prod   0bf59d2e-1dd5-4d24-8aa0-0614bf991…

Fixing Linux Webcams

v4l2-ctl --list-devices v4l2-ctl -d /dev/video0 --list-ctrls v4l2-ctl --get-ctrl=white_balance_temperature v4l2-ctl --set-ctrl=white_balance_temperature=4000 v4l2-ctl --set-ctrl=white_balance_temperature=4000 -d /dev/video0 v4l2-ctl --set-ctrl=white_balance_temperature_auto=1 v4l2-ctl --set-ctrl=white_balance_temperature_auto=0 v4l2-ctl --set-ctrl=white_balance_temperature_auto=4000 v4l2-ctl --set-ctrl=exposure_auto=3 v4l2-ctl --set-ctrl=exposure_auto_priority=0 v4l2-ctl --set-ctrl=exposure_absolute=250 v4l2-ctl --set-ctrl=exposure_absolute=0 v4l2-ctl --set-ctrl=exposure_absolute=250 v4l2-ctl --set-ctrl=gain=0 v4l2-ctl -d /dev/video0 --list-ctrls v4l2-ctl --set-ctrl=white_balance_temperature_auto=4000 v4l2-ctl --set-ctrl=white_balance_temperature_auto=0 v4l2-ctl --set-ctrl=white_balance_temperature=4000 v4l2-ctl -d /dev/video0 --list-ctrls


This article is great:   https://www.kurokesu.com/main/2016/01/16/manual-usb-camera-settings-in-linux/


v4l2-ctl -d /dev/video0 --list-ctrls brightness 0x00980900 (int)…

ODPI's OpenDS4All - Open Source Data Science Content To Teach the World

OpenDS4All


Start learning now:
https://github.com/odpi/OpenDS4All/tree/master/opends4all-resources/opends4all-data-wrangling-and-integration
ODPI has officially announced this recently and it looks great. https://www.odpi.org/news/2020/02/28/odpi-announces-the-opends4all-project
There is a ton of amazing materials including slides, notes, documentation, homework, exercises and Jupyter notebooks covering Data Wrangling, Data Science, the Basics and Apache Spark.   


This“starter set” of training materials can help you build a Data Science program for yourself, your company, your university or your non-profit.    I am going to bring some of these to my meetups and hopefully can help give back with new materials, updates and suggestions.
These are college level materials developed by the University of Pennsylvania and open source via the ODPI with IBM leading.   The code and slides look great.   I can see these helping to enable the world adding another million desperately needed Data Scient…

New and Improved: It's NiFi

Apache NiFi 1.11.3
http://nifi.apache.org/download.html

If you have downloaded anything after NiFi 1.10, please upgrade now.   This has some major improvements and some fixes.

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.3

I am running this now in Anaheim, and it's no Mickey Mouse upgrade.   It's fast and nice.

Some of the more recent upgrades:

https://www.datainmotion.dev/2019/11/exploring-apache-nifi-110-parameters.html

For parameters and stateless, and ability to download a flow as JSON is worth the price of install.



See some more NiFi 1.11 features here:   https://www.datainmotion.dev/2020/02/edgeai-google-coral-with-coral.html


EdgeAI: Google Coral with Coral Environmental Sensors and TPU With NiFi and MiNiFi (Updated EFM)

EdgeAI:   Google Coral with Coral Environmental Sensors and TPU With NiFi and MiNiFi
Building MiNiFi IoT Apps with the new Cloudera EFM 

It is very easy to build a drag and drop EdgeAI application with EFM and then push to all your MiNiFi agents.

Cloudera Edge Management CEM-1.1.1
Download the newest CEM today! https://www.cloudera.com/downloads/cdf/cem.html https://docs.cloudera.com/cem/1.1.1/release-notes/topics/cem-whats-new.html








NiFi Flow Receiving From MiNiFi Java Agent

In a cluster in my CDP-DC Cluster I consume Kafka messages sent from my remote NiFi gateway to publish alerts to Kafka and push records to Apache HBase and Apache Kudu.  We filter our data with Streaming SQL.

We can use SQL to route, create aggregates like averages, chose a subset of fields and limit data returned.   Using the power of Apache Calcite, Streaming SQL in NiFi is a game changer against Record Data Types including CSV, XML, Avro, Parquet, JSON and Grokable text.   Read and write different formats and convert …

Connecting Apache NiFi to Apache Atlas For Data Governance At Scale in Streaming

Connecting Apache NiFi to Apache Atlas For Data Governance At Scale in Streaming
Once connected you can see NiFi and Kafka flowing to Atlas.

You must add AtlasReport to NiFi cluster.



Add a ReportLineageToAtlas under Controller Settings / Reporting Tasks You must add URL for Atlas, Authentication method and if basic, username/password.




You need to set Atlas Configuration directory, NiFi URL to use, Lineage Strategy - Complete Path

Another example with an AWS hosted NiFi and Atlas:



You can now see the lineage state:



Configure Atlas to Be Enabled and Have Kafka

Have Atlas Service enabled in NiFi configuration


Example Configuration

You must have access to Atlas Application Properties.

/etc/atlas/conf

atlas-application.properties
#Generated by Apache NiFi ReportLineageToAtlas ReportingTask at 2020-02-21T17:18:28.493Z #Fri Feb 21 17:18:28 UTC 2020 atlas.kafka.bootstrap.servers=princeton0.field.hortonworks.com\:9092 atlas.enableTLS=false atlas.kafka.client.id=ReportLineageToAtlas.687a48e2-0170-1000-0000-00…

Example SMM Notification Email

Example SMM Notification Email
Notification id: 12f61ec2-11a3-45ba-b7bb-2416d8a1b076,
Root resource name: ANY,
Root resource type: CONSUMER,
Created timestamp: Tue Jan 07 21:13:45 UTC 2020 : 1578431625199,Last updated timestamp: Mon Jan 13 13:09:38 UTC 2020 : 1578920978293,
State: RAISED,

Message:Alert policy : "ALERT IF ( ANY CONSUMER MILLISECONDS_LAPSED_SINCE_CONSUMER_WAS_ACTIVE >= 1200 )" has been evaluated to true Condition : "MILLISECONDS_LAPSED_SINCE_CONSUMER_WAS_ACTIVE>=1200" has been evaluated to true for following CONSUMERS - CONSUMER = "tensorflow-nifi-aws-client" had following attribute values * MILLISECONDS_LAPSED_SINCE_CONSUMER_WAS_ACTIVE = 308208428 - CONSUMER = "atlas" had following attribute values * MILLISECONDS_LAPSED_SINCE_CONSUMER_WAS_ACTIVE = 596819269 - CONSUMER = "nifi-gassensor-aws-client" had following attribute values * MILLISECONDS_LAPSED_SINCE_CONSUMER_WAS_ACTIVE = 310692570 - CONSUMER = "NIFI-TEST-GROU…