Data In Motion

Automating the Building, Migration, Backup, Restore and Testing of Streaming Applications
One of the main things you will want to add to your flows as you restore them from backup or migrate them between clusters is apply appropriate parameters.So you can import the parameter contexts and then connect them to the correct process group(s).nifi-toolkit-1.12.0/bin/cli.sh nifi import-param-context -u http://edge2ai-1.dim.local:8080 -i parameter.jsonNote, values can be encrypted so the NiFi Operator or Developer doesn't have to see keys or protected values.
See an example script:https://github.com/tspannhw/ApacheConAtHome2020/blob/main/scripts/setupnifi.shResourceshttps://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.htmlhttps://www.datainmotion.dev/2020/09/devops-working-with-parameter-contexts.html

Monitoring Mac Laptops With Apache NiFi and osquery
The other way is pass a SQL query to osquery interpreter (ala osqueryi --json "SELECT * FROM $1") and get the query results back as JSON.
We can tail the main file (/var/log/osquery/osqueryd.results.log) and send the JSON to be used at scale as events.  We can also grab any and all osquery logs like INFO, WARN and ERROR via osquery.+.


Either download or brew cask install.    https://osquery.readthedocs.io/en/2.11.2/installation/install-osx/I setup a simple configuration here: (https://github.com/tspannhw/nifi-osquery){"options": {"config_plugin": "filesystem","logger_plugin": "filesystem","logger_path": "/var/log/osquery","disable_logging": "false","disable_events": "false","database_path": "/var/osquery/osquery.db","utc": "true"},
"schedule": {"system_info": …

Tracking Satellites with Apache NiFiThanks to https://www.n2yo.com/ for awesome data feeds.
Again, these types of ingests are so easy in Apache NiFi.   
Step 1, schedule when we want these.   There is a limit of 1,000 calls per hour, so let's keep it to 4 calls a minute for each of the three REST end points.

Let's get satellite information on right above me.
We set parameters for:   your latitude, your longitude, your apikey and then just change up bits of the REST URL.   Note for this one we are using SSL, so make sure you have an SSL context.





Now we have three streams of JSON data that has lat and long, so we can plot this on a map with Cloudera Visual Apps, storing our data in Impala tables in Kudu.
Some example data:
{  "info" : {    "satname" : "SPACE STATION",    "satid" : 25544,    "transactionscount" : "5"  },  "positions" : [ {    "satlatitude" : 37.46839338,    "satlongitude" : 95.127…

 Unveiling the NVIDIA Jetson Nano 2GB and Other NVIDIA GTC 2020 Announcements 





NVIDIA Jetson Nano 2GB Press Releasehttps://nvidianews.nvidia.com/news/nvidia-unveils-jetson-nano-2gb-the-ultimate-ai-and-robotics-starter-kit-for-students-educators-robotics-hobbyists

I have given this one a test run, it has all the features you like for Jetson, with just 2 GB less RAM and 2 less USB ports.   This is a very affordable device to do cool apps.
128-core NVIDIA MaxwellTM 64-bit Quad-core ARM A57 (1.43 GHz) 2 GB 64-bit LPDDR4 (25.6 GB/s bandwidth)Gigabit Ethernet1x USB 3.0 Type A ports, 2x USB 2.0 Type A ports, 1x USB 2.0 Micro-BHDMIWiFiGPIOs, I2C, I2S, SPI, PWM, UART1x MIPI CSI-2 connectorMicroSD Connector12-pin header (Power and related signals, UART)100mm x 80mm x 29mm
USB-C Port for Power Depending where you or or how you buy the package you may need to buy a power supply and USB WiFi. All of my existing workloads have been working fine in the 2GB version, but with a very nice cost saving.  The se…

DevOps:  Working with Parameter Contexts in Apache NiFi 1.11.4+nifi list-param-contexts -u http://localhost:8080 -ot simple
# Id Name Description- ------------------------------------ -------------- -----------1 3a801ff4-1f73-1836-b59c-b9fbc79ab030 backupregistry2 7184b9f4-0171-1000-4627-967e118f3037 health3 3a801faf-1f87-1836-54ba-3d913fa223ad retail4 3a801fde-1f73-1836-957b-a9f4d2c9b73d sensors
#> nifi export-param-context -u http://localhost:8080 -verbose --paramContextId 3a801faf-1f87-1836-54ba-3d913fa223ad
{"name" : "retail","description" : "","parameters" : [ {"parameter" : {"name" : "allquery","description" : "","sensitive" : false,"value" : "SELECT * FROM FLOWFILE"}}, {"parameter" : {"name" : "allrecordssql",

Setup a Google Developers Account'
Use or Create an API Key For Sheets at Developer Console
For Your Google Sheet (If not OAuth, You Need to Make it Visible via URL)Or you will face PERMISSION_DENIED
Enable Google Sheets APIhttps://console.developers.google.com/apis/api/sheets.googleapis.com/overview?project=YOURPROJECTID
View Metricshttps://console.developers.google.com/apis/api/sheets.googleapis.com/overview?project=YOURPROJECTISCOOLAccess The Data Via NIFIhttps://sheets.googleapis.com/v4/spreadsheets/YOURGOOGLESHEET?includeGridData=true&key=YOURKEYReferences:https://community.cloudera.com/t5/Community-Articles/Streaming-Ingest-of-Google-Sheets-with-HDF-2-0/ta-p/247764

Using DJL.AI For Deep Learning BERT Q&A in NiFi DataFlows
Introduction:I will be talking about this processor at Apache Con @ Home 2020 in my "Apache Deep Learning 301" talk with Dr. Ian Brooks.Sometimes you want your Deep Learning Easy and in Java, so let's do that with DJL in a custom Apache NiFi processor running in CDP Data Hubs.   This one does BERT QA.
To use the processor feed in a paragraph to analyze via the paragraph parameter in the NiFi processor.   Also feed in a question, like Why? or something very specific like asking the date or an event.
The pretrained model is BERT QA model using PyTorch. the NiFi Processor Source:https://github.com/tspannhw/nifi-djlqa-processor
Grab the Recent Release NAR to install to your NiFi lib directories:https://github.com/tspannhw/nifi-djlqa-processor/releases/tag/1.2
Example Run



Demo Data Sourcehttps://newsapi.org/v2/everything?q=cloudera&apiKey=REGISTERFORAKEY


Reference:Deep Learning Sentiment Analysis with DJL.aihttps://gi…