Data In Motion

While some lucky people were in DataWorkSummit Training, others of us were in the NoSQL Day.




After NoSQL Day's end party, it was time for meetups includings Apache NiFi and Apache Kafka sessions!    The Apache NiFi meetup was packed and had most of the Apache NiFi team on-site.















Tuesday May 21, 2019

NoSQL Day
Tracking Crime ... Phoenix/HBase/NiFi
https://www.slideshare.net/bunkertor/tracking-crime-as-it-occurs-with-apache-phoenix-apache-hbase-and-apache-nifi
Wednesday May 22, 2019
Expo Theatre 20 minute talk 1:35 pm -  Apache Deep Learning 202 https://www.slideshare.net/bunkertor/apache-deep-learning-202-washington-dc-dws-2019

Thursday May 23, 2019 <

JSON Batch to Single Row Phoenix I grabbed open data on Crime from Philly's Open Data (https://www.opendataphilly.org/dataset/crime-incidents), after a free sign up you get access to JSON crime data (https://data.phila.gov/resource/sspu-uyfa.json) You can grab individual dates or ranges for thousands of records. I wanted to spool each JSON record as a separate HBase row. With the flexibility of Apache NiFi 1.0.0, I can specify run times via cron or other familiar setup. This is my master flow. First I use GetHTTP to retrieve the SSL JSON messages, I split the records up and store them as RAW JSON in HDFS as well as send some of them via Email, format them for Phoenix SQL and store them in Phoenix/HBase. All with no coding and in a simple flow. For extra output, I can send them to Reimann server for monitoring. Setting up SSL for accessing HTTPS data like Philly Crime, require a little configuration and knowing what Java JRE you are using to run NiFi. You can run service nifi statu…

Using CEM - Adding a Processor to a Flow
Looking at Events From CEM

Designing a Java Flow

Configure A Stream Execution

Event Details


Example Apache NiFi Receiver 

CEM Design - Open Flow Screen

Configure a PutFile Processor



If you want to revert your current changes to a previous version

     An Example Flow Java Agent




An Example CPP Flow



Example of Data received in NiFi from CPP Agent