Skip to main content

Posts

Showing posts from November, 2019

Introducing Mm FLaNK... An Apache Flink Stack for Rapid Streaming Development From Edge 2 AI

Introducing Mm FLaNK...  An Apache Flink Stack for Rapid Streaming Development From Edge 2 AI



Source:   https://github.com/tspannhw/MmFLaNK

Stateless NiFi:https://www.datainmotion.dev/2019/11/exploring-apache-nifi-110-parameters.html

To show an example of using the Mm FLaNK stack we have an Apache NiFi flow that reads IoT data (JSON) and send it to Apache Kafka.   An Apache Flink streaming application running in YARN reads it, validates the data and send it to another Kafka topic.  We monitor and check the data with SMM.    The data from that second topic is read by Apache NiFi and pushed to Apache Kudu tables.

Mm FLaNK Stack (MXNet, MiNiFi, Flink, NiFi, Kafka, Kudu)



 First, we rapidly ingest, route, transform, convert, query and process data with Apache NiFi.   Once we have transformed it into a client, schema-validated known data type we can stream it to Kafka for additional processing.



Second, we read from that Kafka topic, iot, with a Java Apache Flink application.   We then filter …

Learning Apache Flink 1.9

Learning Apache Flink 1.9




















Everything is a Stream, important concept to remember when working with Apache Flink.

Submit a Job

https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html


flink run -m yarn-cluster -d -p 2 -ynm Wiki iot-1.0.jar Mm Flank I am working on a nice integration pattern with the Mm Flank Stack of Apache NiFi-MiniFi, MXNet, Flink, NiFi, Kafka and Kudu.
Flink Monitoring
http://princeton0.field.hortonworks.com:8082/config
{"refresh-interval":10000,"timezone-offset":0,"timezone-name":"Coordinated Universal Time","flink-version":"1.9.0-csa1.0.0.0","flink-revision":"3cedceb @ 04.11.2019 @ 13:38:10 UTC"}

http://princeton0.field.hortonworks.com:8082/jobs/overview
--cluster=true

flink run -m yarn-cluster -d -p 2 -yD log4j.configuration.file=log4j.properties -ynm IoT iot-1.0.jar--cluster true flink run -m yarn-cluster -d -p 2 -ynm Wiki iot-1.0.jar
flink run -m yarn-cluster /opt/cloudera/parcels/…

NiFi Toolkit - CLI - For NiFi 1.10

NiFi Toolkit - CLI - For NiFi 1.10 Along with the updated Apache NiFi server, the NiFi 1.10 release also updated the Command Line Interface with some updated and new features.   Let's check them out.
Cool Tools

S2S.sh - send data to Apache NiFi via the CLI.

Formatted as such:
[{"attributes":{"key":"value"},"data":"stuff"}]

Examples

registry import-flow-version


Get Into Interactive Mode

./cli.sh




Get Parameter Contexts (simple or json format)

 nifi list-param-contexts -u http://localhost:8080 -ot simple



Export Parameter Context

nifi export-param-context -u http://localhost:8080 -verbose --paramContextId 8067d863-016e-1000-f0f7-265210d3e7dc 




Get Services

 nifi get-services -u http://localhost:8080


NiFi Dump

../bin/nifi.sh dump filedump.txt

NiFi home: /Users/tspann/Documents/nifi-1.10.0

Bootstrap Config File: /Users/tspann/Documents/nifi-1.10.0/conf/bootstrap.conf

2019-11-18 17:08:04,921 INFO [main] org.apache.nifi.bootstrap.Command Successful…

NiFi 1.10: PostSlack - Easy Image Upload

In Apache NiFi 1.10, we now have a nice PostSlack processor to send images to slack channels.   If you have read my blog I had a lesser processor that I wrote:  https://www.datainmotion.dev/2019/03/posting-images-to-slack-from-apache.html. Time to upgrade my friend.  
To use the new processor you will need to have a Slack account, Slack administrator permissions and a Slack application created so you can have an application token.   You can grab that token here: https://api.slack.com/methods/files.upload/test. Go to Tester tab, pick your app, mine is Nifi. then click test. At the bottom will be a URL with the token=? Grab that token.
Notes:
You also must login with slack admin role
files:write files:write:user
under YOURSLACK /apps/manage .  then dive into app
https://api.slack.com/messaging/files/setup
You can make this a parameter so you can make your PostSlack processor into a stateless module.
See example code here:  https://github.com/tspannhw/nifi-1.10-templates

Our Example Flow: …

Ingest Salesforce Data into Hive Using Apache Nifi

Ingest Salesforce Data into Hive Using Apache NiFiTOOLS AND ACCOUNT USED: 

Salesforce JDBC driver : https://www.progress.com/jdbc/salesforce Nifi : 1.9.2 -https://nifi.apache.org/docs/nifi-docs/html/getting-started.html Hive : 3.1.2 -https://luckymrwang.github.io/2018/03/14/Install-hive-on-Mac-with-Homebrew/ Java : 1.8.0_221  OSS: MacOS Mojave 10.14.5 SDFCDeveloper account:https://developer.salesforce.com/signup - 15 day trial Reference Article :https://www.progress.com/tutorials/jdbc/ingest-salesforce-data-incrementally-into-hive-using-apache-nifi

Overall Flow : Install Hive - Thrift, Metastore, and Hive Table schema that matches SDFC opportunity table.  Install NiFi - Add PutHive3streaming nar  Install and Configure Drivers : SalesforceConnect and Avroreader