Skip to main content

Posts

Showing posts from November, 2019

Introducing Mm FLaNK... An Apache Flink Stack for Rapid Streaming Development From Edge 2 AI

Introducing Mm FLaNK...  An Apache Flink Stack for Rapid Streaming Development From Edge 2 AI Source :    https://github.com/tspannhw/MmFLaNK Stateless NiFi:     https://www.datainmotion.dev/2019/11/exploring-apache-nifi-110-parameters.html To show an example of using the Mm FLaNK stack we have an Apache NiFi flow that reads IoT data (JSON) and send it to Apache Kafka.   An Apache Flink streaming application running in YARN reads it, validates the data and send it to another Kafka topic.  We monitor and check the data with SMM.    The data from that second topic is read by Apache NiFi and pushed to Apache Kudu tables. Mm FLaNK Stack (MXNet, MiNiFi, Flink, NiFi, Kafka, Kudu)  First, we rapidly ingest, route, transform, convert, query and process data with Apache NiFi .   Once we have transformed it into a client, schema-validated known data type we can stream it to Kafka for additional processing. Second, we read from that Kafka topic, iot , with a Ja

Learning Apache Flink 1.9

Learning Apache Flink 1.9 Everything is a Stream, important concept to remember when working with Apache Flink. Submit a Job https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html flink run -m yarn-cluster -d -p 2 -ynm Wiki iot-1.0.jar   Mm Flank I am working on a nice integration pattern with the Mm Flank Stack of Apache NiFi-MiniFi, MXNet, Flink, NiFi, Kafka and Kudu. Flink Monitoring http://princeton0.field.hortonworks.com:8082/config {"refresh-interval":10000,"timezone-offset":0,"timezone-name":"Coordinated Universal Time","flink-version":"1.9.0-csa1.0.0.0","flink-revision":"3cedceb @ 04.11.2019 @ 13:38:10 UTC"} http://princeton0.field.hortonworks.com:8082 /jobs/overview --cluster=true flink run -m yarn-cluster -d -p 2 -yD log4j.configuration.file=log4j.properties -ynm IoT iot-1.0.jar   --cluster tr

NiFi Toolkit - CLI - For NiFi 1.10

NiFi Toolkit - CLI - For NiFi 1.10 Along with the updated Apache NiFi server, the NiFi 1.10 release also updated the Command Line Interface with some updated and new features.   Let's check them out. Cool Tools S2S.sh - send data to Apache NiFi via the CLI. Formatted as such: [{"attributes":{"key":"value"},"data":"stuff"}] Examples registry import-flow-version Get Into Interactive Mode ./cli.sh Get Parameter Contexts (simple or json format)  nifi list-param-contexts -u http://localhost:8080 -ot simple Export Parameter Context nifi export-param-context -u http://localhost:8080 -verbose --paramContextId 8067d863-016e-1000-f0f7-265210d3e7dc  Get Services  nifi get-services -u http://localhost:8080 NiFi Dump ../bin/nifi.sh dump filedump.txt NiFi home: /Users/tspann/Documents/nifi-1.10.0 Bootstrap Config File: /Users/tspann/Documents/nifi-1.10.0/conf/bootstrap.conf 2019-11-18 17:08:04,

NiFi 1.10: PostSlack - Easy Image Upload

In Apache NiFi 1.10, we now have a nice PostSlack processor to send images to slack channels.   If you have read my blog I had a lesser processor that I wrote:   https://www.datainmotion.dev/2019/03/posting-images-to-slack-from-apache.html . Time to upgrade my friend.   To use the new processor you will need to have a Slack account, Slack administrator permissions and a Slack application created so you can have an application token.   You can grab that token here: https://api.slack.com/methods/files.upload/test . Go to Tester tab, pick your app, mine is Nifi. then click test. At the bottom will be a URL with the token=? Grab that token. Notes: You also must login with slack admin role files:write files:write:user under YOURSLACK /apps/manage .  then dive into app https://api.slack.com/messaging/files/setup You can make this a parameter so you can make your PostSlack processor into a stateless module. See example code here:   https://gi

Ingest Salesforce Data into Hive Using Apache Nifi

Ingest Salesforce Data into Hive Using Apache NiFi TOOLS AND ACCOUNT USED:  Salesforce JDBC driver : https://www.progress.com/jdbc/salesforce Nifi : 1.9.2 - https://nifi.apache.org/docs/nifi-docs/html/getting-started.html Hive : 3.1.2 - https://luckymrwang.github.io/2018/03/14/Install-hive-on-Mac-with-Homebrew/ Java : 1.8.0_221  OSS : MacOS Mojave 10.14.5 SDFC Developer account : https://developer.salesforce.com/signup - 15 day trial Reference Article : https://www.progress.com/tutorials/jdbc/ingest-salesforce-data-incrementally-into-hive-using-apache-nifi Overall Flow : Install Hive - Thrift, Metastore, and Hive Table schema that matches SDFC opportunity table.  Install NiFi - Add PutHive3streaming nar  Install and Configure Drivers : SalesforceConnect and Avroreader Add Processors - QueryDatabaseTable and Puthive3Streaming Install Salesforce JDBC and Hive drivers Download DataDirect Salesforce JDBC driver from here . Downlo