Data In Motion

QuickTip:   Ingesting Google Analytics API with Apache NiFi  Design your query / test the API here: https://ga-dev-tools.appspot.com/query-explorer/ Building this NiFi flow is trivial. Add your URL with tokens from the Query Explorer console. You will need to reference the JRE that NiFi is using and it's cacerts if you don't want to build your own trust store.   The default password for JDK 8 is changeit .   No really. Here are our results in clean JSON Here are some attributes NiFi shows. Example JSON Results {   "kind": "analytics#gaData",   "id": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:33&metrics=ga:users,ga:percentNewSessions,ga:sessions&start-date=30daysAgo&end-date=yesterday",   "query": {     "start-date": "30daysAgo",     "end-date": "yesterday",     "ids": "ga:33",     &

NiFi Toolkit - CLI - For NiFi 1.10 Along with the updated Apache NiFi server, the NiFi 1.10 release also updated the Command Line Interface with some updated and new features.   Let's check them out. Cool Tools S2S.sh - send data to Apache NiFi via the CLI. Formatted as such: [{"attributes":{"key":"value"},"data":"stuff"}] Examples registry import-flow-version Get Into Interactive Mode ./cli.sh Get Parameter Contexts (simple or json format)  nifi list-param-contexts -u http://localhost:8080 -ot simple Export Parameter Context nifi export-param-context -u http://localhost:8080 -verbose --paramContextId 8067d863-016e-1000-f0f7-265210d3e7dc  Get Services  nifi get-services -u http://localhost:8080 NiFi Dump ../bin/nifi.sh dump filedump.txt NiFi home: /Users/tspann/Documents/nifi-1.10.0 Bootstrap Config File: /Users/tspann/Documents/nifi-1.10.0/conf/bootstrap.conf 2019-11-18 17:08:04,

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest Source Code:    https://github.com/tspannhw/minifi-grove-sensors Acquiring sensor data from Grove sensors is easy using a GrovePi Hat and some compatible sensors. Just before my talk at the Future of Data Meetup @ Bell Works in Holmdel, NJ, I thought I should ingest some data from a grove sensor interface. It's so easy a sleeping cat could do it. So what does this device look like?   I have a temperature and humidity sensor on there. The distance sonic sensor is in there too, that's for the next article. Let's do this with minimal RAM. That's a 64GB hard drive underneath in the white case with the RPI. I need more data and BACON. We design our MiNiFi Agent Flow in CEM/EFM.   Grab JSON data stream and run sensors. Apache NiFi 1.9.2 / CFM 1.0 Received HTTPS S2S Events From MiNiFi Agent A simple flow to query and convert our

Migrating Apache Flume Flows to Apache NiFi:  Kafka Source to HTTP REST Sink and HTTP REST Source to Kafka Sink Article 6 - This. Article 5 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html Article 4 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html Article 3 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html Article 2 -  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html Article 1  -  https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html Source Code:   https://github.com/tspannhw/flume-to-nifi This is a simple use case of being a gateway between REST API and Kafka.   We can do a lot more than that in NiFi.  We can be a Kafka Consumer and Producer as well as POST REST calls and receive any REST calls on configurable ports.  All with No Code. NiFi can act as a listener for HTTP Requests and provide HTTP