Data In Motion

 Basic Understanding of Cloudera Flow Management - Apache NiFi Topics: NiFi Cluster Architecture Content Repository EncryptedContentRepository and other options Provenance Repository FlowFile Repository FlowFile, Attributes, Process Groups, Connections, Flow Controllers Controller Services Custom Properties Common Attributes (uuid, filename, path, file size, ...) Expression Language Flow Routing Testing and Test Data Generation Relationships Ports Bulletins flow.xml.gz Input Port Output Port Empty Queues Setting Warning Levels Funnels Copy on Write RecordPath  Using Record Processors (Readers/Writers) NiFi Toolkit NiFi CLI NiFi REST API NiFi Registry Integration Handling Errors Parameter Context / Parameters Summary / Cluster / Bulletins Reporting Tasks Sizing NiFi Cluster on # of records * size / amount of time Configuration Files (Changing RAM) Understanding NiFi logs How to add custom processors JVM  Back pressure Prioritized Queues Load Balancing Load Balancing Strategies Prioritiz

Building SSL For Hosting Mobile Web Sites on Apache NiFi  openssl req -x509 -newkey rsa:2048 -keyout admin-private-key.pem -out admin-cert.pem -days 365 -subj "/CN=Admin Q. User/C=US/L=Seattle" -nodes openssl pkcs12 -inkey admin-private-key.pem -in admin-cert.pem -export -out admin-q-user.pfx -passout pass:"SuperSecret" pwd keytool -genkeypair -alias nifiserver -keyalg RSA -keypass SuperSecret -storepass SuperSecret -keystore server_keystore.jks -dname "CN=Test NiFi Server" -noprompt keytool -genkeypair -alias nifiserver -keyalg RSA -keypass SuperSecret -storepass SuperSecret -keystore server_keystore.jks -dname "CN=Test NiFi Server" -noprompt keytool -importcert -v -trustcacerts -alias admin -file admin-cert.pem -keystore server_truststore.jks  -storepass SuperSecret -noprompt # then import into browser / ssl / key certs

 [FLaNK] Smart Weather Websocket Application - Kafka Consumer Part 2 of 2 This is based on Koji Kawamura's excellent GIST:  https://gist.github.com/ijokarumawak/60b9ab2038ef906731ebf4c0eee97176 As part of my Smart Weather Application, I wanted to display weather information as it arrives in a webpage using web sockets.   Koji has an excellent NiFi flow that does it.   I tweaked it and add some things since I am not using Zeppelin.   I am hosting my webpage with NiFi as well. https://www.datainmotion.dev/2020/11/flank-smart-weather-applications-with.html We simply supply a webpage that makes a websocket connection to NiFi and NiFi keeps a cache in HBase to know what the client is doing.  This cache is updated by consuming from Kafka.   We can then feed events as they happen to the page. Here is the JavaScript for the web page interface to websockets: <script> function sendMessage(type, payload) { websocket.send(makeMessage(type, payload)); } function makeMessage(type, payload

 [FLaNK] Smart Weather Applications with Flink SQL  Sometimes you want to acquire, route, transform, live query and analyze all the weather data in the United States while those reports happen.   With FLaNK, it's a trivial process to do. From Kafka to Kudu for Any Schema of Any Type of Data, No Code, Two Steps The Schema Registry has full Swagger-ized Runnable REST API Documentation.   Integrate, DevOps and Migration in a simple script .  Here's your schemas, upload, edit and compare. Validating Data Against a Schema With Your Approved Level of Tolerance.   You want extra fields allowed, you got it. Feed that data to beautiful visual applications running in Cloudera Machine Learning. You like drill down maps, you got them. Query your data fast with Apache Hue against Apache Kudu tables through Apache Impala. Let's ingest all the US weather stations even though they are a zipped directory of a ton of XML files. Weather Ingest is Easy Automagically View All Your Topic Data En

 [FLaNK] Streaming EdgeAI on the new NVIDIA Jetson Nano 2GB with MiNiFi Agents To FLaNK Applications Unboxing :   https://www.youtube.com/watch?v=dVGEtWYkP2c&feature=youtu.be Plug Into Community AI Apps:   https://youtu.be/2T8CG7lDkcU Education Projects:    https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-nano/education-projects/ I am not patient enough to shoot an unboxing video, I was too excited to get this superb machine running.   The NVIDIA Jetson Nano 2GB is now available for purchase for only $ 59 !!! Source and Setup:   https://github.com/tspannhw/SettingUpAJetsonNano2GB/blob/main/README.md The 2GB version of NVIDIA Jetson Nano is great, you really don't miss anything that was removed.   I have copied over my MiNiFi agent and code from other Jetson Nanos, Xavier NX and TX1 and it all works fine.   The speed is fine for most needs especially for development and prototyping.   I prefer the Xavier, but at this price you can't go wrong.   I am

  2020 Events: https://www.linkedin.com/pulse/2020-streaming-edge-ai-events-tim-spann/ August 13 - Industry Event - Including a FLaNK August 28 - 11:30-12:20 - Apache Beam Digital Summit https://2020.beamsummit.org/sessions/leukemia-early-detection-apache-beam/ https://2020.beamsummit.org/speakers/timothy-spann/ Lightning Talk - Using the Mm FLaNK Stack for Edge AI (Flink, NiFi, Kafka, Kudu) 0.1 September 16-17 - https://www.cloudera.com/about/events/cloudera-now-cdp.html Sept 29 - Oct 1 - Apache Con https://apachecon.com/acna2020/ I have some talks here and I am bringing in some superstars to assist me! It's a dream team of speakers that I will be collaborate with. I will release names when we get closer. So I will be covering Apache MXNet, Apache NiFi, MiNiFi, Apache Flink, Apache Kafka, Apache Hue and Apache Kudu. I would be surprised if Apache Spark, Apache Hadoop, Apache Hive, Apache HBase, Apache Phoenix, Apache Zeppelin, Apache Livy Incrementally Streaming RDBMS Data to