Data In Motion

In this quick article are the videos from some of my talks (they didn't record the lightning talks or BoF).   Also is a great talk by my friends from American Water. Check them out. Tracking Crime as It Happens with Apache Phoenix, Apache HBase and Apache NiFi https://dataworkssummit.com/nosql-day-2019/session/tracking-crime-as-it-occurs-with-apache-phoenix-apache-hbase-and-apache-nifi/ Henry Sowell and Timothy Spann All the talks at NoSQL 2019 DataWorks Summit 2019 DC Internet of Fleet Management Things American Water Cold Chain Logistics with NiFi https://dataworkssummit.com/washington-dc-2019/session/cold-supply-chain-logistics/ Timothy Spann and Mehul Shah Edge to AI:  Analytics From Edge to Cloud https://dataworkssummit.com/washington-dc-2019/session/edge-to-ai-analytics-from-edge-to-cloud-with-efficient-movement-of-machine-data-2/ Timothy Spann and John Kuchmek https://github.com/tspannhw/phillycrime-springboot

Performance Testing Apache NiFi - Part 1 - Loading Directories of CSV I am running a lot of different flows on different Apache NiFi configurations to get some performance numbers in different situations. One situation I thought of was access directories of CSV files from HTTP.  Fortunately there's some really nice data available from NOAA (https://www.ncei.noaa.gov/data/global-hourly/access/2019/). Example Flow:    NOAA In this example performance testing flow I use my LinkProcessor to grab all of the links to CSV files on the HTTP download site.  I then split this JSON list into individual records and pull out the URL.   If it's a valid URL with a .CSV ending then I call invokeHTTP to download the CSV.   I then query the CSV for all the records (SELECT *) and for a count (SELECT COUNT(*)).   As part of this the records are written to JSON. In this example we grab a specific CSV file and get 739 records.  This CSVReader  uses Jackson to parse th