Skip to main content

Posts

Showing posts from June, 2019

Dataworks Summit DC and NoSQL DC 2019 Videos

In this quick article are the videos from some of my talks (they didn't record the lightning talks or BoF).   Also is a great talk by my friends from American Water. Check them out. Tracking Crime as It Happens with Apache Phoenix, Apache HBase and Apache NiFi https://dataworkssummit.com/nosql-day-2019/session/tracking-crime-as-it-occurs-with-apache-phoenix-apache-hbase-and-apache-nifi/ Henry Sowell and Timothy Spann All the talks at NoSQL 2019 DataWorks Summit 2019 DC Internet of Fleet Management Things American Water Cold Chain Logistics with NiFi https://dataworkssummit.com/washington-dc-2019/session/cold-supply-chain-logistics/ Timothy Spann and Mehul Shah Edge to AI:  Analytics From Edge to Cloud https://dataworkssummit.com/washington-dc-2019/session/edge-to-ai-analytics-from-edge-to-cloud-with-efficient-movement-of-machine-data-2/ Timothy Spann and John Kuchmek https://github.com/tspannhw/phillycrime-springboot

Performance Testing Apache NiFi - Part 1 - Loading Directories of CSV

Performance Testing Apache NiFi - Part 1 - Loading Directories of CSV I am running a lot of different flows on different Apache NiFi configurations to get some performance numbers in different situations. One situation I thought of was access directories of CSV files from HTTP.  Fortunately there's some really nice data available from NOAA (https://www.ncei.noaa.gov/data/global-hourly/access/2019/). Example Flow:    NOAA In this example performance testing flow I use my LinkProcessor to grab all of the links to CSV files on the HTTP download site.  I then split this JSON list into individual records and pull out the URL.   If it's a valid URL with a .CSV ending then I call invokeHTTP to download the CSV.   I then query the CSV for all the records (SELECT *) and for a count (SELECT COUNT(*)).   As part of this the records are written to JSON. In this example we grab a specific CSV file and get 739 records.  This CSVReader  uses Jackson to parse th