FLaNK: Using Apache Kudu As a Cache For FDA Updates
- InvokeHTTP: We invoke the RSS feed to get our data.
- QueryRecord: Convert RSS to JSON
- SplitJson: Split one file into individual records. (Should refactor to ForkRecord)
- EvaluateJSONPath: Extract attributes you need.
- ProcessGroup for SPL Processing
We can see the results of our flow in Apache Atlas to see full governance of our solution.
So with some simple NiFi flow, we can ingest all the new updates to DailyMed and not reprocess anything that we already have.