Skip to main content

Posts

Showing posts with the label hdf3.5.0

SQL Reporting Task for Cloudera Flow Management / HDF / Apache NiFi

SQL Reporting Task for Cloudera Flow Management / HDF / Apache NiFi Would you like to have reporting tasks gathering metrics and sending them to your database or Kafka from NiFi based on a query of NiFi provenance, bulletins, metrics, processor status or other KPI? Now you can.   If you are using HDF 3.5.0, this Reporting task NAR is pre installed and ready to go. Let's add some Reporting tasks that use SQL!!!  QueryNiFiReportingTask. The first one that was interesting for me was to write queries against provenance for one processor that consumes from a certain topic, I decided to query it every 10 seconds.   My query and some results are below. So let's go to Controller Settings / Reporting Tasks and then add QueryNiFiReportingTask : We add one per item we want to monitor.   Then for the reporting task we will need a place to send the records (a sink), we can send it to a JDBC Database ( DatabaseRecordSink, KafkaRecordSink, PrometheusRecordSink, ScriptedRec