Migrating Apache Flume Flows to Apache NiFi: SYSLOG to KAFKA

Migrating Apache Flume Flows to Apache NiFi:  SYSLOG  to KAFKA

Article 5 - This
Article 4 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html

Article 3 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html

Article 2 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html

Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

Source Code:  https://github.com/tspannhw/flume-to-nifi

This is a simple use case of being a smart gateway/proxy between SYSLOGand Kafka.   We can do a lot more than that in NiFi.  We can be a Kafka Consumer and Producer as well as read and parse all types of logs including SYSLOG.  We have a GrokReader for converting semistructured logs into manageable tabular style data with schemas.   Log->JSON/CSV/AVRO/PARQUET.  All with No Code.

It is extremely easy to do this in NiFi.

See:  Log Parsing:   https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

Detailed Tutorials

https://blog.davidvassallo.me/2018/09/19/apache-nifi-from-syslog-to-elasticsearch/
https://community.cloudera.com/t5/Community-Articles/NiFi-Send-to-syslog/ta-p/248638
http://www.youritgoeslinux.com/impl/bigdata/nifi/syslog

Processors

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ListenSyslog/

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ParseSyslog/index.html

References

• https://community.cloudera.com/t5/Community-Articles/Su-Su-Sussudio-Sudoers-Log-Parsing-with-Apache-NiFi/ta-p/249461