Skip to main content

Posts

Showing posts with the label streams messaging manager

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest to Parquet, Kudu, ORC, Kafka, Hive and Impala

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest Source Code:    https://github.com/tspannhw/minifi-grove-sensors Acquiring sensor data from Grove sensors is easy using a GrovePi Hat and some compatible sensors. Just before my talk at the Future of Data Meetup @ Bell Works in Holmdel, NJ, I thought I should ingest some data from a grove sensor interface. It's so easy a sleeping cat could do it. So what does this device look like?   I have a temperature and humidity sensor on there. The distance sonic sensor is in there too, that's for the next article. Let's do this with minimal RAM. That's a 64GB hard drive underneath in the white case with the RPI. I need more data and BACON. We design our MiNiFi Agent Flow in CEM/EFM.   Grab JSON data stream and run sensors. Apache NiFi 1.9.2 / CFM 1.0 Received HTTPS S2S Events From MiNiFi Agent A simple flow to query and convert our

Using Cloudera Streams Messaging Manager for Apache Kafka Monitoring, Management, Analytics and CRUD

Using Cloudera Streams Messaging Manager for Apache Kafka Monitoring, Management, Analytics and CRUD SMM is powerful tool to work with Apache Kafka and provide you with monitoring, management, analytics and creating Kafka topics.   You will be able to monitor servers, brokers, consumers, producers, topics and messages.   You will also be able to easily build alerts based on various events that can occur with those entities. From Cloudera Manager, we can now install and manage Kafka, SMM, NiFi and Hadoop services. Let's create a Kafka topic, no command-line! For a simple topic, we select Low size for replication factor of one and replica count of one.  We also set a cleanup policy of delete . Let's create an alert. For this one if the nifi-reader consumer group has a lag then send an email to me. Let's browse our Kafka infrastructure in our AWS Cloudera Kafka cluster, so easy to navigate. You can dive into a t