Skip to main content

Posts

Showing posts with the label cloud

Using Cloudera Data Platform with Flow Management and Streams on Azure

Using Cloudera Data Platform with Flow Management and Streams on Azure Today I am going to be walking you through using Cloudera Data Platform (CDP) with Flow Management and Streams on Azure Cloud.  To see a streaming demo video, please join my webinar (or see it on demand) at  Streaming Data Pipelines with CDF in Azure .  I'll share some additional how-to videos on using Apache NiFi and Apache Kafka in Azure very soon.    Apache NiFi on Azure CDP Data Hub Sensors to ADLS/HDFS and Kafka In the above process group we are using QueryRecord to segment JSON records and only pick ones where the Temperature in Fahrenheit is over 80 degrees then we pick out a few attributes to display from the record and send them to a slack channel. To become a Kafka Producer you set a Record Reader for the type coming in, this is JSON in my case and then set a Record Writer for the type to send to the  sensors  topic.    In this case we kept it as JSON, but we could convert to AVRO.   I usually do that

Cloudera Flow Management 101: Let's Build a Simple REST Ingest to Cloud Datawarehouse With LowCode. Powered by Apache NiFi

Use NiFi to call REST API, transform, route and store the data Pick any REST API of your choice, but I have walked through this one to grab a number of weather stations reports.  Weather or not we have good weather, we can query it anyway. We are going to build a GenerateFlowFile to feed our REST calls. [ { " url " : " http://weather.gov/xml/current_obs/CWAV.xml " }, { " url " : " http://weather.gov/xml/current_obs/KTTN.xml " }, { " url " : " http://weather.gov/xml/current_obs/KEWR.xml " }, { " url " : " http://weather.gov/xml/current_obs/KEWR.xml " }, { " url " : " http://weather.gov/xml/current_obs/CWDK.xml " }, { " url " : " http://weather.gov/xml/current_obs/CWDZ.xml " }, { " url " : " http://weather.gov/xml/current_obs/CWFJ.xml " }, { " url " : " http://weather.gov/xml/current_obs/PAEC.xml " }, { " url " : "