Using Apache NiFi in OpenShift and Anywhere Else to Act as Your Global Integration Gateway
What does it look like?
Where Can I Run This Magic Engine:
Private Cloud, Public Cloud, Hybrid Cloud, VM, Bare Metal, Single Node, Laptop, Raspberry Pi or anywhere you have a 1GB of RAM and some CPU is a good place to run a powerful graphical integration and dataflow engine. You can also run MiNiFi C++ or Java agents if you want it even smaller.
Sounds Too Powerful and Expensive:
Apache NiFi is Open Source and can be run freely anywhere.
For What Use Cases:
Microservices, Images, Deep Learning and Machine Learning Models, Structured Data, Unstructured Data, NLP, Sentiment Analysis, Semistructured Data, Hive, Hadoop, MongoDB, ElasticSearch, SOLR, ETL/ELT, MySQL CDC, MySQL Insert/Update/Delete/Query, Hosting Unlimited REST Services, Interactive with Websockets, Ingesting Any REST API, Natively Converting JSON/XML/CSV/TSV/Logs/Avro/Parquet, Excel, PDF, Word Documents, Syslog, Kafka, JMS, MQTT, TCP/IP, UDP, FTP, sFTP, Files, Directories, Google Forms, Object Stores, NoSQL, Lookups, Hosting Web sites, Updates and live SQL on data streams.
MySQL/REST/MQTT/JMS/REST/Files/S3/Object Stores. You also have an expert available on NiFi here. https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_9.html.
This makes these tasks much easier to develop, deploy, manage and control. A single Data Engineer can now build, deploy and manage thousands of data streams in batch, microbatch and streams.
- https://www.datainmotion.dev/2020/12/simple-change-data-capture-cdc-with-sql.html
- https://www.datainmotion.dev/2021/01/flank-real-time-transit-information-for.html
- https://www.datainmotion.dev/2021/01/flank-using-apache-kudu-as-cache-for.html
- https://www.datainmotion.dev/2020/12/ingesting-websocket-data-for-live-stock.html
- https://www.datainmotion.dev/2020/12/smart-stocks-with-flank-nifi-kafka.html?es_id=1fb9486166
- https://dzone.com/articles/lets-build-a-simple-ingest-to-cloud-data-warehouse
- https://dzone.com/articles/real-time-streaming-deep-learning-pipelines-with-d
- https://www.datainmotion.dev/2020/10/automating-building-migration-backup.html
- https://www.datainmotion.dev/2020/10/tracking-satellites-with-apache-nifi.html
- https://www.datainmotion.dev/2020/07/ingesting-all-weather-data-with-apache.html
- https://dev.to/tspannhw/ingesting-all-the-weather-data-with-apache-nifi-2ho4
How about version control?
NiFi Registry provides easy to integrate version control with full REST API and can export your flows to a Git repository like Github.
- https://www.datainmotion.dev/2019/11/nifi-toolkit-cli-for-nifi-110.html
- https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/
- https://dzone.com/articles/devops-for-apache-nifi-17-and-more
- https://community.cloudera.com/t5/Community-Articles/Big-Data-DevOps-Apache-NiFi-Flow-Versioning-and-Automation/ta-p/247976
DevOps?
- https://www.datainmotion.dev/2021/01/automating-starting-services-in-apache.html
- https://www.datainmotion.dev/2020/09/devops-working-with-parameter-contexts.html
- https://www.datainmotion.dev/2019/11/nifi-toolkit-cli-for-nifi-110.html
How about deployment?
Apache NiFi can run anywhere!
You can run Apache NiFi on a single VM or localhost or laptop:
https://nifi.apache.org/download.html
On OpenShift
https://catalog.redhat.com/software/containers/cdt-common-rns/nifi/6026bb6c2937380b51711b73
https://github.com/rromannissen/nifi-openshift
Apache NiFi Stateless can run all in RAM, one event at a time like a Job or Function as a Service:
https://github.com/SamHjelmfelt/OpenWhisk-YarnDeployment
Docker
https://hub.docker.com/r/apache/nifi
What If I don't like easy to use Web UIs?
You can code everything with REST calls:
https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
Okay, maybe not that low-level, what about a CLI?
You can run and install it here: https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI.
Can I get more information:
- https://github.com/tspannhw/EverythingApacheNiFi
- https://nifi.apache.org/docs.html
- https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html
- https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
- https://www.youtube.com/watch?v=RjWstt7nRVY
- https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
- https://nifi.apache.org/docs/nifi-docs/html/overview.html
- https://nifi.apache.org/docs/nifi-docs/html/getting-started.html
- https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
- https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html
- https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html
- https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
- https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
- https://nifi.apache.org/docs/nifi-docs/html/user-guide.html
- https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html
- https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html
I need support:
A partner of Amazon, Google, Microsoft, Oracle, IBM and thousands more, you can trust Cloudera for enterprise cloud hosting, support and development. Cloudera has a majority of developers of Apache NiFi working on Open Source.
https://www.cloudera.com/products/cdf.html
