Showing posts with label open source. Show all posts
Showing posts with label open source. Show all posts

FLaNK Stack Weekly for 26 June 2023

 

26-June-2023

FLiPN-FLaNK Stack Weekly

Tim Spann @PaaSDev

My friend wrote an awesome new book on streaming, I highly recommend picking up a copy!

https://leanpub.com/streamprocessingwithapacheflink/c/ucQ5dLcZYAo2

Join me in person for steak & stack or virtually for FLaNK Stack

https://www.meetup.com/futureofdata-princeton/events/292976004/

meetup

Wednesday, June 28, 2023 at 6:00 PM to Wednesday, June 28, 2023 at 8:00 PM EDT Add to calendar The Capital Grille 310 W Wisconsin Ave · Milwaukee, WI

Also live streamed to Youtube

This will be a hybrid event with a Zoom. The in-person event will be in Milwaukee.

In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns. He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg. If you wish to follow along, please download open source projects beforehand. You can also download this helpful streaming platform: https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html All source code and slides will be shared for those interested in building their own FLaNK Apps. https://www.flankstack.dev/

https://www.thecapitalgrille.com/locations/wi/milwaukee/milwaukee/8027

cunkflank

Hardware For FLaNK

The amazing team at Ampere Computing sent us a 2U Mt Jade.

https://amperecomputing.com/en/systems/altra/2u-mt-jade-2s-nvme

We will be running some AI, IoT, MiNiFi, NiFi, Kafka, Flink, Pulsar, Spark, Iceberg, Ozone, HBase, Kudu, Hive, Impala, Jupyter and more workloads here.

Updates

CDF-PC 2.5 on CDP Public Cloud

https://docs.cloudera.com/dataflow/cloud/deploy-flows/topics/cdf-flow-deployment-autoscaling.html

New Advanced UIs:

  • The Flow Designer now supports the advanced configuration UI for UpdateAttribute.
  • The Flow Designer now supports the advanced configuration UI for JoltTransformJson.
  • New Canvas navigation: The Flow Designer now supports Birdseye and Zoom controls.
  • New troubleshooting: The Flow Designer now supports Processor Diagnostics with an active Test Session.
  • Multi-Select: The Flow Designer now supports multi-selection on the canvas and bulk actions for Start, Stop, Enable, Disable, Move, Change parent group, Copy/Paste, and Delete.

New ReadyFlows for this release:

  • CDW Ingest
  • CDP Kafka to Snowflake
  • Slack to S3
  • Updated Confluent Cloud to Snowflake using new Snowpipe processors

CODE + COMMUNITY

Please join my meetup group NJ/NYC/Philly/Virtual.

http://www.meetup.com/futureofdata-princeton/

https://www.meetup.com/futureofdata-newyork/

https://www.meetup.com/futureofdata-philadelphia/

**This is Issue #91 **

You may notice a version jump, Linked in says we had 89 already, so I am assuming two other articles got assimilated. I will go with this, since 90 is a better number.

https://github.com/tspannhw/FLiPStackWeekly

https://www.linkedin.com/pulse/schedule-2023-tim-spann-/

Courses

https://www.cloudera.com/about/training/courses/apache-nifi-anti-patterns.html

Videos

https://www.youtube.com/watch?v=H1SYOuLcUTI&ab_channel=Ververica

https://www.youtube.com/watch?app=desktop&v=8cZJ9CyLYyI

Conference Videos

Hail Hydrate! From Stream to Lake https://www.youtube.com/watch?v=IBpqa8re--o&ab_channel=PowerShell.org

Articles

https://medium.com/@tspann/ingesting-events-into-dockerized-ibm-db2-jdbc-with-apache-nifi-f0ca452d1351

https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/

https://dzone.com/articles/apache-nifi-10-cheatsheet

https://www.linkedin.com/posts/excalidraw_re-keying-a-kafka-topic-activity-7077942003837100033-KfnM/

https://medium.com/@tspann/functions-anywhere-faas-ee92ecedb248

Events

June 26-28, 2023: NLIT Summit. Milwaukee.
https://www.fbcinc.com/e/nlit/default.aspx

June 28, 2023: NiFi Meetup. Milwaukee and Hybrid. https://www.meetup.com/futureofdata-princeton/events/292976004/

meetup

July 19, 2023: 2-Hours to Data Innovation: Data Flow https://www.cloudera.com/about/events/hands-on-lab-series-2-hours-to-data-innovation.html

October 18, 2023: 2-Hours to Data Innovation: Data Flow https://www.cloudera.com/about/events/hands-on-lab-series-2-hours-to-data-innovation.html

Cloudera Events https://www.cloudera.com/about/events.html

More Events: https://www.linkedin.com/pulse/schedule-2023-tim-spann-/

Code

https://github.com/polyzos/stream-processing-with-apache-flink

NiFi Code

Tools

© 2020-2023 Tim Spann

on
Labels: , , ,

FLiPN-FLaNK Stack Weekly - 12 June 2023

 

12-June-2023

FLiPN-FLaNK Stack Weekly

Tim Spann @PaaSDev

Cloudera Now is coming and we have a cool Edge Demo.

https://www.cloudera.com/about/events/cloudera-now-cdp.html

Hope everyone is safe, air quality readings from the many forest firest have been very high. Stay indoors East Coast people.

diagram

CODE + COMMUNITY

Please join my meetup group NJ/NYC/Philly/Virtual.

http://www.meetup.com/futureofdata-princeton/

https://www.meetup.com/futureofdata-newyork/

https://www.meetup.com/futureofdata-philadelphia/

This is Issue #87

https://github.com/tspannhw/FLiPStackWeekly

https://www.linkedin.com/pulse/schedule-2023-tim-spann-/

Videos

https://www.youtube.com/watch?v=_uYp8s6_6GA&t=1s&pp=ygUQIlRpbSBzcGFubiIgbmlmaQ%3D%3D

https://www.youtube.com/watch?v=X7m4nZH8bUw&t=1s&pp=ygUQIlRpbSBzcGFubiIgbmlmaQ%3D%3D

https://www.youtube.com/watch?v=WvPqE8J3ZOE&t=1700s&pp=ygUQIlRpbSBzcGFubiIgbmlmaQ%3D%3D

https://www.youtube.com/watch?v=NMgkPFEQ0jA&pp=ygUOVGltIFNwYW5uIG5pZmk%3D

Articles

https://medium.com/@tspann/harnessing-the-power-of-nifi-building-a-seamless-flow-to-ingest-pm2-5-90246393fcab

https://medium.com/@tspann/wildfires-air-quality-time-to-fire-up-the-sensors-and-start-flanking-12ea0ba33f63

https://blog.cloudera.com/aaand-the-new-nifi-champion-is/

https://platform.openai.com/docs/guides/gpt-best-practices/six-strategies-for-getting-better-results

https://blog.devgenius.io/presto-introduction-10b3ba5020e8

https://medium.com/swlh/become-a-smart-content-creator-in-these-4-steps-6d4d813863c2

https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-programming-environment-on-an-ubuntu-20-04-server

https://www.cloudera.com/about/customers/metro-de-madrid.html

https://prestodb.io/docs/current/connector/kafka-tutorial.html

https://education.cloudera.com/store/2515343-apache-nifi-anti-patterns#description

https://medium.com/cloudera-inc/seamless-integration-unleashing-the-power-of-real-time-groceries-with-nifi-kafka-flink-and-32d659fe0903

https://docs.cloudera.com/cem/1.5.1/using-asset-push-command/topics/cem-using-asset-push-command.html

https://eng.lyft.com/gotchas-of-streaming-pipelines-profiling-performance-improvements-301439f46412

https://bryanbende.com/development/2021/07/19/apache-nifi-1-14-0-secure-by-default

https://docs.cloudera.com/cem/1.5.1/rest-api-reference/index.html#api-AgentClassManifestConfig

https://docs.cloudera.com/csp/2.0.1/monitoring-end-to-end-latency/topics/smm-enabling-interceptors.html

https://blog.cloudera.com/implementing-and-using-udfs-in-cloudera-sql-stream-builder/

Recent Talks

https://www.slideshare.net/bunkertor/ossna-building-modern-data-streaming-apps

https://www.slideshare.net/bunkertor/using-apache-nifi-with-apache-pulsar-for-fast-data-onramp

https://www.slideshare.net/bunkertor/budapest-dataml-building-modern-data-streaming-apps-with-nifi-flink-and-kafka

https://grafana.com/docs/grafana-cloud/data-configuration/metrics/prometheus-config-examples/the-apache-software-foundation-minifi/

Events

https://www.linkedin.com/posts/cloudera-partners_llm-opensource-llms-activity-7064751460844015616-gF45?utm_source=share&utm_medium=member_desktop

https://www.youtube.com/watch?v=Ws7YmAHE1O8

https://www.cloudera.com/about/events/evolve.html

https://web.cvent.com/event/7598f981-2f7e-4915-b662-bd7be9b5f48d/summary?RefId=homepage_impact24

https://www.cloudera.com/about/events/cloudera-now-cdp.html?i

June 14: 12PM EDT Cloudera Now - Virtual https://www.cloudera.com/about/events/cloudera-now-cdp.html?internal_keyplay=ALL&internal_campaign=FY24-Q2_AMER_CNOW_Q2_WEB_EP_P07_2023-06-14&cid=7012H000001ZLmyQAG&internal_link=p07

June 26-28, 2023: NLIT Summit. Milwaukee.
https://www.fbcinc.com/e/nlit/default.aspx

June 28, 2023: NiFi Meetup. Milwaukee and Hybrid. https://www.meetup.com/futureofdata-princeton/events/292976004/

meetup

July 19, 2023: 2-Hours to Data Innovation: Data Flow https://www.cloudera.com/about/events/hands-on-lab-series-2-hours-to-data-innovation.html

October 18, 2023: 2-Hours to Data Innovation: Data Flow https://www.cloudera.com/about/events/hands-on-lab-series-2-hours-to-data-innovation.html

Cloudera Events https://www.cloudera.com/about/events.html

More Events: https://www.linkedin.com/pulse/schedule-2023-tim-spann-/

Open Source Fonts

https://github.com/intel/intel-one-mono/releases

Tools

MiNiFi Tip

Start agent to beartbeat (register to efm) before the kafka nar was added

If not, change the agent class name the agent belongs to, restart minifi

© 2020-2023 Tim Spann

on
Labels: , , ,

FLiPN-FLaNK Stack Weekly for April 3, 2023

 

FLiPN-FLaNK Stack Weekly

Tim Spann @PaaSDev

Java 20!!!!

image

CODE + COMMUNITY

Join my meetup group NJ/NYC/Philly/Virtual.

https://www.meetup.com/new-york-city-apache-pulsar-meetup/

https://www.meetup.com/futureofdata-princeton/

https://www.meetup.com/futureofdata-sanfrancisco/events/292453316/

This is Issue #77

https://github.com/tspannhw/FLiPStackWeekly

https://www.linkedin.com/pulse/schedule-2023-tim-spann-/

Meetup

http://www.meetup.com/futureofdata-princeton/

https://www.meetup.com/phillyjug/events/291103971/

https://www.cloudera.com/solutions/data-practitioners.html

https://www.cloudera.com/about/events.html

Videos

https://www.youtube.com/watch?v=iT60STl-Wuk

https://www.youtube.com/watch?v=4X5Yky3CT6I&t=13s

https://www.youtube.com/watch?v=V_DpqTo4bQ0

https://www.youtube.com/watch?v=p9-Y1PRYDn4&t=2s

https://www.youtube.com/watch?v=s80sz3NWwHo

Articles

https://community.cloudera.com/t5/What-s-New-Cloudera/Cloudera-DataFlow-Designer-for-self-service-data-flow/ba-p/366039

https://posthog.com/blog/dev-marketing-for-startups#its-ok-for-other-companies-to-be-much-better-than-you-at-social-media

https://developerrelations.com/devrel-roundtable/looking-ahead-to-conference-season

https://ossinsight.io/collections/chat-gpt-alternatives/

https://robertsahlin.substack.com/p/the-data-engineer-is-dead-long-live

https://blogs.oracle.com/javamagazine/

https://www.infoq.com/articles/billions-messages-minute/?

https://technology.amis.nl/big-data-database/apache-nifi-automating-tasks-using-nipyapi/

https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm

https://www.linkedin.com/posts/michael-kohs-27a17525_snowpipe-snowflake-nifi-activity-7047694779786084352-ArdL/

https://thenewstack.io/linkedin-unifies-stream-and-batch-processing-with-apache-beam/

Recent Talks

Trenton Computer Festival Pro https://www.slideshare.net/bunkertor/itpc-building-modern-data-streaming-apps

https://www.youtube.com/watch?v=iT60STl-Wuk&list=PLIJGKvnQWB-u0SPXIwozegOWCG2V85WGe&index=12

Documentation

https://docs.cloudera.com/dataflow/cloud/aws-lambda-functions/topics/cdf-create-aws-lambda-function.html

https://docs.cloudera.com/cdp-public-cloud/cloud/release-summaries/topics/announcement-202303.html

Events

https://www.cloudera.com/about/events/evolve.html

https://web.cvent.com/event/7598f981-2f7e-4915-b662-bd7be9b5f48d/summary?RefId=homepage_impact24

Next week!!!

April 4-6, 2023: DevNexus: Atlanta, GA. In-Person. https://devnexus.com/

April 24-26, 2023: Real-Time Analytics Summit: San Francisco, CA. In-Person. https://rtasummit.com/

April 25, 2023: Future of Data Meetup: San Francisco, CA. In-Person. https://www.meetup.com/futureofdata-princeton/ https://www.meetup.com/futureofdata-sanfrancisco/events/292453316/

May 9, 2023: Garden State Java User Group. In-Person. New Jersey https://gsjug.org/

May 10-12, 2023: Open Source Summit North America. Virtual https://events.linuxfoundation.org/open-source-summit-north-america/

May 23, 2023: Pulsar Summit Europe. Virtual https://pulsar-summit.org/

Cloudera Events https://www.cloudera.com/about/events.html

More Events: https://www.linkedin.com/pulse/schedule-2023-tim-spann-/

Code

https://github.com/pdefusco/Oozie2CDE_Migration

https://github.com/SuperEllipse/edge2ai_pred_maint

https://github.com/tspannhw/FLaNK-AllTheStreams

https://github.com/tspannhw/CloudDemo2023

Tools

https://github.com/bencgreenberg/stackexchange-tutorial-themes

https://github.com/jaymody/picoGPT

https://regex.ai/

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.JoinEnrichment/additionalDetails.html

https://clickhouse.com/docs/en/integrations/nifi

https://github.com/TheoKanning/openai-java

https://pyscript.net/

https://tmate.io/

https://github.com/dylanaraps/neofetch

https://github.com/jesseduffield/lazydocker

https://github.com/httpie/httpie

https://github.com/GothenburgBitFactory/taskwarrior

https://github.com/newsboat/newsboat

https://github.com/jarun/ddgr

https://github.com/cointop-sh/cointop

https://github.com/Byron/dua-cli

https://nicolargo.github.io/glances/

https://github.com/aristocratos/bpytop

https://github.com/hacker1024/coretemp

https://github.com/bcicen/ctop

https://github.com/imsnif/bandwhich

https://github.com/jbruchon/jdupes

https://exiftool.org/

https://github.com/aria2/aria2

https://github.com/muesli/duf

https://github.com/ajeetdsouza/zoxide

https://github.com/PrefectHQ/marvin

https://github.com/libAudioFlux/audioFlux

https://github.com/jamesturk/scrapeghost/

https://gut-cli.dev/

https://yakgpt.vercel.app/

https://github.com/HamburgChimps/apple-notes-liberator

https://www.cursor.so/

https://orbstack.dev/

https://a16z.com/2023/03/30/b2b-generative-ai-synthai/

https://github.com/twitter/the-algorithm

https://github.com/twitter/the-algorithm-ml

https://github.com/fipso/ccurl.sh

https://donuts-are-good.github.io/shhhbb/

© 2023 Tim Spann

on
Labels: , , ,
Subscribe to: Posts (Atom)