All Data and AI Weekly #215: 10 November 2025

( AI, Data, NiFi, Iceberg, Polaris, Streamlit, Flink, Kafka, Python, Java, SQL, MCP, LLM, RAG, Cortex AI, AISQL, Search, Unstructured Data )

#215: 10 November 2025

https://www.reddit.com/r/DataEngineeringForAI/hot/ https://lu.ma/PINSAI

This week was dominated by a wave of major announcements from Snowflake BUILD 2025. The central theme is clear: the rapid integration of agentic AI, advanced developer tools, and more open data pipelines directly into the enterprise data stack.

In This Issue:

• The Big Story: Snowflake BUILD 2025 & The Rise of Enterprise AI
• AI Deep Dive: Cortex Agents, AI-SQL, and Audio/Video
• Data Engineering: Openflow, Postgres Integration, and New Pipelines
• Developer Focus: Well-Architected Framework & New Dev Tools
• Security & Evaluation: Practical AI Agent Security & Observability
• Events & Learning: Upcoming Webinars & Courses
• Around the Web: Other Noteworthy Tech

The Big Story: Snowflake BUILD 2025

The main headlines from Snowflake BUILD 2025, focusing on a major push to supercharge enterprise-grade AI development and make data more accessible.

• [Snowflake Press Release]: Snowflake Unveils New Developer Tools to Supercharge Enterprise-Grade Agentic AI Development
• [Snowflake Press Release]: Snowflake Delivers the Enterprise Lakehouse with Enhanced Open Data Access
• [Official Blog]: Snowflake Intelligence: The Engine for Enterprise AI
• [Official Blog]: Fuel Your AI Strategy with Connected, Accessible Data
• [Official Blog]: Delivering Intelligent, Governed AI at Scale
• [What's New (GitHub)]: See What's New in Snowsight from BUILD 2025
• [Video Overview]: Snowflake Intelligence (Video)

AI Deep Dive: Cortex AI & Agents

The most significant updates are centered on Snowflake Cortex AI, with new tools for building, managing, and observing AI agents.

• [Getting Started]: Get Started with Snowflake Intelligence
• [Cortex Agents]: Getting Started with Cortex Agents
  • [Example YAML]: Sample sales_metrics_model.yaml for a Cortex Agent
  • [Awesome List]: Awesome Custom Cortex Agents (REST API & React App)
• [Cortex AI-SQL]:
  • [GA Announcement]: Cortex AI-SQL Operators Now GA
  • [Engineering Blog]: Real-time Text-to-SQL with Snowflake Intelligence
  • [Guide]: Getting Started with Cortex AI-SQL
  • [Examples]: More AI-SQL Examples
• [Cortex Audio/Video]:
  • [GA Announcement]: Cortex AI Transcribe Now GA
  • [User Guide]: Cortex AI Audio Functions Guide
  • [Guide]: Extracting Insights from Multimodal Customer Data
• [Cortex Search]: Cortex Search Overview
• [MS Teams Integration]:
  • [Blog]: Cortex Agents for Microsoft Marketplace
  • [Guide]: Chat with Snowflake Data from Microsoft Teams
  • [Docs]: Cortex Agents Teams Integration Docs

Data Engineering: Openflow, Postgres Integration & New Pipelines

Significant updates for data integration, including the new "pg_lake" for Postgres, Openflow for CDC, and Dynamic Iceberg Tables.

• [Postgres Lakehouse (pg_lake)]:
  • [Engineering Blog]: pg_lake: A New Era for Postgres Lakehouse Integration
  • [GitHub Repo]: pg_lake GitHub Repository
• [Openflow (CDC)]:
  • [Guide]: Getting Started with Openflow for CDC on SQL Server
  • [Guide]: Getting Started with Openflow (SPCS)
  • [Guide]: Setup Openflow (BYOC - Bring Your Own Compute)
  • [Demo]: Openflow Structured Data Pipeline Demo
• [Interactive & Dynamic Tables]:
  • [Guide]: Getting Started with Interactive Tables
  • [Docs]: Interactive Tables Documentation
  • [Guide]: Declarative Pipelines with Dynamic Tables
  • [Example]: Dynamic Iceberg Tables (Python Example)
• [Other Integrations]:
  • [GitHub]: GitHub Integration with Snowflake Openflow
  • [DeltaStream]: DeltaStream Enables Real-Time Streaming Natively in Snowflake

Developer Focus: Well-Architected Framework & New Dev Tools

Snowflake is doubling down on developer experience with a new Well-Architected Framework, an integrated developer environment, and new ML workflow guides.

• [Well-Architected Framework]:
  • [Product Page]: Snowflake Well-Architected Framework
  • [Engineering Blog]: Announcing the Snowflake Well-Architected Framework
  • [Quickstart]: Well-Architected Framework: Security & Governance Guide
• [Developer Environment]:
  • [Blog]: An Integrated Developer Environment for AI
  • [Docs]: Notebooks in Workspaces (Limited Access)
  • [Demo]: Notebooks in Workspaces Demo Repo
• [MLOps & Workflows]:
  • [Guide]: From Dev to Production: Why ML Teams are Migrating to Snowflake
  • [Guide]: Build an End-to-End ML Workflow in Snowflake
  • [Toolkit]: Snowflake AI Toolkit (GitHub)

Security & Evaluation: Practical AI Agent Security & Observability

As AI agents become more powerful, security and evaluation are critical. These resources provide frameworks for securing and evaluating your new AI tools.

• [Security]:
  • [Meta Blog]: A Guide to Practical AI Agent Security
  • [Snowflake Blog]: Security and Governance Practices for Snowflake Intelligence
  • [Anthropic Blog]: Code Execution with Model-written Code Patches (MCP)
  • [GitHub]: Anthropic's Model-written Code Patches (MCP) Repo
• [Evaluation & Observability]:
  • [Snowflake Blog]: An AI Agent Evaluation Framework: The GPA Framework
  • [Docs]: Cortex AI Observability
  • [TruLens Guide]: Web Search Agent Evaluation with TruLens
  • [Video]: Evaluating AI Agents (Snowflake)

Events & Learning

Want to learn more? Here are some upcoming webinars and hands-on labs to get you started.

• [Webinar]: Unlock the Power of Your Data with Snowflake Intelligence (Nov 13, 2025)
• [Hands-on Lab]: Build Powerful AI Agents and Apps Using OpenAI in Snowflake Cortex AI (Nov 18, 2025)
• [Course]: DeepLearning.AI & Snowflake: Building and Evaluating Data Agents
• [Hands-on Lab]: MCP (Model-written Code Patches) HOL from BUILD 2025
• https://www.snowflake.com/en/webinars/demo/build-semanticaware-ai-with-snowflake-and-dbt-labs-2025-11-25/

Around the Web: Other Noteworthy Tech

A few other interesting projects and articles that caught our eye this week.

• [Neo4j]: NODES 2025 Talk: From Air Quality to Aircraft, Data Is Everywhere (Slides)
• [Data Pipelines]: dlt (Data Load Tool) GitHub Repo
• [Supabase]: Supabase ETL - A Postgres-based ETL
• [Postgres]: pg_incremental - Incremental View Maintenance for Postgres
• [Geospatial]: GeoParquet.org - Open Standard for Geospatial Vector Data
• [Developer Tools]: Cursor: Semantic Search in Your Codebase
• [Developer Tools]: Task - A Task Runner / Build Tool
• [And For Fun...]: How to Calibrate an RTL-SDR (Software Defined Radio)

Thanks

https://sessionize.com/tspann

https://github.com/timothyspann

© 2020-2025 Tim Spann https://www.youtube.com/@FLaNK-Stack