The Upriver Blog

A hands-on guide to AI agents for data engineering across dbt, Airflow, and Dagster. See how Claude Code and Codex can investigate failures, find root causes, and validate fixes.

From the Data Splash Podcast, hosted by Upriver's CEO Ido Bronstein. This episode's guest is Jessica Talisman — semantic engineer, information architect, creator of the Ontology Pipeline framework, and a librarian by training who went on to build knowledge graphs at Amazon and Adobe.

Learn how proactive AI agents for data engineering detect pipeline failures, gather context, perform root cause analysis, and automate workflows across dbt, Airflow, Snowflake, and the modern data stack.

Most data teams now run more than one kind of AI on their operational work. A coding assistant, the AI built into the data tools they already own, a general-purpose model they prompt by hand. Almost none of them got the week back, because getting the week back takes four things at once, and no single tool holds more than one or two.

Your AI pilots return answers that look right, and nobody can say exactly why they don't hold up. The models are rarely the reason.

Learn what data engineering benchmarks for AI agents are, then compare leading benchmarks including ADE-bench, data-eng-bench, ELT-Bench, and DataClawEval to see what they measure, where they fall short, and how to evaluate agents on data engineering tasks.

An edited Q&A from the Data Splash podcast, hosted by Ido Bronstein and brought to you by Upriver. Our guest is Tiankai Feng, a data and AI leader with deep expertise in data governance and the author of two books, Humanizing Data Strategy and Humanizing AI Strategy. Tiankai has worked on data throughout his career — as a product owner, analyst, data scientist, and engineer — and now focuses on strategy and governance, with a particular interest in the human side of making data and AI work.

An edited Q&A from the Data Splash podcast, hosted by Ido Bronstein and brought to you by Upriver. Our guest is Abe Gong, CEO and co-founder of Great Expectations, the open-source data quality framework used by data teams everywhere. Abe came up through data engineering and data science, and kept discovering that the real blockers to building "cool algorithms" were on the engineering side - an experience that made him fascinated with how organizations manage what they know.

Snowflake and Databricks converged on the same 2026 roadmap: agentic data engineering. Where each stands, what they announced, and where the market is heading.

An edited Q&A from the Data Splash podcast, hosted by Omri and brought to you by Upriver. Our guest is Amaury Desrosiers, Director of Solution Consulting at Nimble, a real-time web data platform that deploys AI-powered web search agents to browse the live internet, verify what they find, and turn it into clean, structured data that AI agents and analytics can actually use. Amaury leads the solution consulting team, translating what customers need into how the product can solve it.

With backing from Hetz Ventures and Valley Capital Partners, the company is automating the data engineering work that has prevented enterprise AI initiatives from reaching production, and delivering real ROI.

An edited Q&A from the Data Splash podcast, hosted by Ido Bronstein and brought to you by Upriver. Our guest is Shachar Meir, a well-known data advisor who works with chief data officers and VPs of data to fix struggling data teams. Shachar previously served as Director of Data Engineering at Meta and led the data team at PayPal, and has spent more than 20 years in data — starting his career as a DBA.

For 30 years, every Semantic Layer attempt failed for the same reason: the people who carry the meaning were never the ones building the layer. AI Analytics is finally changing that.

An edited Q&A from the Data Splash podcast, hosted by Ido Bronstein and brought to you by Upriver. Our guest is Harel Shein, Senior Engineering Manager at Datadog, where he works on data observability products. Harel previously led data engineering and R&D teams at WeWork and Astronomer, and is a Technical Steering Committee member and contributor to OpenLineage, the open standard for data pipeline lineage.

A practical framework for data platform AI-readiness - 5 dimensions, before-and-after examples, quick wins, and a scorecard you can run with your team.

An edited Q&A from the Data Splash podcast, hosted by Ido Bronstein and brought to you by Upriver. Our guest is Prakash Reddy, Head of Data Engineering & AI Enablement at Atlassian. Prakash started his career in big data when "big" meant Oracle, was one of the early data practitioners at LinkedIn leading data products for Ads, and is an active investor and advisor for startups.

A practical framework for data platform AI-readiness - 5 dimensions, before-and-after examples, quick wins, and a scorecard you can run with your team.

Production AI agents need more than powerful models. Learn how tool design, context, memory, guardrails, orchestration, and domain-specific validation make agentic systems reliable.

Knowledge engineering disappeared into data pipelines and infrastructure. Now AI is bringing it back. Here's why 95% of AI investments fail without it - and what the new knowledge engineer looks like.

Learn why AI agents fail on unprepared data platforms - and how better context, metadata, testing, lineage, and guardrails make data engineering agent-ready.

AI agents need more than warehouse access. Learn why the data context layer requires human ownership, continuous evolution, and a new role: the Knowledge Engineer.

How Upriver & Nimble together allow building real-time pricing pipelines natively in Snowflake with UDTFs.

Learn why AI code generators struggle with data engineering and how data context semantics, lineage, usage, and rules—helps AI build reliable pipelines.

Learn why streaming data needs real-time quality monitoring, schema validation, anomaly detection, and governance to prevent bad data from spreading downstream.

See how Bright Data uses Upriver for data observability, automated data quality monitoring, and root cause analysis across hundreds of data sources.

In this post, we’ll break down why monitoring data in motion - as it’s pulled, interpreted, and acted on—is essential for building reliable agentic systems

Learn how shifting data ownership left gives producers control over data quality, observability, and governance for faster feedback and more reliable data.