General coding benchmarks can tell us whether an agent writes code.
Data engineering benchmarks tell us whether an Agent can understand a warehouse, modify a pipeline, preserve business logic, run the project, debug failures, and produce correct data.
That gap is why data engineering benchmarks for AI agents are starting to matter.
Agentic data engineering goes beyond code generation to context-aware action and validation. Writing valid SQL is only one small part of data engineering. A useful agent must also understand an unfamiliar warehouse, navigate a repository, trace dependencies, interpret business rules, execute code, diagnose failures, and verify that the resulting data is actually correct.
A new group of benchmarks including ADE-bench, data-eng-bench, ELT-Bench, and DataClawEval, is attempting to measure these capabilities more realistically.
This article explains what each benchmark measures, where it falls short, and how data teams can build a practical evaluation program of their own.
What is a data engineering benchmark for AI agents?
A data engineering agent benchmark is a repeatable set of tasks, execution environments, and graders designed to test whether an AI agent can complete real data work.
Instead of asking a model to return a SQL string, the benchmark places an agent inside a working environment. Depending on the benchmark, the environment may include dbt repositories, warehouses, sample data, pipeline code, documentation, and tests.
The agent then receives a task such as “repair this broken incremental model,” “build a customer lifetime value mart,” or “create and execute a streaming aggregation.” It must inspect the environment, implement a solution, run it, respond to errors, and produce a materialized result.
That means the thing being evaluated is not just the language model. It is the complete system:
Model + agent harness + tools + context strategy + execution environment + permissions
Both Snowflake and Databricks have reported that changing the agent harness while holding the underlying model constant can materially change quality, token usage, and total task cost.
In other words, a model does not have a single, context-free “data engineering score.”
Why SWE-bench and text-to-SQL benchmarks are not enough
General software-engineering benchmarks are still useful. They test whether agents can inspect repositories, edit files, run tests, and resolve issues. Those are important capabilities for a data engineering agent.
But data systems introduce their own failure modes.
A code patch can compile while a data model silently changes its grain. A query can run while mishandling late-arriving events. A pipeline can produce the expected columns while incorrectly allocating revenue across currencies. A model can pass on the first run but duplicate rows when executed again.
These are not simply syntax problems. They involve data semantics, lineage, state, timing, materialization behavior, and business rules.
Text-to-SQL benchmarks have made significant progress toward more realistic database evaluation. BIRD includes 12,751 question-and-SQL pairs across 95 databases, while Spider 2.0 contains 632 enterprise-oriented workflow problems involving large schemas, multiple SQL dialects, and cloud systems such as BigQuery and Snowflake.
But the primary outcome is still usually a query or answer. Most production data engineering work extends beyond that point.
A data engineering agent may need to:
- Understand an existing transformation DAG.
- Create or modify several interdependent models and tables.
- Validate row counts, keys, metrics, and edge cases
- Configure sources, debug compilation, data, or runtime failures
What should a good data engineering benchmark measure?
The best benchmarks evaluate more than code quality. They test how the agent behaves throughout the task.
Context discovery
A data engineering benchmark should test whether the agent can find the information required to complete the task. In a real data environment, context is distributed across multiple systems. The agent needs to distinguish useful evidence from nearby but unrelated information without reading the entire codebase or profiling every warehouse table.
This also means making your data platform agent-ready, with reliable context, clear structure, tests, and lineage that agents can use without human hand-holding.
Proactiveness
A capable agent should not need every investigative and validation step spelled out in the prompt. The benchmark should test whether it takes sensible follow-up actions on its own: inspecting upstream models, sampling source data, checking lineage, running relevant tests, reviewing the materialized output, and investigating results that look suspicious.
Repository and lineage reasoning
Data work rarely happens in a blank file. The agent should understand sources, staging layers, intermediate models, marts, macros, tests, and dependency references before changing the project.
Execution and debugging
The agent should run its work in a real or faithfully simulated environment. It must be able to interpret compiler output, warehouse errors, failing tests, and unexpected data.
Reliability across repeated runs
Agent behavior is nondeterministic. One successful attempt can hide an unstable system. A production-oriented benchmark should repeat tasks and report both average success and consistency.
Scope and safety
A benchmark should detect unnecessary file edits, destructive database operations, attempts to retrieve hidden solutions, or changes outside the requested scope. This becomes especially important when agents receive shell and warehouse access.
Top 6 leading data engineering benchmarks for AI Agents
1. ADE-bench
ADE-bench is an open-source framework created by Benn Stancil in collaboration with dbt Labs. It focuses on analytics and data engineering work inside realistic dbt projects.
Each task can include a complete dbt repository containing models, macros, tests, and configuration files, along with an actual database. The benchmark supports DuckDB environments and Snowflake variants, and each task runs inside a temporary sandbox.
ADE-bench can also modify the environment before the agent starts. A setup script might remove a comma, delete an intermediate model, alter source data, or introduce another failure. After the agent finishes, the benchmark primarily uses dbt tests and result-table comparisons to evaluate the solution. A task passes only when all of its required tests pass.
The results of this benchmark highlight how much both the model and the surrounding tooling matter. On DuckDB with dbt Core, OpenAI Codex with GPT-5.1 passed 56% of tests at a cost of about $14.90 . Claude Sonnet 4.5 landed on the same 56%, at nearly the same cost: two frontier models from two different labs, effectively indistinguishable. On Snowflake, Claude Sonnet 4.5 scored 50% with dbt Core at $13.72, adding MCP increased the pass rate to 54%.
This benchmark suggests that AI agents are already capable of solving a meaningful share of real data-engineering problems, but model choice alone does not determine performance.
The main limitation is that its center of gravity is analytics engineering and dbt. It does not attempt to represent the full range of ingestion, distributed processing, orchestration, and streaming work found in broader data engineering organizations.
2. Data-eng-bench
Snowflake’s data-eng-bench is another open-source benchmark for repository-level dbt work, developed with Bespoke Labs.
Its 103 tasks run against a shared retail warehouse containing 579 source tables, 19 schemas, and roughly 8,000 columns. The benchmark includes 84 build tasks and 19 fix tasks. Build tasks range from creating a new dbt project to extending an existing layered project, while fix tasks require the agent to diagnose and repair subtly incorrect models.
This benchmark requires agents to work through dbt mechanics rather than write isolated SQL. Solutions commonly involve source declarations, macros, materializations, multiple models, and ‘ref’ dependencies across a DAG.
Its most important design choice is the grading method. Each task includes a hidden verifier with multiple assertions that directly interrogate the materialized output.
3. ELT-Bench
ELT-Bench expands the evaluation scope beyond transformations. It asks agents to build complete extract-load-transform pipelines.
This benchmark contains 100 pipelines, 835 source tables, and 203 target data models. Agents work with heterogeneous sources including PostgreSQL, MongoDB, REST APIs, Amazon S3-style storage, and flat files. They must configure Airbyte and Terraform to extract and load the data, then create dbt models that transform it inside a warehouse.
It evaluates the orchestration of heterogeneous sources, ingestion tooling, warehouse loading, and transformation as one connected workflow. This benchmark suggests that despite strong text-to-SQL results on simpler benchmarks, Agents performance drops sharply on full data-engineering workflows.
The main limitation of the ELT-Bench is that the environment is relatively expensive and complicated to operate.
4. DataClawEval
DataClawEval takes a broader, multi-engine view of data engineering. It includes 100 end-to-end tasks across five execution engines - PySpark, MySQL, HiveSQL, PrestoSQL, & FlinkSQL.
A typical task requires an agent to inspect schemas and sample rows, infer join keys and transformation rules, write executable SQL or Python, run it in a sandbox, diagnose errors, and materialize the required output.
DataClawEval grades two dimensions.
The artifact score evaluates properties such as executability, schema correctness, row-level alignment, numerical accuracy, and business correctness.
The process score evaluates exploration, execution efficiency, and self-verification.
The final score generally gives more weight to artifact correctness while retaining a smaller process component.
The initial paper evaluated 16 agent configurations and reported a top overall score of 74.9 out of 100. It also found that no single model led across every execution engine and that making more tool calls did not correlate with better performance.
5. Databricks’ internal benchmark
Databricks’ benchmark is not a public data engineering benchmark in the same sense as ADE-bench or DataClawEval. It is an internal coding-agent benchmark built by Databricks for their internal evaluation.
The results showed 3 broad capability tiers, with cheaper models often handling routine work effectively while the strongest models remained better for harder tasks.
It still belongs in this discussion because it provides a practical blueprint for what mature teams may eventually need to build themselves.
Databricks selected recent, human-written pull requests containing strong test coverage and relatively self-contained changes. The tasks span more than ten languages and technologies, including Python, Scala, Go, Rust, Java, TypeScript, Bazel, Protobuf, and gRPC. Engineers rewrote task prompts to describe the desired outcome without revealing the original solution, then held tests back until the agent had completed its work.
6. Spider 2.0 and BIRD
Spider 2.0 and BIRD are better described as adjacent benchmarks than complete data engineering benchmarks.
Spider 2.0 includes 632 enterprise text-to-SQL workflow problems. Its databases can contain more than 1,000 columns and include cloud platforms such as BigQuery and Snowflake. Tasks may require consulting documentation, navigating schemas, and producing multiple SQL operations.
BIRD provides a large, cross-domain text-to-SQL dataset with 12,751 question-and-SQL pairs across 95 databases. It emphasizes realistic database contents, external knowledge, and efficient SQL generation.
These benchmarks are useful for isolating SQL reasoning and schema-linking ability. They can tell you whether an agent is likely to identify the right tables and produce a correct query.
This benchmark suggests that strong performance on conventional text-to-SQL benchmarks does not necessarily translate to real enterprise data environments, where schema complexity, reasoning, context handling, and multi-step workflows remain major challenges.
What the AI Data Engineering benchmarks tell us
The benchmark landscape is young, but several patterns already appear consistently.
Agent harness matters almost as much as the model
A model does not inspect a warehouse or edit a repository on its own. The harness determines what tools are available, how context is selected, how previous interactions are summarized, and when the agent runs validation.
In Snowflake’s published data-eng-bench experiments, changing the harness affected both quality and cost. Its strongest reported configuration achieved a 73.8% average single-attempt pass rate, while alternative harnesses using the same underlying model were less accurate or more expensive.
This is why production agentic systems depend as much on the harness, context, and guardrails as the underlying model.
Cost per token is less useful than cost per resolved task
Teams often compare model prices before they know how many tokens each agent will actually consume.
A cheaper model may explore more tables, reread more files, repeat unsuccessful queries, or take more turns before finishing. A more expensive model may solve the task with fewer interactions and a lower total bill.
Databricks reported an example in which a model that was cheaper per token cost more per task and scored lower because it consumed substantially more tokens.
Cost per resolved task = total model, tool, and compute cost ÷ fully resolved tasks
Reliability is different from peak capability
An agent that solves a task once and fails twice is not production-ready.
Snowflake’s top published configuration reached a 73.8% average Pass@1 score but passed on all three runs for only 64.1% of tasks. DataClawEval also found large differences between models’ best and worst runs, leading its authors to argue that repeated-run stability should be treated as a first-class quality dimension.
Teams evaluating autonomous or recurring workloads should care more about consistent success than an impressive best-case trace.
How data teams should benchmark AI agents internally
Public benchmarks are useful for screening tools and understanding broad capabilities. They cannot fully represent an individual company’s schemas, conventions, deployment process, or business logic.
A meaningful benchmark should therefore test not only whether an agent acts, but whether it acts for the right reason, at the right time, with the right scope. It should penalize false alarms, unnecessary exploration, speculative fixes, and actions taken without appropriate approval just as seriously as it penalizes missed incidents
What is still missing from the data engineering stack?
Today’s data engineering benchmarks answer an important question: can an agent complete a defined task in a realistic data environment?
But the production data engineering contains another step before the task begins. Someone then has to gather the relevant context and decide whether the issue requires a code change, a rerun, an escalation, or no action at all.
That work is still fragmented across observability platforms, orchestrators, warehouses, repositories, and the experience of individual engineers.
Better agents will need more than tools, they’ll need an AI Context Layer that keeps the system’s context available as they work.
The next improvement will come from connecting those systems so that an agent can understand what changed, investigate why it matters, and return a bounded, verifiable next action.
This is also how proactiveness should be evaluated. It is one that notices consequential issues early, gathers the right evidence, avoids unnecessary intervention, and knows when a human decision is required.
Upriver is one example of this emerging approach. By connecting context from the warehouse, codebase, orchestrator, lineage, and documentation, Upriver helps agents investigate scoped issues, recommend or perform bounded actions, and return evidence for review.
Public benchmarks are beginning to show whether agents can perform data engineering tasks.
The more difficult and useful test will be whether they can help maintain the data system between those tasks.
Evaluating agents on your own stack
Upriver is the AI Data Engineering Platform that connects your warehouse, code, orchestrator, and lineage into the grounded context an agent needs across Snowflake, Databricks, BigQuery, dbt and more so the work an agent does can be verified instead of trusted.
