How REAP is Solving the AI Coding Agent Benchmark Crisis

Let's be real: most AI coding benchmarks are broken.

They measure how well models perform on neatly curated datasets that have nothing to do with what actually happens when developers use these tools in real production environments. It's like training for a marathon on a treadmill and then wondering why you collapse three miles into an actual race.

Meta's research team just published a paper that takes a serious swing at fixing this. They built REAP (Relevance and Execution-Audited Pipeline), an automated system that creates evaluation benchmarks directly from real developer-agent interactions in production.

The Benchmark Problem Nobody Talks About

Here's what most people miss about evaluating AI coding agents: the methods companies actually use are all deeply flawed.

Online A/B testing gives you real signals, but it takes weeks to run and you're gambling with actual user experience. Deploy a broken agent, watch your metrics tank, write your incident report.

Shadow deployment lets you test in parallel without affecting users, but the results aren't reproducible. Run it again tomorrow and you'll get different numbers because your codebase changed, your test cases shifted, or the agent had an off day.

Public benchmarks like HumanEval or MBPP? Fine for academic comparisons, but they don't reflect real production workloads. Different languages, different prompt styles, different codebase structures. A model crushing HumanEval might completely bomb on your actual monorepo.

The research team spotted something crucial: what we actually need is in-distribution evaluation — benchmarks that match how developers will actually use these agents. But building those benchmarks manually is a nightmare.

Enter REAP: Automated Curation at Scale

REAP solves this by automatically building production-derived benchmarks from real developer-agent sessions. Instead of hand-crafting test cases, the system pulls from actual usage and verifies each task automatically so humans don't have to babysit the process.

The pipeline tackles three reliability killers that plague automated benchmark curation:

1. Untestable prompts — Some tasks simply can't be verified automatically. REAP filters these out.

2. Misaligned tests — The test written for a task might not actually verify what the task asks for. REAP validates test relevance agentically.

3. Test flakiness — Results that bounce between pass and fail make metrics meaningless. REAP runs stability checks across multiple executions.

The key insight is that in large monorepos, build infrastructure state is ephemeral. A benchmark you curate today might be invalid tomorrow as the codebase evolves. Manual auditing can't keep up with that cadence, so REAP automates the verification layer entirely.

HARVEST: Production-Grade Benchmarks Built for the Real World

The team used REAP to create HARVEST, a benchmark where every task comes from a real developer prompt and gets verified against fail-to-pass tests pulled from production.

HARVEST spans multiple programming languages (with a majority of tasks from Hack), giving a much more realistic picture of model capabilities across different ecosystems.

The results? Frontier models achieved solve rates between 42.9% and 58.2% — numbers that actually mean something because they come from real production scenarios, not synthetic test cases.

Why This Matters for Your Team

If you're evaluating AI coding agents for your organization, REAP-style approaches are worth understanding for a few reasons:

• Faster iteration cycles — Automated curation means you can update benchmarks continuously as your codebase evolves, without armies of human labelers.

• Trustworthy signals — When the benchmark comes from actual production usage, you're measuring what actually matters for your deployment.

• Informed deployment decisions — The capability differences surfaced by realistic benchmarks help you choose which models actually fit your workflow.

The gap between "works great in demos" and "works great in production" has always been where AI tools die. REAP represents a serious attempt to close that gap by bringing evaluation closer to reality.

Source: arxiv.org/abs/2604.01527