How REAP is Solving the AI Coding Agent Benchmark Crisis
How REAP is Solving the AI Coding Agent Benchmark Crisis
Let's be honest: most AI coding benchmarks are garbage.
They measure how well models perform on curated datasets that look nothing like what happens when developers actually use these tools in production. It's like training for a marathon by running on a treadmill in a climate-controlled gym and then wondering why you gas out three miles into a muddy trail race.
Meta's research team just dropped a paper that takes a serious crack at fixing this problem. They've built REAP (Relevance and Execution-Audited Pipeline), an automated system that constructs evaluation benchmarks directly from real developer-agent interactions in production environments.
The Benchmark Problem Nobody Talks About
Here's what most people don't realize about evaluating AI coding agents: the methods companies actually use in production are all deeply flawed.
Online A/B testing gets you real signals, but it takes weeks to run and you're gambling with actual user experience. Ship a broken agent, watch your metrics tank, cry into your incident report.
Shadow deployment lets you test in parallel without affecting users, but the results aren't reproducible. Run it again tomorrow and you'll get different numbers because your codebase changed, your test cases shifted, or the agent had an off day.
Public benchmarks like HumanEval or MBPP? They're fine for academic comparisons, but they don't reflect real production workloads. Different languages, different prompt styles, different codebase structures. A model crushing HumanEval might stumble badly on your actual monorepo.
The research team identified something crucial: what we actually need is in-distribution evaluation — benchmarks that match how developers will actually use these agents. But building those benchmarks manually is a nightmare.
Enter REAP: Automated Curation at Scale
REAP solves this by automatically building production-derived benchmarks from real developer-agent sessions. Instead of hand-crafting test cases, the system pulls from actual usage, verifying each task automatically so humans don't have to babysit the process.
The pipeline addresses three reliability killers that plague automated benchmark curation:
Untestable prompts — Some tasks simply can't be verified automatically. REAP filters these out.
Misaligned tests — The test written for a task might not actually verify what the task asks for. REAP validates test relevance agentically.
Test flakiness — Results that bounce between pass and fail make metrics meaningless. REAP runs stability checks across multiple executions.
The key insight is that in large monorepos, build infrastructure state is ephemeral. A benchmark you curate today might be invalid tomorrow as the codebase evolves. Manual auditing can't keep up with that cadence, so REAP automates the verification layer entirely.
HARVEST: Production-Grade Benchmarks Built for the Real World
The team used REAP to create HARVEST, a benchmark where every task comes from a real developer prompt and gets verified against fail-to-pass tests pulled from production.
HARVEST spans multiple programming languages (with a majority of tasks from Hack), giving a much more realistic picture of model capabilities across different ecosystems.
The results? Frontier models achieved solve rates between 42.9% and 58.2% — numbers that actually mean something because they come from real production scenarios, not synthetic test cases.
Why This Matters for Your Team
If you're evaluating AI coding agents for your organization, REAP-style approaches are worth understanding for a few reasons:
Faster iteration cycles — Automated curation means you can update benchmarks continuously as your codebase evolves, without armies of human labelers.
Trustworthy signals — When the benchmark comes from actual production usage, you're measuring what actually matters for your deployment.
Informed deployment decisions — The capability differences surfaced by realistic benchmarks help you choose which models actually fit your workflow.
The gap between "works great in demos" and "works great in production" has always been where AI tools die. REAP represents a serious attempt to close that gap by bringing evaluation closer to reality.
Source: arxiv.org/abs/2604.01527