Engineering Trade-offs in Benchmarking AI Coding Agents on Long-Horizon Tasks
Benchmarking AI coding agents on long-horizon engineering work is fundamentally a multi-dimensional optimization problem, not a single score. This article breaks down what makes long-horizon tasks hard, explains the four core metrics (pass rate, completion, latency, and cost) and why they are strongly negatively correlated, and shows how the evaluation scaffolding often affects results more than the model itself. It walks through how to correctly read leaderboard data, maps different agent strategies to a decision matrix, and lays out a practical method for building your own benchmark with tiered acceptance gates, multi-run sampling, and budget-limited validation loops. It also covers common pitfalls and offers concrete guidance for putting these evaluations into practice.
1. First, understand what makes “long-horizon engineering tasks” hard
There is a clear dividing line between everyday coding assistance and long-horizon engineering work. The former is short-horizon: completing a single function, writing a parsing routine, or explaining an error message. The context is small, the success criteria are clear, and a single round of interaction usually converges. The latter refers to work that can only be finished by crossing dozens of files, hundreds of tool calls, and several hours of wall-clock time. Typical forms fall into four categories:
- Application scaffolding: starting from an empty directory, deciding on the project structure, dependency management, and build configuration on your own, until you produce a runnable, testable project.
- Feature development: locating the insertion point in an existing codebase, filling in the UI, logic, and data layers, while staying consistent with the original architectural style.
- Code migration: batch-replacing a set of legacy interfaces and API usages with a new system, which requires auditing every call site and verifying semantic equivalence.
- Platform porting: mapping an interface paradigm, lifecycle model, and permission model from one mobile ecosystem to another, a classic cross-paradigm refactor.
The challenges can be summarized in four points. First, state management: a task may span several hours and involve failures and retries along the way, so the agent must maintain its own task list and be able to restore its state after an interruption. Second, context decay: early constraints (for example, “do not introduce new dependencies”) are easily drowned out by the large volume of subsequent output, so retrieval, summarization, or external memory mechanisms are needed as a fallback. Third, feedback-loop quality: whether the model can actually read and write files, run builds, execute tests, and inspect logs determines whether it is “guessing” or “verifying,” which is one of the main sources of the completion gap. Fourth, global consistency: a locally correct change may break global constraints, such as dependency version conflicts, interface signature drift, or broken build scripts. Any evaluation that only measures “single-point code correctness” cannot capture these capabilities.
2. Four core metrics: why you cannot look at just one
These benchmarks usually report four quantities at the same time, each answering a different question.
Pass rate is a task-level binary verdict: a task passes only when it end-to-end meets all acceptance criteria, and partial completion scores zero. It measures the upper bound of capability, but it is insensitive to minor model version iterations and tends to have high variance. Average completion breaks the acceptance criteria into a number of checkboxes and measures the proportion satisfied, which is a continuous signal with more value for trend monitoring than pass rate. Average latency is wall-clock time, including model inference, tool execution, build queuing, and retry overhead. Cost per benchmark run is typically composed of token billing plus sandbox infrastructure overhead.
The key conclusion is that these four metrics are strongly negatively correlated. To push pass rate up, you usually rely on more exploration rounds, longer context, and more trial-and-error, which simultaneously drives up latency and cost; conversely, compressing the budget and timeout ceiling almost inevitably drops the pass rate. So “a given approach leads on one dimension” is not a conclusion on its own — it has to be viewed in the multi-dimensional space.
A few practical guidelines for choosing metrics: use pass rate to judge the upper bound of capability; use completion for regression and trend monitoring; use P50/P95 latency percentiles rather than the mean for SLA design, because the long tail severely skews the mean; and use cost as a feasibility gate — first decide “can we afford it,” then discuss “how good is the result.”
3. Hidden variables: the evaluation scaffolding often affects scores more than the model itself
This is the most easily overlooked point and the most common source of misreading. For the same model, swapping in a different agent scaffolding (system prompt, available tool set, context-compression strategy, whether sub-task parallelism is supported, retry and rollback strategy, sandbox base image) can produce differences of more than ten percentage points — this is not uncommon. The reason is that the success or failure of long-horizon tasks depends heavily on “whether the tools are used correctly” and “whether the context is managed well,” and both are determined mainly by the scaffolding, not by the model weights.
So any cross-comparison must first fix the following variables, otherwise the scores are not comparable:
- Exactly the same task set and acceptance scripts, with the scripts unmodified before and after evaluation;
- The same maximum turns, wall-clock timeout, and cost budget ceiling;
- The same sandbox environment, dependency image, and network policy (whether pulling external dependencies is allowed);
- The same judge, preferably automated checks like “build passes + tests pass,” to avoid the subjective drift introduced by human evaluation.
If you remember only one thing: when reporting scores, you must record the model version, the scaffolding version, and the evaluation date — none of the three can be missing.
4. How to correctly read a leaderboard
The table below is a redacted illustration showing the three typical shapes that this kind of data exhibits. (The values reflect the common magnitudes seen on public leaderboards, used to show the coupling between metrics; they do not represent any specific product.)
| Shape | Pass Rate | Avg. Completion | Avg. Latency | Cost per Run | Strategy Trait |
|---|---|---|---|---|---|
| Top-of-leaderboard | Highest (~28%–29%) | Highest | Moderately long | Medium-high | Thorough exploration, most balanced trade-off |
| Long-horizon deep | Second highest (~24%) | High (~82%) | Longest (~27 hours) | Highest (~$300 range) | Trades time for quality |
| Lightweight and fast | Medium (~17%) | Medium (~71%) | Shortest (~12 hours) | Lowest (~$10 range) | Trades speed for cost |
Three interpretations are worth emphasizing.
First, “high completion + long latency” is a classic exploration strategy. It is willing to spend several times the time and cost to repeatedly verify, trading resource redundancy for correctness. This shape suits unattended scenarios: overnight batch refactors, one-time codebase migrations, and rewrites of correctness-critical core modules — settings where wall-clock time is not sensitive and the cost of rework is extremely high.
Second, “slightly lower pass rate but fast and cheap” is a heuristic strategy. It quickly produces usable drafts and leaves the last part of the finishing work to humans. This suits interactive completion, improvement suggestions during code review, and rapid prototype validation. Note that its cost-effectiveness is not low: completion is still around 70%, meaning most of the work is already done and humans only need to handle the key acceptance items.
Third, pay special attention to the gap between completion and pass rate. When completion is clearly higher than pass rate (for example, 82% vs. 24%), it means “a lot was done, but the key acceptance items were not met.” This usually reveals not a code-generation weakness but a shortfall in task planning and self-verification — the model failed to recognize the hard constraints in the acceptance criteria, or did not proactively run tests to discover what it missed. This kind of shortfall is easier to mitigate with engineering means than a pure generation gap, for example by explicitly injecting the acceptance criteria into the prompt and enforcing build-and-test gates.
5. A decision matrix for engineering trade-offs
Mapping the shapes above to real-world selection:
- Budget-sensitive, high-frequency invocation (large teams, high daily task volume): choose the low-cost approach and accept humans filling in the tail work. Here pass rate is not the key; completion and cost are.
- Latency-sensitive, interactive use: choose the short-latency approach and set strict timeout and turn limits, handing control back to the user when it times out.
- Correctness-sensitive, one-time delivery (migrations, pre-launch refactors): choose the long-latency, high-completion approach, paired with full automated acceptance and human review.
- Want both cost and quality: do not expect a single approach to solve it; switch to tiered scheduling — first use a low-cost approach to triage tasks and produce first drafts, then use a high-intensity approach to attack only the few tasks flagged as hard. This is usually an order of magnitude cheaper than applying the strongest approach uniformly across the board.
6. A practical method for building your own benchmark
Leaderboards are an external reference, but what really drives iteration is your own evaluation asset. The steps below are recommended.
1. Build the task set: sample from the real issue pool and code review records rather than writing your own problems. Filter out three kinds of noise — tasks that depend on private services, tasks that need human confirmation to proceed, and tasks whose environment is not reproducible. In terms of scale, 20–50 tasks are already enough to reflect trends, with around 30 being a common balance point.
2. Script the acceptance criteria and gate them in tiers:
- L0: the project builds, and static checks show no new warnings;
- L1: all unit tests pass, with coverage no lower than the baseline;
- L2: integration/end-to-end cases on critical paths pass;
- L3: human review, looking only at the parts that passed L2.
3. Sample multiple times: run at least 3–5 independent runs, report both pass@1 and pass@k, and provide confidence intervals. A single run’s score carries almost no information, because the variance of long-horizon tasks is large.
4. Front-load the cost screening: first run L0/L1 to filter out obviously failing tasks, then run the full suite only on the survivors, which saves most of the review effort.
Illustrative code for metric aggregation and utility evaluation:
# Pseudocode: metric aggregation and utility evaluation for a long-horizon engineering benchmarkfrom dataclasses import dataclassimport statistics as st@dataclassclass RunRecord:task_id: strpassed: boolcompletion: float # proportion of acceptance criteria met, 0.0 ~ 1.0latency_hours: float # end-to-end wall-clock latencycost_usd: float # cost per runtool_calls: intretries: int@dataclassclass Report:pass_rate: floatmean_completion: floatp50_latency: floatp95_latency: floatmean_cost: floatdef summarize(runs: list[RunRecord]) -> Report:n = len(runs)assert n > 0, "empty sample set"lat = sorted(r.latency_hours for r in runs)return Report(pass_rate=sum(r.passed for r in runs) / n,mean_completion=st.fmean(r.completion for r in runs),p50_latency=lat[int(0.50 * (n - 1))],p95_latency=lat[int(0.95 * (n - 1))], # the long tail dictates the SLA; do not look at the meanmean_cost=st.fmean(r.cost_usd for r in runs),)# Normalize to 0~1 then weight; weights are configured by business priority, not fixeddef utility(rep: Report, base: Report,w_pass=0.45, w_comp=0.20, w_speed=0.20, w_cost=0.15) -> float:return (w_pass * rep.pass_rate+ w_comp * rep.mean_completion+ w_speed * base.p50_latency / rep.p50_latency+ w_cost * base.mean_cost / rep.mean_cost)
Here is an illustrative evaluation loop with a budget gate. The key point is that every round must run the real gates, otherwise you get no valid feedback:
# Illustrative evaluation loop: fixed budget ceiling, enforced validation loopMAX_TURNS = 400WALL_CLOCK_LIMIT_H = 30.0BUDGET_USD = 20.0def run_task(agent, task, sandbox):transcript, spent, t0 = [], 0.0, clock()for _ in range(MAX_TURNS):if (clock() - t0) / 3600 > WALL_CLOCK_LIMIT_H or spent >= BUDGET_USD:break # budget/time gate, suppresses unbounded explorationaction = agent.plan(transcript) # task description + historical trajectoryobs = sandbox.execute(action) # file read/write / build / test / logsspent += obs.token_costtranscript.append((action, obs))if obs.gate_passed: # gate: build + lint + testbreakverdict = grade(task, sandbox) # unified judge, task-level binary resultreturn verdict, spent, (clock() - t0) / 3600
7. Common pitfalls and anti-patterns
- Reporting only pass rate, not cost. This makes selection conclusions completely distorted, because high scores are often bought with budget.
- Using the mean instead of percentiles. The latency of long-horizon tasks follows a long-tailed distribution; the mean hides the worst case, and the worst case is exactly what determines user experience and timeout strategy.
- Ignoring data contamination and leaderboard overfitting. If the benchmark tasks and their solutions already appear in the training corpus, scores are systematically inflated. The mitigation is to keep a private, continuously rotating held-out set and use the public set only for reference.
- Not recording versions and dates. Model versions drift silently, and a score without version information becomes invalid three weeks later; historical data also cannot be reproduced.
- Replacing automated acceptance with human scoring. Human scoring introduces inter-rater inconsistency and is hard to regress. Human review should only be an L3 supplementary layer.
- Unbounded automated exploration. An agent without a time/cost gate will consume resources indefinitely on unsolvable tasks; convergence must be enforced through both budget and turn limits.
8. Practical recommendations
- Prioritize tiered scheduling over single-approach selection. A low-cost approach handles task triage and first drafts, while the high-intensity approach attacks only the few tasks flagged as hard. This is the most cost-effective structure.
- Enforce the validation loop. Any code produced by an agent must pass build, static-check, and test gates before it enters review, turning “self-verification” from a model behavior into a process constraint.
- Minimize the sandbox and permissions. Agents execute in an isolated environment with only the minimum permissions needed to complete the task; keys and credentials are issued as temporary credentials, and direct access to production is prohibited.
- Provide observability support. Use logging services and monitoring alerts to record each agent run’s token consumption, latency, failure reasons, and turn distribution, forming cost attribution — this is what lets you answer “which stage is the money going to, and which tasks are the most expensive.”
- Use the benchmark as a regression test. The evaluation set should be part of continuous integration, running automatically every time the prompt, scaffolding, or model version changes, so score fluctuations become immediately visible and silent degradation is avoided.
To sum up: the real value of these leaderboards lies not in the ranking, but in the fact that they reveal that benchmarking long-horizon engineering tasks must simultaneously constrain three dimensions — quality, latency, and cost. Understanding where each approach sits on this trade-off curve is far more meaningful for engineering than memorizing any specific number.
