AI Agent Arenas: How to Build and Run Long-Horizon Agent Evaluations

    Static leaderboards cannot answer the questions that matter for long-horizon agents. This article explains why single-turn benchmarks break down on multi-step, stateful, side-effect-producing tasks, and walks through the mechanics of arena-style evaluation: blind pairwise matchups, rating aggregation with Elo-style online updates and Bradley–Terry batch estimation, and bootstrap confidence intervals that keep rankings honest. It then covers layered leaderboards split by deployment constraints, task domain, and time window, along with the fine-grained metrics that actually drive engineering decisions: net improvement rate, command-line execution reliability, steerability, and tool, context, and cost efficiency. Finally, it describes reproducible evaluation infrastructure — sandboxed and deterministic execution, tiered scoring, anti-contamination practices, and an internal arena for model and toolchain selection — plus a checklist of common evaluation pitfalls.

    1. Why Static Benchmarks Cannot Answer Questions About Long-Horizon Agents

    Most traditional NLP benchmarks rest on a “single-turn, stateless, side-effect-free” assumption: you present a prompt, the model returns an answer, and you score it with string matching or human ratings. That paradigm works well for question answering, summarization, and translation, but it breaks down quickly once the target is an agent.

    Long-horizon agent tasks typically have four characteristics:

    1. Multiple steps with state dependencies: the correct action at step 10 depends on the real state that the previous nine steps left behind in the environment. Being correct on any single step does not mean the overall trajectory is correct.
    2. Tool and environment side effects: file writes, process launches, and service calls genuinely change the state of the world, so evaluation must be able to reconstruct or isolate those side effects.
    3. Error accumulation and irreversibility: one bad command can produce a dirty state that takes dozens of steps to recover from. Failures cascade rather than add up linearly.
    4. Mid-course intervention: real users add constraints and redirect the work while it is in progress, so “quality of the one-shot output” is not enough to characterize the experience.

    For exactly these reasons, the industry has been moving toward dynamic, arena-style evaluation: real long-horizon sessions initiated by real users serve as the “question bank,” candidate agents execute the same class of tasks under anonymized conditions, and real users or automated judges supply preferences that are aggregated into a layered leaderboard. In essence, this replaces static exam questions with an online stream of adversarial samples.

    2. The Core Mechanics of an Arena: Blind Pairing and Score Aggregation

    2.1 Anonymous Pairing to Remove Prior Bias

    The basic unit of an arena is the “match.” For the same task, two candidate systems are scheduled in parallel, and their outputs are presented to the reviewer anonymously as A/B with randomized left-right placement. This design removes two kinds of systematic bias: first, brand priors (reviewers rate a familiar name higher); second, position preference (a general tendency to pick the option listed first). On the engineering side, the presentation layer should forcibly strip every identity marker, including stylistic phrasing, fixed opening lines, and habitual formatting — these “implicit watermarks” undermine blind testing just as surely as an explicit name would.

    2.2 From Match Outcomes to an Ordered Ranking

    Win/loss sequences need to be aggregated into comparable scores. Two families of methods are common in practice:

    • Online update style (Elo-like): scores are adjusted after each match according to the gap from the expected win rate. Computation is cheap, which suits live leaderboards.
    • Batch estimation style (Bradley–Terry-like): maximum likelihood estimation over all match results. Statistically more efficient, and well suited to offline recomputation.

    Illustrative implementation (online update style):

    1. def expected_score(r_a: float, r_b: float) -> float:
    2. return 1.0 / (1.0 + 10 ** ((r_b - r_a) / 400))
    3. def elo_update(r_a: float, r_b: float, score_a: float, k: int = 24) -> tuple[float, float]:
    4. """score_a: 1.0 means A wins, 0.5 means a tie, 0.0 means A loses"""
    5. e_a = expected_score(r_a, r_b)
    6. delta = k * (score_a - e_a)
    7. return r_a + delta, r_b - delta

    Key engineering detail: the K value determines how sensitive the leaderboard is to recent samples. Too large and rankings jitter; too small and convergence drags. Candidates with the same score should be ordered by confidence interval rather than by point estimate.

    2.3 Sample Size and Confidence Intervals Must Be Reported Together

    The credibility of a leaderboard is determined jointly by the scale of participating sessions and by match coverage for each candidate pair. Once the underlying session volume reaches five or six figures, the overall ranking usually has adequate statistical power — but for sub-dimensions (the win rate on a specific capability, for example), the effective sample is often far smaller than the total. The standard practice is therefore to publish a confidence interval alongside every ranking: candidates whose intervals overlap should not be described as showing a real gap.

    1. import random
    2. def bootstrap_ci(wins, ties, losses, iters=5000, alpha=0.05):
    3. pool = [1.0] * wins + [0.5] * ties + [0.0] * losses
    4. stats = []
    5. n = len(pool)
    6. for _ in range(iters):
    7. sample = [pool[random.randrange(n)] for _ in range(n)]
    8. stats.append(sum(sample) / n)
    9. stats.sort()
    10. lo = stats[int(alpha / 2 * iters)]
    11. hi = stats[int((1 - alpha / 2) * iters) - 1]
    12. return round(sum(stats) / iters, 4), (round(lo, 4), round(hi, 4))

    3. Layered Leaderboards: Ranking by Constraint Dimension, Not a Single Aggregate

    A single aggregate leaderboard has limited practical decision value, because users operate under very different constraints. A mature leaderboard design makes at least two layers of cuts:

    First layer: split by deployment constraints. Open-weight models and models offered only as an API service differ completely in data sovereignty, offline deployment, fine-tuning feasibility, and per-call cost structure. Putting the two groups on one aggregate leaderboard looks intuitive but hides the real motive behind “why would I pick this one.” The standard practice is to publish an overall leaderboard plus separate open-weight and API-service leaderboards, clearly labeling the candidate set boundary of each.

    Second layer: split by task domain. Agent tasks generally fall into several broad categories, for example office workflows (document processing, information organization, moving data across systems), software engineering (code comprehension, defect fixing, the build-and-test loop), and open-ended conversation (consultation, clarification, solution discussion). Within a single evaluation, capability levels across these three domains are often out of sync, so “first in one domain” and “first overall” are two different claims — any citation must carry the qualifier.

    Third layer: split by time window. A leaderboard is a snapshot, not a permanent conclusion. The statistical window should be labeled explicitly (for example, the snapshot date of a real-time board, or the historical ranking cited around a financial disclosure date), so that rankings from different points in time are not mixed and presented as facts about the same moment. A common historical misuse is treating a model’s ranking “at launch” as equivalent to its “current” ranking — a practice that produces large distortions during periods of rapid version iteration.

    4. Fine-Grained Metrics That Long-Horizon Agent Evaluation Must Add

    An aggregate score is only useful for coarse screening. What actually drives engineering decisions are decomposable, fine-grained metrics.

    4.1 Task Completion and Net Improvement Rate

    Task completion is a binary or graded judgment (fully achieved / partially achieved / failed). Completion alone does not reflect the experience, because “the same success can take several times the steps and elapsed time.” Metrics of the net improvement family take this approach: against a baseline system, on the same task set, count the win/tie/loss distribution of each task relative to the baseline, and offset positive gains against negative regressions to produce a net value.

    $
    \text{NetImprovement} = \frac{N{win} - N{loss}}{N_{total}} \times 100\%
    $

    The value of this metric is that it penalizes regressions as well: a candidate that leads slightly on most tasks but degrades severely on a few critical ones will see its net value pulled down substantially. When the delta against the previous-generation baseline is cited, the baseline’s conditions must be stated at the same time (same prompt version, same tool set, same budget), otherwise the difference is not comparable.

    4.2 Command-Line Execution Reliability

    Failures in software-engineering agents overwhelmingly happen not in “writing code” but in the “run and verify” stage. This dimension needs to cover:

    • Command construction correctness: whether arguments, quotes, paths, and wildcards are properly escaped;
    • Exit code handling: whether non-zero return codes are read and handled correctly rather than pushed past;
    • Failure recovery: after an error, whether the agent can read the output, locate the cause, and fix and retry;
    • Dangerous operation constraints: whether it avoids irreversible actions such as recursive deletion without confirmation, force pushes, and overwrite writes.

    In evaluation, it is better to collect “dangerous operation trigger rate” and “one-shot fix success rate after failure” as separate metrics instead of folding them into the aggregate score.

    4.3 Steerability

    This is the most underrated metric for long-horizon agents: it measures how well the system converges after a user raises a new requirement mid-course or points out a mistake. Quantifiable sub-items include:

    • Response correctness: whether the first action after intervention complies with the new instruction;
    • Redundant redo rate: whether it ignores already-completed work and starts the whole task over;
    • Correction rounds: how many turns are needed to get back on the correct track after an intervention;
    • Constraint retention: whether fixing problem A breaks a previously satisfied constraint B.

    Illustrative evaluation script skeleton:

    1. def evaluate_steerability(env, task, interventions):
    2. result = run_agent_until(env, task, stop_when="awaiting_input")
    3. rounds = []
    4. for step in interventions:
    5. reply = result.respond(step.instruction)
    6. rounds.append({
    7. "compliant": judge_compliance(reply, step.expected_effect),
    8. "constraints_kept": check_constraints(env, step.prior_constraints),
    9. })
    10. return {
    11. "first_turn_compliance": rounds[0]["compliant"],
    12. "constraint_retention": mean(r["constraints_kept"] for r in rounds),
    13. "total_rounds": len(rounds),
    14. }

    4.4 Tool Calls, Context, and Cost

    Also worth collecting: parameter validity of tool selection (JSON Schema pass rate), number of calls and redundant call ratio, context management strategy (whether unnecessary history truncation or summary loss occurs), end-to-end latency distribution (P50/P95 rather than the mean), and per-task resource and call cost. Cost metrics are especially necessary — at equal win rates, a candidate that costs twice as much is often not viable in production.

    5. Reproducible Evaluation Engineering

    The credibility of a leaderboard ultimately depends on the evaluation infrastructure. If different candidates run in different environments, the comparison is meaningless.

    Sandbox isolation and determinism. Each task runs in its own container, with fixed dependency versions and a pinned system image digest, networking disabled or restricted to an explicit allowlist, and writes limited to a temporary volume. Environments are recycled uniformly when evaluation finishes, so dirty state does not carry over and contaminate other tasks.

    1. harness:
    2. sandbox:
    3. image: "agent-eval-base@sha256:<digest>"
    4. network: none
    5. resources: { cpu: 2, memory: "4Gi" }
    6. timeout_seconds: 900
    7. task:
    8. corpus: "object://eval-bucket/corpus-v3" # scenario corpus stored in object storage
    9. seeds: [7, 19, 43] # multiple random seeds for stable conclusions
    10. scoring:
    11. - rule_based # assertions on structured files/artifacts
    12. - unit_tests # compilation, tests, static checks
    13. - llm_judge # subjective dimensions; agreement must be reported
    14. report:
    15. metrics: [pass_rate, net_improvement, steerability, p95_latency, cost]
    16. confidence: bootstrap

    Tiered scoring. Prefer deterministic criteria first: does it compile, do unit tests pass, are the fields in the target file correct, are interface contracts satisfied. Only then bring in reviewer models or human reviewers for subjective dimensions, and always report reviewer agreement (an inter-annotator agreement coefficient, for example) plus a symmetry check on reviewer position; when agreement is too low, that dimension should not enter the aggregate score.

    Anti-contamination and anti-overfitting. Build prompt deduplication and similarity detection to remove samples that heavily overlap with public evaluation sets; use time-based splits and rotate hidden cases periodically; version-record changes to prompts, tool sets, and sampling parameters so that leaderboard updates remain traceable.

    An internal arena for technology selection. Organizations can reuse the same mechanism directly to choose models and toolchains: first collect de-identified tasks from real tickets and engineering workflows as the corpus, freeze a baseline version, then plug a new candidate into the same sandbox to run the same corpus, using net improvement rate and P95 latency as release gates. These capabilities are typically hosted on general-purpose cloud infrastructure: a container platform provides isolated execution, object storage holds corpora and artifacts, a message queue decouples task scheduling from scoring, and logging and monitoring services support metric dashboards and regression alerts.

    6. A Checklist of Common Pitfalls

    1. Treating the aggregate leaderboard as the conclusion: ignoring confidence intervals and task-domain differences leads to a mismatch between the selection and the business goal.
    2. Baseline condition drift: comparing against the “previous generation” without fixing the prompt, tool set, and budget makes the difference untrustworthy.
    3. Reporting success but not regressions: without net-value metrics, severe backsliding in key scenarios stays hidden.
    4. Inconsistent environments: different sandboxes, dependencies, and network policies across candidates make results non-comparable.
    5. Uncorrected reviewer bias: no anonymization, no position randomization, no agreement checks.
    6. Overfitting to public evaluation sets: tuning long-term on the same public set raises the ranking while the real user experience declines.

    7. Summary

    The value of an agent arena is not producing a “which place did it finish” conclusion, but providing a measurement method that can keep running: real long-horizon sessions as the sample stream, blind pairwise matching and score aggregation for comparable rankings, layered leaderboards that answer selection questions under different constraints, and fine-grained metrics that pinpoint specific weaknesses. For engineering teams, the practical focus is to make evaluation into infrastructure — fixed sandboxes, deterministic scoring, reproducible experiments, explicit confidence intervals, and release gates. Only when the evaluation itself can be audited and recomputed do ranking numbers carry any real weight in guiding production decisions.