AI Agent Arenas: Designing and Building Agent Evaluation Systems

    Static benchmarks such as multiple-choice and coding tests measure a model's single-step capability but fall short for AI agents, where users mainly care whether a task actually gets completed. This article explains how Agent Arena leaderboards are built from real user tasks, blind head-to-head comparisons, and continuously updated rankings, and how to interpret key metrics including net improvement rate, shell execution reliability, and steerability. It also offers guidance on viewing rankings rationally and building your own evaluation pipeline—from task collection and sandboxed execution to result judging, metric calculation, and cloud-based implementation—for developers and teams evaluating or selecting AI agent products.

    Introduction: When Benchmarks Are No Longer Enough

    Historically, evaluating large language models relied mainly on static benchmarks such as multiple-choice and coding questions. These evaluations have clear input-output boundaries and answers that can be graded automatically, but they only capture a model’s single-step capability. When deployed as an AI agent, what users really care about is “whether the task gets done”: can it understand a vague goal, plan reasonable steps, call tools, run commands, read results, and self-correct in a real environment, and finally deliver an output that meets expectations.

    Static benchmarks struggle to answer these questions for three reasons:

    • Real tasks require multi-turn interaction, along with changing environments and mid-course failures;
    • The same task often has multiple correct solutions, so there is no single answer to grade against;
    • Models iterate quickly and have many versions, so evaluation sets are easily covered by training data, leading to inflated scores.

    Against this backdrop, the “Agent Arena”—centered on real tasks, blind head-to-head battles, and continuously updated leaderboards—has gradually become an important window for observing agent capabilities. Several recently released models cited arena leaderboard results at launch, and the leaderboard has become a key reference for measuring the product strength of agent offerings. This article addresses three questions:

    1. How are rankings on these leaderboards produced;
    2. How should the key metrics in the leaderboard be interpreted;
    3. How developers should view leaderboards correctly and build their own evaluation system.

    How Arena Leaderboards Are Produced

    Task Sources: From “Exam Questions” to Real Workflows

    Tasks in an arena typically come from real user-initiated sessions rather than manually written questions. Such tasks share the following characteristics:

    • Long-horizon: spanning multiple rounds of dialogue and multiple tool calls, taking anywhere from minutes to hours;
    • Open-ended goals: users often describe only their intent without specifying steps, leaving “how to solve it” entirely to the agent;
    • Complex environments: requiring access to the file system, running commands, browsing web pages, and calling external APIs;
    • Diverse outcomes: the same task may have several reasonable solutions, hard to judge against a standard answer.

    To ensure statistical significance, a platform generally only includes sessions that meet a minimum threshold of turns or duration. Some platforms base their rankings on tens of thousands of real, long-horizon sessions from users worldwide—a volume large enough for scores to stabilize.

    The value of real data lies in its closeness to actual usage, but it also brings privacy and compliance challenges. Task descriptions may contain personal information, internal materials, or access credentials, so platforms must complete data anonymization with users’ informed consent and establish access controls and retention policies.

    Blind Head-to-Head Battles: Users Decide Who’s Better

    The core mechanism of an arena is anonymous battles:

    1. Give the same task to two anonymous models to execute;
    2. The user sees both results at once and chooses “better,” “worse,” or “tie,” without knowing the model identities;
    3. The display order of the two results is randomized to eliminate positional bias (people tend to favor content they see first);
    4. Each battle serves as a data point feeding into a global score.

    The advantage of blind testing is that the evaluation dimensions are determined by users’ actual experience rather than predefined metrics set by the evaluation designers. For outputs that are hard to quantify—such as “whether code runs” or “whether a report is adequate”—users’ holistic judgment often comes closer to real value.

    From Battle Results to the Leaderboard

    Each battle yields only one of three outcomes: win, loss, or tie. The platform must aggregate large numbers of scattered battles into a unified ranking. Two common approaches are:

    • Pairwise comparison scoring: similar to a chess rating system, each model has a score. Winning more, and beating stronger opponents, raises the score more; as a model’s performance changes over time, the score is also dynamically adjusted;
    • Statistical model estimation: treating each battle as an observation, using statistical modeling to estimate each model’s relative strength and its uncertainty.

    Whichever method is used, a ranking must come with uncertainty information. Taking pairwise comparison scoring as an example, the score is not true ability but an estimate with error. The larger the sample, the more reliable the estimate; with a small sample or a small score gap between two models, the difference in rank may not be significant. Therefore, when citing a leaderboard, prioritize confidence intervals, sample size, and statistical time window rather than viewing the rank in isolation.

    Category Leaderboards: Where Ranking Differences Come From

    A mature arena usually maintains multiple leaderboards, separately tallied by openness (e.g., open-weight models vs. closed-source models), capability domain (e.g., work, coding, chat), or task difficulty.

    The same model’s ranking can differ greatly across leaderboards. For example, a model may rank first in the open-weight sub-leaderboard but mid-pack in the overall leaderboard across all participating models; conversely, it may sit near the top of the overall leaderboard yet trail more specialized models in a particular domain. This difference stems mainly from three factors:

    • Different participating sets mean different comparison objects;
    • Different task distributions mean different weights across task types;
    • Newly launched models have fewer samples early on, leading to greater statistical fluctuation.

    Therefore, a leaderboard ranking answers the question of “compared against whom, on what tasks.” Discussing rank without specifying the exact criteria easily leads to one-sided conclusions.

    Interpreting Core Metrics: From “Total Score” to “Capability Profile”

    Net Improvement Rate: Measuring Generational Progress

    The overall ranking reflects a model’s relative position within the current participating set, but it does not show how much a new generation has improved over the previous one. The net improvement rate is the metric designed for this: in head-to-head battles between the two, it tallies the new generation’s net win ratio relative to the old.

    1. def net_improvement(wins, losses):
    2. """Calculate the net improvement rate from head-to-head battles, returning a percentage"""
    3. total = wins + losses
    4. if total == 0:
    5. return None # Insufficient sample; result is not statistically meaningful
    6. return round(100.0 * (wins - losses) / total, 1)
    7. # Example: new generation vs. previous generation, 500 head-to-head battles,
    8. # 150 wins, 100 losses (250 ties)
    9. # Net improvement rate = (150 - 100) / (150 + 100) = +20.0%
    10. print(net_improvement(150, 100))

    Different platforms may handle ties differently. The implementation above excludes ties from the denominator; alternatively, you could count each tie as half a point. When comparing data from different sources, be sure the criteria are consistent.

    The metric value is just an arithmetic result; what matters is how it is interpreted. The following hypothetical data illustrate the point:

    Comparison Head-to-head sample Net improvement rate
    Previous gen vs. one before 1000 +0.9%
    New gen vs. previous gen 2000 +5.1%
    • Absolute magnitude: the new generation’s net improvement rate is 4.2 percentage points higher than the previous generation’s;
    • Relative magnitude: rising from +0.9% to +5.1% is about a 5.7x increase; looking at absolute numbers alone underestimates the progress;
    • Generational decay: compared to the generation before, the previous generation’s improvement was already close to “even,” indicating that under the task distribution at the time, capability gains were diminishing, and the new generation’s value lies in reopening the upward trajectory;
    • Statistical constraints: the net improvement rate only shows that “between old and new, users prefer the new,” not that the model has reached some absolute level. If it loses heavily against stronger models, neither generational improvement nor increased user preference can mask the capability gap.

    Shell Execution Reliability: Are the Agent’s “Hands and Feet” Steady

    Tasks like coding, operations, and data processing rely heavily on executing shell commands. A command error itself is not frightening; what really hurts the experience is whether the agent can identify the cause of a failure and correct it in time. Execution reliability can be measured at three levels:

    • Execution layer: can the command be parsed correctly and finish normally within a time limit;
    • Verification layer: can the execution result be automatically judged as success or failure;
    • Remediation layer: after a failure, can it re-plan and complete the fix.
    Failure type Typical manifestation Impact on the task
    Execution failure command not found, syntax error, insufficient permissions, timeout task halts directly
    Verification failure command returns success, but file not generated, tests fail wrong result propagates downstream
    Remediation failure repeats the same command after an error, or retries blindly steps and time cost rise sharply
    1. def shell_reliability(runs):
    2. """runs: each item contains timeout, missing_cmd, exit_code, check_passed fields"""
    3. timeout = sum(r["timeout"] for r in runs)
    4. missing = sum(r["missing_cmd"] for r in runs)
    5. exec_fail = timeout + missing + sum(r["exit_code"] != 0 for r in runs)
    6. check_fail = sum(r["exit_code"] == 0 and not r["check_passed"] for r in runs)
    7. total = len(runs)
    8. return {
    9. "execution_success_rate": round(1 - exec_fail / total, 3),
    10. "check_pass_rate": round((total - exec_fail - check_fail) / total, 3),
    11. "failure_type_distribution": {"execution_failure": exec_fail, "check_failure": check_fail},
    12. }

    When measuring reliability, it is recommended to record each command’s exit code, error message, and duration, and to define task-level verification scripts that distinguish “genuine fix” from “coincidental pass.” This helps pinpoint a model’s weaknesses in command understanding, error diagnosis, and recovery strategy.

    Steerability: Can It Understand and Execute Corrections

    Long-horizon tasks are rarely “one and done”; they are adjusted as you go. Steerability refers to whether an agent, when it deviates from user intent during a long task, can understand and act on the user’s corrective instructions. An evaluation can be designed as a “deviation injection” controlled experiment:

    1. At a key step in the task, inject a corrective instruction (e.g., “don’t touch the config file, only modify the code,” “switch to the alternative approach”);
    2. Observe whether the agent promptly stops the erroneous action and adjusts subsequent behavior to the new constraint;
    3. After the task ends, give a combined score based on task completion, extra steps consumed, and time cost.

    Steerability includes at least four layers of capability:

    • Understanding: accurately identifying which constraint the instruction changes and which old constraints remain in effect;
    • Timeliness: stopping operations that conflict with the goal as soon as possible, rolling back completed steps when necessary;
    • Completeness: all subsequent actions obey the new constraint, not just the first step or two;
    • Resistance to misdirection: when facing wrong instructions, being able to point out conflicts and offer suggestions rather than blindly complying.

    Steerability needs to be balanced with autonomy: being too compliant causes tasks to change direction frequently under slight intervention, hurting efficiency; being too autonomous makes it hard for users to course-correct. A good agent should flexibly switch between “executing its own plan” and “accepting user adjustments.”

    Metric System Design Suggestions

    Beyond the three core metrics above, a complete evaluation system should also cover: tool-call correctness, step and cost efficiency, constraint retention over long contexts, security and permission compliance, consistency across repeated runs, and more.

    When designing metrics, three principles are recommended:

    • Automatically judgeable: score with objective signals like exit codes, file diffs, and test pass rates wherever possible;
    • Attributable: each metric maps to a clear capability dimension, avoiding a vague “total score” that masks problems;
    • Have a baseline: all metrics should be compared against a baseline or candidate version, not viewed as absolute values alone.

    Engineering Implementation: Building Your Own Agent Evaluation Pipeline

    Reference Architecture

    An arena is essentially a large-scale, long-horizon evaluation pipeline whose core stages include:

    1. Session and task collection: collect anonymized real sessions and filter for long-horizon tasks meeting turn and duration criteria;
    2. Sandboxed execution: allocate an isolated environment for each task, inject the model under test, and drive its tool calls;
    3. Result judging: automatically verify outputs, introducing human or anonymous comparison where necessary;
    4. Metric calculation and ranking update: compute metrics such as net improvement rate and reliability, and roll updates to the leaderboard;
    5. Monitoring and governance: monitor failure rates, cost, and latency anomalies, and handle violations and low-quality sessions.

    Cloud Capability Building Points

    Enterprises can also build a private evaluation environment based on general cloud products from platforms such as Baidu AI Cloud:

    Capability stage General cloud product category Role
    Sandboxed execution container services, function computing isolate runtime environments, scale elastically per task
    Artifact and log retention object storage, log services store dialogues, command output, and execution traces for replay and audit
    Task scheduling message queues, task orchestration services asynchronously dispatch long-running tasks, smoothing load
    Access control identity and access management control access scope for model keys, sandboxes, and data
    Observability monitoring and alerting, distributed tracing detect ranking jumps, rising failure rates, and cost anomalies

    Regardless of which cloud platform is used, sandbox design should focus on four points:

    • Environment consistency: unify base images and dependency versions to avoid “runs locally but errors in the sandbox”;
    • Least privilege: restrict network access and file-system permissions, and require a second confirmation for high-risk operations like deletion and privilege escalation;
    • Cost control: set timeouts and resource limits per task to avoid failed tasks occupying resources long-term;
    • Traceable results: fully record model version, prompt version, execution logs, and scoring rationale.

    How to Rationally View Leaderboard Rankings

    Arena leaderboards provide valuable real-world feedback, but they still require caution about the following biases when used:

    • User population bias: arena users are mostly tech early adopters, and their task preferences may not represent your business scenario;
    • Task distribution drift: over time, new task types change the difficulty structure, reducing comparability between historical data and the current leaderboard;
    • Statistical fluctuation: with limited samples early in a model’s launch, ranks can change noticeably due to a few battles;
    • Overfitting risk: repeatedly tuning against public evaluation tasks may yield high leaderboard scores yet cause performance to fall back in new scenarios;
    • Local advantages amplified: ranking first in a sub-leaderboard does not mean leading equally on overall tasks.

    A more robust approach is a “three-step” process:

    1. Use arena leaderboards to narrow down the candidate set;
    2. Extract typical tasks from your own business to build a private evaluation set that fits your scenario;
    3. Complete comparative testing in a unified sandbox and bring all metrics into continuous monitoring.

    Conclusion

    The value of agent arenas lies not in producing a “single correct” ranking, but in turning agent evaluation from subjective impressions into a quantifiable, reproducible, and sustainable engineering practice. Understanding the logic behind real-task collection, blind head-to-head battles, category leaderboards, and core metrics helps developers avoid being misled by a single rank when selecting models, and provides a methodological reference for enterprises building an autonomous, controllable evaluation loop. Model iteration will continue, and only by establishing an evaluation system that fits your own business can you consistently make reliable decisions in a rapidly changing technical environment.