A New Paradigm for AI Agent Evaluation: From Human Voting to Real-World Signal Mining

As AI agents transition from demos to production environments, evaluation mechanisms are becoming a critical bottleneck for iteration efficiency. The traditional “double-blind comparison + human voting” approach worked reasonably well in the chatbot era. However, for long-horizon agent tasks that routinely involve dozens of steps and multi-turn tool calls, voting methods expose significant limitations: sparse sample sizes, high subjective bias, and an inability to pinpoint exactly where failures occur. This article explores an evaluation leaderboard mechanism designed for real-world agent tasks, analyzing its signal taxonomy, statistical methods, and engineering implications, while discussing how to implement this paradigm as reusable evaluation infrastructure.

1. Why Traditional Voting-Based Evaluation Fails for Agents

In single-turn Q&A scenarios, asking users to choose between two responses is a low-cost preference signal that is easy to aggregate. However, the inherent characteristics of agent tasks make this method difficult to apply:

• Long Task Horizons: A single agent session might involve dozens of tool calls, file read/write operations, and command executions. The quality of the final outcome often depends on a decision made in an intermediate step rather than the final reply. Users find it extremely difficult to formulate an overall preference judgment for an entire trajectory.
• Binary Success/Failure Nature: Agent tasks are typically judged by whether a goal is “completed” rather than “which response is better.” Voting artificially creates relative preferences, thereby losing the absolute context of success or failure.
• Inherent Signal Availability: In real-world sessions, users naturally confirm results, correct directions, and complain about errors. These behaviors are high-quality evaluation signals themselves, requiring no additional manual annotation.
• High Sampling Costs: Human voting requires paired presentations and massive annotation efforts. Real-world sessions, on the other hand, are generated “for free” at a scale of tens of thousands, making them far more suitable for statistical inference.

Therefore, the evolution direction for evaluation is: replace human preferences with behavioral signals from real conversations, and replace simple vote counting with statistical inference.

2. Evaluation Targets and Signal Taxonomy Design

This category of leaderboard focuses on performance in real-world agent tasks. Its core data source consists of long-horizon agent sessions generated by global users. Using public data from a specific open-weight model as an example, the ranking is based on over 17,500 real long-horizon agent task sessions, covering three major scenarios: work, coding, and chat. This scale means evaluation no longer relies on small-sample manual assessments but is built on real-world distributions.

Signal extraction is the foundation of this methodology. The system automatically extracts multiple key signals from session trajectories, which can be categorized into four dimensions:

1. Explicit Confirmation Signals: The user explicitly states that the task is complete or the result is correct, such as through affirmative replies or directly adopting the output. This is the strongest positive signal.
2. Sentiment and Evaluation Signals: The user expresses praise or complaints. Praise typically corresponds to exceeding expectations, while complaints usually point to comprehension deviations or execution failures.
3. Correction Instruction Signals: The user requests a change of direction, asks to redo a step, or corrects parameters. These signals reflect the model’s “steerability”—whether it can be effectively guided by the user.
4. Execution Error Signals: For example, shell command execution failures or abnormal tool call returns. These signals are objective, automatically detectable hard metrics that directly reflect execution reliability.

The value of these signals lies in the fact that they are not “evaluations” but “behavioral traces.” The advantages of behavioral traces are that they are automatable, scalable, and highly consistent with real-world usage scenarios.

3. Causal Tracking: How Randomized Inference Estimates Net Improvement Rate

Signals alone are not enough; the key is how to aggregate them into comparable model scores. This type of leaderboard employs a randomized inference method called “causal tracking.” Its core philosophy is: under the premise of controlling for confounding factors, estimate the net improvement rate of each model relative to the average level.

This can be understood as a randomized controlled inference pipeline:

• Randomized Assignment: In real traffic, sessions are routed to different models randomly. This mitigates the bias caused by “user self-selection”—for instance, users might only use a specific model for simple tasks, artificially inflating its score.
• Signal Normalization: Different tasks have different signal baselines. For example, the error rate in coding tasks is naturally higher than in chat tasks. The method normalizes each signal against the average level of all models to obtain a “relative net improvement.”
• Net Improvement Rate Aggregation: Multiple types of signals are weighted and aggregated into a single net improvement rate metric. A positive value indicates performance above the average level, while a negative value indicates performance below it.

The statistical logic can be roughly expressed with the following pseudocode:

# Pseudocode: Simplified inference process for net improvement rate
for session in randomized_sessions:
    model = session.assigned_model          # Determined by random assignment
    signals = extract_signals(session)      # Confirmation/Praise/Complaint/Correction/Error
    normalized = normalize(signals, baseline=global_mean)
    yield model, aggregate(normalized)
net_lift = bootstrap_inference(model_scores)  # Randomized inference + confidence intervals

The key advantage of this design is that it estimates not “absolute capability” but the “net effect relative to the average level.” This ensures comparability in environments where multiple models coexist and allows for the generation of statistical uncertainty through randomization and resampling.

4. Metric Interpretation: Net Improvement Rate and Granular Capabilities

In the public data, a specific open-weight model achieved a comprehensive net improvement rate of +5.1%. Compared to the +0.9% of the previous generation in the same series, this represents a 4.2 percentage point increase. This magnitude of improvement is statistically significant, indicating that inter-generational progress is real and measurable, rather than just noise.

Beyond the comprehensive metric, granular metrics better reveal the capability structure:

• Shell Command Execution Reliability (Bash Reliability): Measures the success rate and stability of the agent when executing command-line operations. This is the “foundation” of long-horizon tasks; a single command failure can interrupt the entire trajectory.
• Directional Adjustment and Correction Capability (Steerability): Measures whether the model can effectively change its behavior upon receiving user correction instructions. High steerability means users can pull a task back on track in fewer turns, directly reducing interaction costs.

In the public results of this leaderboard, the aforementioned model ranked first comprehensively among open-weight models and 13th among all participating models. It topped the open-weight model category across work, coding, and chat scenarios, and also ranked at the top among open-weight models in granular metrics like Bash Reliability and Steerability. The key takeaway for interpreting these results is: comprehensive rankings reflect the overall net effect, while granular rankings reveal capability strengths and weaknesses. Combining both is essential for guiding model selection.

5. Engineering Implementation: Building In-House Evaluation Capabilities

For teams looking to replicate this paradigm internally, the following implementation path can be used as a reference:

1. Telemetry Design: Record turns, tool calls, execution results, and user feedback events throughout the agent session chain to ensure signals are traceable. The focus should be on retaining “raw behaviors” rather than just “final scores.”
2. Signal Extraction Pipeline: Regularize the extraction of events like confirmations, corrections, and errors to form a structured signal table. Lightweight classification models can supplement the extraction of sentiment-based signals.
3. Randomized Routing: Randomly distribute traffic to models in a gray-release environment to ensure sample comparability and prevent traffic skew from contaminating conclusions.
4. Statistical Inference Layer: Adopt normalization + resampling inference to output the net improvement rate and confidence intervals, rather than a simple ranking.
5. Granular Metrics Dashboard: Break down data by task type and capability dimension to locate concentrated failure points, forming a closed loop of “evaluation -> attribution -> iteration.”

It must be emphasized that the value of an evaluation infrastructure lies not in “leaderboard rankings,” but in transforming real-world usage behaviors into actionable engineering signals. Only when signals are explainable, inference is reproducible, and metrics are attributable can evaluation truly drive the continuous improvement of agent capabilities.