Evaluating AI Agents: From “Pairwise Voting” to “Signal-Driven” Methodology

Evaluating the capabilities of AI Agents has become a technical proposition distinct from general conversational model evaluation. Traditional model leaderboards mostly rely on users casting pairwise votes on two responses. While effective for short Q&A scenarios, this approach quickly fails in long-horizon, multi-step agent tasks that require tool invocation. Users struggle to identify the root cause of a failure—whether it’s a planning error, incorrect tool parameters, or an unstable environment. Furthermore, the cost of acquiring vote samples scales linearly with conversation length.

Consequently, a new evaluation paradigm is emerging: instead of asking users for explicit scores, we collect real-world agent sessions, automatically extract behavioral signals, and use randomized inference methods to estimate the model’s net improvement rate. This article explores this methodology, discussing its signal system, statistical inference mechanisms, the performance dimensions of open-weight models, and key engineering trade-offs during implementation.

1. Why Agent Evaluation Cannot Follow the Conversational Paradigm

There are three structural differences between agent tasks and single-turn conversations that necessitate a change in evaluation methodology.

First, success determination is sparse and post-hoc. An agent session might involve dozens of tool calls, and whether the goal was achieved is often only apparent at the very end. Asking users to recall and score the process at the finish leads to severe information decay; interrupting them at every step to ask for feedback breaks task continuity.

Second, failure attribution is multi-layered. An agent failure might originate at the planning layer (incorrect step sequence), execution layer (command syntax or path errors), environment layer (missing dependencies or insufficient permissions), or interaction layer (failing to correct a misunderstanding of user intent). A single scalar score cannot capture this multi-dimensional information.

Third, user behavior itself carries strong signals. In real-world usage, users naturally express confirmation, praise, complaints, and corrective instructions. They also retry commands repeatedly after execution errors. These behaviors serve as natural annotations for evaluation, requiring no additional collection costs.

Based on these points, the design goal of an evaluation system should shift from “collecting subjective scores” to “parsing objective behaviors.”

2. Signal System: What to Automatically Extract from Sessions

Signals extracted from real sessions can be broadly categorized into four types, each corresponding to a different capability dimension.

1. Task Confirmation Signals

When a task is completed, users explicitly state phrases like “that’s fine,” “exactly,” or “solved.” These are positive terminal signals. It is crucial to distinguish between explicit confirmation and implicit acceptance: the former is actively expressed by the user and carries high weight; the latter is observed when the user stops making modification requests and moves directly to the next task, carrying lower weight but offering a larger sample size. In engineering practice, different weights are assigned to avoid over-interpreting implicit acceptance.

2. Praise and Complaint Signals

Praise signals (e.g., positive feedback on results) reflect output quality, while complaint signals (e.g., “this isn’t what I wanted” or “you got it wrong again”) reflect intent comprehension deviations. Complaint signals are more valuable than praise signals because they pinpoint specific failure stages, whereas praise typically only provides a general impression.

3. Corrective Instruction Signals

Corrective commands inserted by users during an agent’s execution are the core metric for evaluating Steerability. A highly steerable model should converge on the correct direction after receiving a correction, rather than repeatedly reverting to the wrong path. The “number of turns required for the first success after a correction” can be used as a quantitative metric:

def steerability_score(turns, correction_index):
    # turns: boolean sequence indicating whether each turn was a success
    # correction_index: the turn where the user issued a corrective command
    after = turns[correction_index + 1:]
    for i, ok in enumerate(after):
        if ok:
            return 1.0 / (i + 1)   # Higher score for earlier recovery
    return 0.0                  # Never recovered after correction

4. Command Execution Error Signals

In agent scenarios equipped with shell execution capabilities, errors such as non-zero exit codes, non-existent paths, and missing dependencies are hard signals that can be programmatically detected. These signals are highly objective and low-noise, making them ideal for measuring Bash Reliability.

Aggregating these signals yields a multi-dimensional signal vector. The key to evaluation is not compressing this vector into a single score, but preserving the dimensional information to make rankings explainable.

3. Causal Tracing: Estimating Net Improvement from Observational Data

Signals alone are insufficient because their distribution is confounded by task difficulty, user demographics, and session length. For instance, if a model handles a higher proportion of simple tasks, it naturally receives more confirmation signals, but that doesn’t mean the model is inherently stronger. Therefore, a de-confounding inference step is required. The “causal tracing” approach is essentially a combination of randomized inference + stratified weighting, broken down into four steps:

Step 1: Session Stratification. Stratify sessions by task type (work, coding, chat), session length intervals, and tool invocation counts to ensure that sessions within the same stratum are comparable in difficulty.

Step 2: Model Assignment Randomization. Within the same stratum, the matching between users and models is approximately random (typically guaranteed by routing strategies in real platforms), making intra-stratum comparisons close to a randomized controlled trial. If routing is not perfectly random, propensity score weighting must be introduced for correction.

Step 3: Estimate Intra-Stratum Effects. For each stratum, calculate the difference in the signal vector between the target model and the stratum’s average level to obtain the intra-stratum net improvement. Use bootstrap resampling to estimate confidence intervals, preventing small-sample strata from generating false advantages.

Step 4: Cross-Stratum Weighted Aggregation. Weight by the proportion of sessions in each stratum to obtain the overall net improvement rate. Dimensional metrics (reliability, steerability, etc.) are aggregated separately and not blended with the overall score.

import numpy as np
def net_improvement(signals_by_stratum, model_id, n_boot=1000):
    """signals_by_stratum: {stratum: {model: [signal...]}}"""
    strata = list(signals_by_stratum)
    weights = np.array([len(signals_by_stratum[s][model_id]) for s in strata],
                       dtype=float)
    weights /= weights.sum()
    point = 0.0
    boot = np.zeros(n_boot)
    for w, s in zip(weights, strata):
        vals = np.array(signals_by_stratum[s][model_id], dtype=float)
        mean_all = np.mean([v for m in signals_by_stratum[s]
                            for v in signals_by_stratum[s][m]])
        point += w * (vals.mean() - mean_all)
        idx = np.random.randint(0, len(vals), (n_boot, len(vals)))
        boot += w * (vals[idx].mean(axis=1) - mean_all)
    lo, hi = np.percentile(boot, [2.5, 97.5])
    return point, lo, hi

This code embodies the core of the methodology: the score is not an absolute capability value, but a net shift relative to the average level, and it must be presented with confidence intervals. Rankings without interval information have limited reference value in engineering decisions.

4. Interpreting Performance Dimensions of Open-Weight Models

In an evaluation based on over 17,000 real long-horizon sessions from global users, an open-weight model achieved first place in the open-weight category and 13th place overall among all evaluated models. It ranked first among open-weight models across work, coding, and chat categories, with a net improvement rate of approximately +5.1%, an increase of about 4.2 percentage points over its previous generation. While these numbers are product conclusions, the underlying dimensional structure is more valuable to developers.

Significance of Overall Net Improvement Tiers. The net improvement rate measures the shift relative to the average level, where a positive value indicates above-average performance. An increase from +0.9% to +5.1% suggests that improvements are concentrated in high-frequency, high-weight signals rather than a few long-tail scenarios. The more engineering-valuable approach is to decompose this number back into stratified results to observe which task types and session length intervals drove the improvement.

Execution Reliability. In command execution scenarios, the reliability metric directly corresponds to the “first-time success” ratio. This dimension is particularly critical for agents with tool-calling capabilities, as it is influenced by both generation quality and self-correction ability. Improvements typically come from two sources: reducing illegal parameters during the generation phase, and adjusting retries based on error outputs after execution failures.

Steerability. This dimension measures how quickly a model converges after receiving a user correction. A typical manifestation of poor steerability is “apologizing but repeating the mistake”—the model acknowledges the error but follows the original path in the next turn. Improving steerability often requires reinforcing immediate compliance with corrective instructions during training, rather than merely improving single-turn response quality.

Using Categorical Rankings. Work, coding, and chat tasks have distinctly different capability requirements. Categorical rankings guide model selection better than overall rankings. In practice, it is recommended to weight the three scores based on your business’s task distribution rather than directly adopting the overall rank.

5. Engineering Implementation: Building a Reproducible Pipeline

When implementing this methodology as an internal evaluation system, focus on the following engineering essentials:

Session Collection and Desensitization. Real sessions contain user privacy and sensitive business information. The collection pipeline must perform field-level desensitization and store aggregated signal extraction results rather than raw text. It is recommended to extract signals directly at the collection endpoint, uploading only signal vectors and anonymized session metadata.

Rule-Based and Model-Based Signal Extraction. Semantic signals like confirmation, praise, and complaints can be extracted using small model classifiers, while hard signals like command errors are matched using rules. These two types of signals should be stored separately to facilitate subsequent ablation analysis: when rankings fluctuate abnormally, you can first pinpoint whether it’s a semantic signal drift or a hard signal change.

Stability of Stratification Variables. Stratification relies on metadata like task type and session length, and the definitions of these fields must be versioned. Once the stratification criteria change, historical scores cannot be directly compared, and the criteria version must be noted in the evaluation report.

Presentation of Confidence Intervals. Differences between closely ranked models may fall within confidence intervals. Engineering dashboards should display intervals by default and explicitly mark models as “statistically indistinguishable” when differences are insignificant, preventing selection decisions from being misled by noise.

Continuous Regression Evaluation. Agent models iterate rapidly, and evaluation should run as part of continuous integration: after each model version update, rerun the inference pipeline on a fixed stratified sample to compare trends in net improvement rates and dimensional metrics.

6. Boundaries of the Methodology

It is necessary to objectively point out the limitations of this method. First, the signal distribution of real sessions is influenced by the product’s form factor, meaning scores cannot be directly compared horizontally across different products. Second, semantic signal extraction depends on classifier quality, and the classifier’s inherent bias will propagate to the rankings. Third, randomized inference assumes that routing strategies are approximately random; if a product employs heavily personalized routing, additional propensity score correction is required.

Therefore, such leaderboards are best used as a reference for selection and a guide for improvement directions, rather than the sole decision-making basis. A more robust practice is to use public evaluations to shortlist candidates, and then build a small-scale internal evaluation using your own business data for cross-validation.

Conclusion

The core challenge of agent evaluation is not “how to score,” but “how to obtain trustworthy signals without disturbing the user.” The methodology outlined here—using real sessions as the data source, multi-dimensional behavioral signals as annotations, and randomized inference for de-confounding—provides an engineering-ready answer. For developers, understanding this signal system and inference process holds greater long-term value than memorizing any specific ranking, because models will iterate, but the methodology can be reused.