Evaluating AI Agents: From Preference Voting to Causal Tracking

Background: Why Preference Voting Is No Longer Enough

In the era of conversational models, “let users vote on which response is better” was the dominant evaluation method: simple to implement, easy to collect samples, and a direct reflection of subjective preference. But as models evolve from “chat assistants” to “agents”—capable of planning tasks, calling tools, executing commands, reading and writing files, and working continuously over dozens of turns—this evaluation approach exposes critical weaknesses:

1. Misaligned evaluation target. Voting measures “how good does this response look,” while agent users actually care about “was the task ultimately completed.” A conversation with a rough middle phase that still delivers the correct final result, and a smooth-sounding conversation that fails at a key step, may receive similar scores under a voting system.
2. Longer task cycles. Real agent tasks often span multiple turns, lasting hours or even days. Regular users rarely bother to rate a long session explicitly, so voting samples naturally skew toward short dialogues.
3. Low signal density. A single vote carries just one bit of information—positive or negative. But in a real session, user confirmations, complaints, correction instructions, and environment error messages all contain quality evidence far richer than a single click.

As a result, a new class of evaluation methods has emerged in the industry: instead of relying on explicit votes, these methods deeply mine real user-agent conversations, automatically extract quality signals from the session stream, and use statistical inference to aggregate massive numbers of signals into trustworthy rankings. This article breaks down the full pipeline of such methods and discusses how enterprises can adopt and implement them.

Data Foundation: Collecting and Governing Real Sessions

The data source for this evaluation approach is not a lab-constructed test set, but real user-agent sessions generated during actual product usage. Compared to human-annotated benchmarks, real sessions offer three significant advantages:

• Realistic task distribution: They cover real scenarios like work, programming, chat, and data analysis, mitigating the benchmark leakage problem of “teaching to the test.”
• Naturally occurring feedback: User confirmations and dissatisfaction arise organically during task execution, requiring no extra incentives.
• Long-horizon observability: Tens of thousands of real long-running sessions provide sufficient sample sizes for statistical inference.

Of course, real sessions also come with governance costs. At minimum, three things must be handled in practice:

1. Privacy anonymization: Pattern-based detection and replacement of personal information, secrets, and internal network addresses in sessions.
2. Task classification: Use lightweight classifiers to categorize sessions into work, programming, chat, etc., so metrics can be computed per domain.
3. Quality filtering: Remove ultra-short sessions, bot traffic, and clearly anomalous samples to avoid contaminating statistical results.

Core Stage: Automatically Extracting Key Signals from Session Streams

The “quality evidence” in agent sessions is mostly implicit, requiring an extraction pipeline that combines rules with models. The industry generally focuses on four types of signals:

1. Explicit confirmations. The user gives a clear completion confirmation at the end of the task, e.g., “OK, go ahead and deploy it like this.” This is the strongest positive signal and should carry the highest weight.
2. Praise and complaints. Emotional expressions during the process, such as “that step was done beautifully” or “you changed it wrong again.” Weaker than explicit confirmations, but abundant in quantity and broad in coverage.
3. Correction instructions. The user asks the model to change direction or redo work, e.g., “No, revert to the previous version” or “You misunderstood—I meant the other API.” Each correction implies a deviation in prior output, making it a high-quality negative signal.
4. Environment execution errors. Agents rely heavily on tools and command execution. Non-zero exit codes from shell commands, error stack traces, and repeated retries are objective quality evidence that is difficult to fake.

An illustrative signal extraction result is shown below:

{
  "session_id": "sess_0601_001",
  "task_type": "coding",
  "turns": 23,
  "signals": [
    {"type": "confirmation", "quote": "OK, deploy it with this plan", "polarity": "positive"},
    {"type": "correction",   "quote": "No, revert the dependencies to the previous version", "polarity": "negative"},
    {"type": "shell_error",  "exit_code": 1, "stderr": "module not found", "retried": true}
  ]
}

On the engineering side, the extraction pipeline is typically layered: first, regular expressions and keyword rules capture high-confidence signals; then, small-parameter models perform semantic judgment (e.g., distinguishing “correction” from a casual follow-up question); finally, signals are aggregated with confidence weighting to avoid amplifying errors from any single signal.

Statistical Methods: Estimating Net Improvement with Causal Tracking

Once signals are extracted, how do we aggregate sessions across different models and task difficulties into a fair ranking? Directly comparing “task success rates” is confounded by task difficulty distribution—if one model happens to receive more easy tasks, its success rate will appear inflated. New-generation evaluation methods generally introduce randomized design:

• Random assignment: User sessions are assigned across models in an approximately random fashion, effectively creating a large-scale online experiment.
• Causal tracking: Under randomization, the difference between “a given model’s performance” and “the overall average” can be approximated as the causal effect attributable to that model—i.e., its net improvement rate.
• Relative baseline: The net improvement rate measures a model’s position relative to the average, not the absolute success rate, which naturally cancels out fluctuations in task difficulty.

For example (illustrative numbers): one model generation shows a net improvement rate of +0.9%, close to the average; the next generation rises to +5.1%, a generational gain of 4.2 percentage points—indicating that the new version produces observable progress under the same real task distribution. Combined with confidence intervals and significance tests, this also reveals whether the improvement is a real effect or sampling noise. For granular metrics such as shell command execution reliability and steerability (the ability to adjust direction after user corrections), the same methodology can be applied separately to form a multi-dimensional profile.

Illustrative computation logic:

def net_improvement(model_sessions, all_sessions):
    # Under random assignment, the difference between a model group and the overall mean approximates the causal effect
    y_model = weighted_success_rate(model_sessions)
    y_base  = weighted_success_rate(all_sessions)
    return y_model - y_base   # e.g., +0.051 means a net improvement rate of +5.1%

Explainability: Granular Metrics Make Rankings Interpretable

A single aggregate score is insufficient to guide model improvement. Mature evaluation systems break down metrics along two dimensions:

• Capability dimension: For example, shell command execution reliability (measuring one-pass success rate, and self-repair rate after errors) and steerability (measuring how quickly a model gets back on track after user corrections).
• Scenario dimension: Rankings are computed separately under task categories like work, programming, and chat, since the same model may perform very differently across scenarios.

This granular framework also provides an important observation window: whether the gap between open-weight and closed-source models is narrowing in real-world tasks. Recent evaluation data has repeatedly shown that some open-weight models have entered the top tier of real agent task leaderboards, even ranking first among open models in certain scenario categories. This indicates that the old impression of “open-weight equals capability lag” is being corrected by real data. For enterprises, private deployment of open-weight models is now practically feasible from a capability standpoint.

Enterprise Implementation: Building Your Own Agent Evaluation System

Enterprises don’t need to replicate public leaderboards, but they can reuse the methodology and build an internal evaluation system in four steps:

1. Data pipeline: Build an automated “session collection—anonymization—classification—signal extraction” pipeline using common cloud capabilities such as log services, object storage, workflow orchestration, and monitoring alerts.
2. Signal dictionary: Define high-confidence signals relevant to your own business—for example, whether a support ticket is closed, whether code passes CI checks, or whether a user escalates to human takeover mid-session.
3. Statistical methodology: Adopt the relative-baseline net improvement rate approach to avoid misjudgment caused by varying task difficulty across business lines. Any significant conclusion must be accompanied by confidence intervals.
4. Continuous regression: Treat evaluation as ongoing monitoring rather than a one-time exam. Every model or prompt iteration should trigger a signal comparison run, preventing “fixing one area while degrading three others.”

Limitations and a Sober Perspective

Real-session evaluation is not perfect: the user base and task distribution carry inherent biases, so conclusions may not transfer to other scenarios; emotional signals contain noise, and “sarcastic praise” may be misclassified; causal tracking relies on the randomization assumption, and once traffic allocation is intervened by business rules, the estimates become distorted. Therefore, leaderboards should serve as a reference system rather than the sole source of truth. Critical decisions still require cross-validation with human evaluation and business metrics.

Conclusion

From “casting a vote” to “reading sessions,” from “win-rate rankings” to “net improvement rates,” agent evaluation is shifting from subjective preference toward statistical inference grounded in real task evidence. Understanding the signal design and causal thinking behind these methods not only helps in correctly interpreting various leaderboards, but also enables teams to internalize evaluation as an engineering capability—so that every model iteration can be validated against real task data.