Evaluating AI Agents: From Manual Voting to Causal Tracking

1. Background: Why Agent Evaluation Is Difficult

As large language models evolve from conversational assistants into autonomous agents, evaluation difficulty increases significantly. This stems from three primary characteristics of agent tasks:

1. Long cycles and complex structures: An agent session is rarely a simple Q&A. It involves planning, tool calling, file I/O, command execution, and multi-round reworks that can last for hours or days. Users care less about whether a specific sentence sounds good and more about whether the task was ultimately accomplished.
2. Objective environmental feedback: An agent’s actions leave traces in the real environment: whether a command executed successfully, whether the code ran, or whether a file was correctly generated. These signals are more rigid than subjective human preferences but harder to collect and attribute.
3. Frequent user intervention: When a model goes off track, users issue correction commands. The frequency of these corrections and the model’s recovery speed inherently reflect its usability.

Traditional evaluation methods expose clear limitations in this context:

• Static benchmarks: Fixed questions are easily gamed. The task distribution often misaligns with real-world usage scenarios, causing a severe disconnect between benchmark scores and actual user experience.
• Pairwise voting (users choosing the better of two responses): Suitable for short-form Q&A preference comparisons. However, agent tasks are too long with too many variables; users are reluctant to compare two complete task processes.
• Manual review: High cost, slow speed, and an inability to cover large-scale real-world traffic.

Consequently, a new evaluation paradigm has emerged: instead of relying on manual voting, directly “observing” real user-agent sessions, automatically extracting evaluation signals from massive datasets, and using statistical methods to estimate the relative performance of different models. This article breaks down its signal taxonomy, core algorithms, and engineering implementation.

2. Signal Taxonomy: Extracting Four Key Signals from Real Sessions

Real-world sessions are natural evaluation data—users “vote with their feet,” and their behavior is the feedback. Common automatically extractable signals fall into four categories:

1. Explicit task confirmation: Users explicitly stating “problem solved” or “this is the right effect.” This is the most direct evidence of task completion. It carries a high weight but occurs relatively infrequently.
2. Praise and complaints: Positive or negative emotional feedback naturally expressed by users during the session. The advantage is broad coverage, but the noise is high. It requires distinguishing between “evaluating the model” and “complaining about the objective world.”
3. Correction commands: When a user says “That’s not what I meant, you should…” or “Do it another way,” it indicates the model deviated from expectations. The frequency, timing, and whether the model quickly converges after a correction form the basis of “steerability” evaluation.
4. Environment-level failure signals: Represented by command-line execution errors: non-zero exit codes from model-generated commands, script runtime errors, and tool call failures. These signals are completely objective, independent of user expression, and serve as the core metric for “operational reliability.”

Combining these signals creates an interwoven positive and negative “experience profile”: confirmations and praises act as positive votes, while complaints, corrections, and execution errors act as negative votes.

To stably extract these signals in production, three challenges must be addressed:

• Session parsing: Real sessions interleave multi-turn dialogues, tool calls, and system messages. Structural segmentation is required first to align “what the user said,” “what the model did,” and “what the environment returned” into distinct turns.
• Signal identification: Simple keyword matching has limited recall. In practice, a hybrid approach of “rule filtering + discriminative model” is typically used: rules capture high-confidence signals, while a classification model handles ambiguous expressions.
• Signal attribution: A failure in a session might originate from the model, user input, or the environment itself (e.g., network timeout, insufficient permissions). Attribution rules must exclude non-model factors to avoid systematically underestimating or overestimating model performance.

3. Core Methodology: Causal Tracking and Net Improvement Rate

Once signals are extracted, the next question is: how do we aggregate them into comparable model scores?

The intuitive approach is to “calculate the average score of each model’s sessions.” However, this introduces obvious selection bias: different models serve different user demographics, task types, and difficulty distributions. If a model happens to be used more frequently for high-difficulty tasks, its raw failure rate will naturally be higher, making direct comparison unfair.

The core concept of “causal tracking” borrows from randomized inference: without constructing artificial controlled experiments, it uses the “average level of all sessions” as a natural baseline to estimate a specific model’s net effect relative to that baseline—the Net Improvement Rate. Its meaning can be understood as follows:

• The average performance of all sessions is set as the zero point.
• A positive net improvement rate for a model means its sessions have relatively more positive signals (confirmations, praises) and relatively fewer negative signals (complaints, corrections, execution errors), performing above average overall.
• A negative rate means it is below average; the absolute magnitude reflects the degree of deviation.

The estimation process is expressed in the following pseudo-code:

# Pseudo-code: Net Improvement Rate estimation (for illustration only)
def net_improvement_rate(session_scores):
    baseline = mean(all_scores)          # Global average as the zero point
    rates = {}
    for model, scores in group_by_model(session_scores).items():
        rates[model] = (mean(scores) - baseline) / abs(baseline)
    return rates

In a rigorous implementation, several statistical steps must be supplemented:

• Confidence interval estimation: Methods like bootstrap resampling provide an interval rather than a single point value for the net improvement rate, preventing erratic rankings for small-sample models. Empirically, ranked models should be backed by thousands or tens of thousands of real sessions.
• Stratification and deconfounding: Stratifying statistics by task type, language, and session length prevents Simpson’s paradox caused by disproportionate task distributions.
• Time windows: The distribution of real users and model versions rolls over time. Rolling window statistics are typically employed, with higher weights assigned to recent samples to reflect the current state.

Compared to manual voting, this methodology offers clear advantages: evaluation occurs entirely within real tasks without organizing dedicated reviews; signal extraction is automated with near-zero marginal cost; and the net improvement rate, expressed as a relative baseline, naturally circumvents most biases caused by uneven task distributions.

4. Metric Design: From a Single Score to a Multi-dimensional Profile

A single composite score easily masks shortcomings. In practice, multiple granular dimensions are derived beneath the composite net improvement rate, such as:

• Shell command execution reliability: The failure rate of model-generated commands and scripts in the environment, directly reflecting the robustness of “hands-on capability.”
• Steerability (error correction capability): The frequency of correction commands and whether the model quickly returns to the correct track after being corrected, reflecting the collaborative experience.
• Task completion confirmation rate: The proportion of sessions where the user explicitly confirms task completion.
• Long-task performance: Stratifying sessions by duration or turn count to observe the capability decay curve of the model on long-cycle tasks.

The value of a multi-dimensional profile lies in supporting “scenario-based selection.” Two models with similar composite scores might differ: one could have highly stable command execution but sluggish error correction, while the other is highly steerable but frequently makes operational errors. The optimal choice differs completely between a team focused on automated DevOps and one focused on interactive collaboration.

5. Engineering Implementation: Building a Similar Evaluation System

If a team wishes to adopt this approach to build internal agent evaluation capabilities, the following practices are recommended.

1. Data collection and compliance first: Session data often contains sensitive user information. Explicit notification and authorization should be obtained before collection. Storage should implement desensitization and minimal retention. Signal extraction should ideally operate on desensitized structured data.
2. Build a signal extraction pipeline: A typical workflow is: session log ingestion -> structural segmentation -> signal identification -> attribution filtering -> aggregate calculation. Illustrated below:

# Pseudo-code: Session signal extraction pipeline (for illustration only)
for session in clean(raw_sessions):
    turns = align_turns(session)             # Align user/model/environment messages
    for turn in turns:
        if is_confirmation(turn):    emit("confirm",   +1)
        elif is_praise(turn):        emit("praise",    +1)
        elif is_complaint(turn):     emit("complaint", -1)
        elif is_correction(turn):    emit("steer",     -1)
    for call in session.tool_calls:
        if call.failed and not caused_by_env(call):
            emit("cmd_error", -1)

1. Leverage general cloud infrastructure: Capabilities like log ingestion, session storage, task scheduling, and anomaly alerting usually do not need to be built from scratch. Mainstream cloud platforms like Baidu AI Cloud offer general-purpose products such as logging services, object storage, message queues, and monitoring/alerting, which can be directly combined to build the pipeline above.
2. Prevent “exam-oriented” overfitting: If evaluation criteria are fully public, model providers might optimize for the evaluator rather than real capability. Viable mitigation strategies include: keeping the specific rules for signal identification private, regularly iterating the discriminative model, and performing multi-signal cross-validation (e.g., jointly evaluating a correction command alongside the subsequent session trajectory).
3. Establish version tracking mechanisms: Given frequent model iterations, supporting version-sliced statistics makes the “change in net improvement rate of the new version versus the old” a direct basis for iteration decisions. This is the inherent advantage of the net improvement rate method: it answers not “how good is the model,” but “how much better is this version compared to the average and the previous generation.”

6. Conclusion

The essence of agent evaluation is transforming subjective experiences scattered across countless real sessions into objective, statistically comparable, and trackable evidence. Using real sessions as the data source, automated signal extraction as the means, and a causal tracking-style relative baseline as the metric, this methodology provides an actionable path toward “evaluation grounded in real-world usage.” For developers, understanding its signal taxonomy and statistical logic not only aids in correctly interpreting various agent leaderboards but can also be directly transferred to internal model selection and iterative evaluation, ensuring every version upgrade is evidence-based.