AI Agent Evaluation: A New Paradigm Based on Real Conversation Signals and Causal Inference

1. Why Traditional Evaluation Fails in the Agent Era

Once large language models evolved from “chat assistants” into “agents,” evaluation became dramatically harder. The two methods the industry has long relied on both show clear weaknesses.

Limitations of static benchmarks

A static benchmark consists of fixed questions that a model answers in a controlled environment, scored against reference answers. This works reasonably well for single-turn Q&A, but for agents it has three fundamental problems:

• Task-shape mismatch: Real agent tasks are long-horizon and multi-turn, involving repeated tool calls and self-correction. Fixed questions can hardly capture this complexity.
• Data contamination risk: Public benchmark items may leak into training corpora and inflate scores—the model ends up memorizing answers instead of solving problems.
• Insufficient metric granularity: A single aggregate score cannot answer the questions developers care about most, such as “Is this model dependable when it executes shell commands?” or “Can it get back on track quickly after a user correction?”

Limitations of blind human voting

The other common approach is to have users blindly vote between the responses of two models. For chat scenarios, this effectively reflects human preference; in agent scenarios it breaks down. An agent session can span dozens of turns with numerous tool calls and file operations, and an outside rater has neither the patience nor the ability to determine within a short time which side actually got the job done. Preference voting measures “looks good”; agents need to be measured on “got it done.”

Both points lead to the same conclusion: agent evaluation needs new data sources and new statistical methods.

2. The New Approach: Passively Mining Evaluation Signals from Real Conversations

In recent years, an “observational evaluation” approach has emerged: instead of setting exams for the model, teams mine the conversation logs of real users interacting with agents and extract quality signals from massive volumes of naturally occurring interactions. The core assumption is that how users behave on real tasks is itself the most honest feedback.

Four key signal types

From conversation text and execution trajectories, the following signals can typically be extracted:

• Explicit confirmation: The user explicitly says the task is done—“it worked,” “it runs”—a strong positive signal.
• Praise and complaints: Spontaneous user evaluations of the process or the result, with sentiment intensity well above that of neutral interactions.
• Correction requests: The user asks the model to switch approaches, redo the work, or change parameters—evidence that the model’s previous direction was wrong.
• Command execution errors: Non-zero exit codes or error stack traces from tool calls—objective, structured evidence of failure.

Compared with manual annotation, these signals are cheap to collect, available at scale, and rooted in genuine user intent, which inherently avoids the exam-distortion problem.

Engineering signal extraction

A first version can be built with a “rules engine + sequence parsing” design. Sample code:

import re
from collections import defaultdict
POSITIVE_PATTERNS = [r"it worked", r"runs successfully", r"no problems now"]
NEGATIVE_PATTERNS = [r"not right", r"wrong again", r"switch .* approach", r"start over"]
def extract_signals(session):
    """session is a time-ordered sequence of interactions"""
    signals = defaultdict(int)
    for turn in session.turns:
        if turn.role == "user":
            if any(re.search(p, turn.text) for p in POSITIVE_PATTERNS):
                signals["explicit_confirm"] += 1
            if any(re.search(p, turn.text) for p in NEGATIVE_PATTERNS):
                signals["correction_or_complaint"] += 1
        elif turn.role == "tool":
            if turn.exit_code != 0:
                signals["shell_error"] += 1
    return signals

In production, pure rules have limited recall. Teams typically layer a lightweight text classifier on top to recognize semantic-level praise, complaints, and corrections, with dedicated handling for noisy samples such as sarcasm and polite pleasantries.

3. The Attribution Problem: Turning Correlation into Causation with Causal Inference

Collecting signals is only the first step; attribution is harder. The outcome of a session is jointly determined by many factors: model capability, scaffolding and tool configuration, task difficulty, user skill—even the service’s condition that day. If you simply observe that “sessions running model A had a higher success rate” and conclude that A is stronger, you commit a classic confounding error: the choice of model may itself be strongly correlated with task type.

Randomized inference

The solution shares its DNA with online experimentation (A/B testing): randomize session assignment at the traffic-allocation layer so that the tasks handled by different models follow approximately the same difficulty distribution, then perform statistical inference under this randomization framework. This can be called “causal-tracking” randomized inference. Its output is typically a net improvement rate: an estimate of a model’s net effect relative to the average level of all models being evaluated.

An illustrative example: suppose a new-generation model posts a composite net improvement rate of +5.1% while its predecessor sits at +0.9%. The 4.2-percentage-point difference can then be read as “the genuine capability gain delivered by the version iteration, after excluding confounders such as task difficulty.” Because it is expressed relative to a baseline, this metric resists task-distribution drift far better than absolute scores.

Statistical power and confidence intervals

A net improvement rate is an estimate, so confidence intervals must be reported alongside it. Agent sessions have huge variance—the same task may take one user two turns and another thirty—so conclusions require adequate sample size. As a rule of thumb, credible evaluation results are typically built on tens of thousands of real long-horizon sessions, with stratified estimation across categories such as work, coding, and everyday chat, so that signals from large categories do not drown out differences in small ones.

def net_improvement(sessions, target_model, n_boot=2000):
    treated = [s.outcome for s in sessions if s.model == target_model]
    control = [s.outcome for s in sessions if s.model != target_model]
    delta = mean(treated) - mean(control)
    ci = bootstrap_ci(treated, control, n_boot)
    return delta, ci  # return both the point estimate and the 95% confidence interval

4. The Metric System: Fine-Grained Capability Profiles Beneath the Aggregate Score

A single net improvement rate is good for rankings, but for developers choosing a model, fine-grained metrics are more valuable. Common profile dimensions include:

• Command execution reliability: The first-pass rate of shell commands initiated by the model and its self-recovery rate after errors—directly reflecting how steady its “hands” are in engineering operations.
• Correctability: After a user issues a correction, does the model return to the right direction within a few turns, and does it avoid repeating mistakes? This reflects whether it “listens.”
• Category performance: The same model’s net improvement rate may differ significantly across coding, office productivity, and everyday conversation. Category-level splits prevent one strong area from masking weaknesses elsewhere.
• Iteration comparison: The difference in net improvement rate between successive versions of the same model family is a direct yardstick for whether an R&D iteration delivers real gains.

5. Engineering It: Building an Internal Evaluation Pipeline

Enterprises that want to apply this methodology to internal agent selection or in-house model iteration can organize the work as the following pipeline:

1. Data collection and de-identification: Record complete session trajectories at the gateway layer (user inputs, model outputs, tool calls, exit codes), and de-identify and anonymize personal information to meet privacy-compliance requirements.
2. Session parsing and signal labeling: Use rules plus a classifier to extract the four signal types and persist them as a structured event table.
3. Randomization and traffic orchestration: Randomly split traffic among candidate models at the canary layer so that all versions face the same task distribution.
4. Statistical estimation and dashboards: Periodically compute net improvement rates and confidence intervals for each model and version, visualized by category and metric dimension.
5. Continuous monitoring: Track task-distribution drift and signal noise rates so that historical conclusions do not silently expire.

Every stage can be assembled from general-purpose cloud capabilities: a log service to ingest conversation streams, object storage to archive raw trajectories, a container platform to run batch processing and statistical jobs, and monitoring and alerting to track metric fluctuations. Generic cloud product categories—such as the log services and object storage offered by Baidu AI Cloud—are sufficient to assemble a minimum viable evaluation pipeline without introducing custom components.

6. Common Pitfalls and Best Practices

• Selection bias: A model that happens to attract highly skilled users will naturally show higher session success rates. Randomized traffic splitting is the antidote; where randomization is impossible, apply stratified weighting corrections.
• Signal noise: Polite thank-yous and sarcastic compliments pollute labels. Run regular manual spot checks and recalibrate classifier thresholds.
• Incentive distortion: Once a team starts treating leaderboard rank as the goal, over-optimization against the evaluation distribution tends to follow. The right posture is to treat the leaderboard as a thermometer, not a KPI.
• Leaderboard shelf life: Model capability and task distributions are both evolving fast. Evaluation conclusions should carry a time window and be re-estimated periodically.

Conclusion

Agent evaluation is shifting from “setting exams” to “observing behavior,” and from “human preference voting” to “causal estimation of net effects.” For developers, the value of understanding this methodology lies not in chasing a particular ranking but in acquiring a transferable capability: using real interaction data and rigorous statistical inference to answer the question, “Does this model actually work in my scenario?” When evaluation turns from marketing talk into engineering practice, the adoption of agent technology can truly accelerate.