Agent Evaluation’s New Paradigm: Real Conversations and Causal Tracking

1. Why Agent Evaluation Is Getting Harder

Large language models are shifting from single-turn Q&A assistants to agents that plan tasks, call tools, and execute commands. This shift puts traditional evaluation methods under pressure on three fronts:

1. Longer task chains make point-by-point scoring unreliable. Traditional evaluation measures the quality of a single answer. Agent tasks, by contrast, often involve understanding requirements, breaking down steps, calling tools, and correcting course based on feedback. A mistake in one intermediate step does not necessarily mean the final task fails, and getting every intermediate step right does not guarantee user satisfaction. The final outcome has to be observed across the full conversation.

2. Static benchmarks have inherent flaws. Fixed-question benchmarks are vulnerable to data contamination, since a model under test may have seen the questions during training. At the same time, the question distribution may drift away from real user needs, so scoring well on a benchmark does not equal doing well on real work.

3. Manual blind voting does not cover agent scenarios. Two-option voting works well for comparing two short responses, but agent tasks can last hours or even days, with multi-turn, long-horizon interactions between the user and the system. Asking users to stop and score two agents interrupts their actual workflow, makes it hard to collect enough samples, and cannot capture process-level capabilities such as whether the model effectively corrects course in the direction the user wants.

Together, these challenges are pushing evaluation from an exam-style setup toward observing real work.

2. The New Paradigm: Treating Real Conversations as the Testbed

The core idea of the next-generation agent evaluation approach is to stop artificially constructing controlled comparisons and instead use real user-agent conversation logs as the evaluation data source. This approach has three advantages:

• Realistic task distribution: Users arrive with real needs, so task difficulty and diversity naturally resemble production environments.
• Complete session lifecycle: The full process can be observed from task initiation to completion, including rework and course correction.
• Naturally generated feedback: User confirmations, complaints, and corrections are already high-quality feedback signals, so no additional interruption is required.

For this approach to work, three questions must be answered: Where do the signals come from? How are results attributed? How can different models be compared fairly? The following sections address each of these.

3. Extracting Key Signals: Turning Natural Language Feedback into Computable Metrics

Real conversations are semi-structured or even unstructured. The first step in an evaluation system is to define a dictionary of signals that can be extracted automatically. Common signals fall into four categories:

Signal Type	Typical Manifestation	Nature
Explicit confirmation	The user says something like “That works” or “Use it this way”	Explicit, positive
Praise and complaints	Strong positive or negative expressions	Explicit, context-dependent
Correction commands	The user interrupts, asks for a different direction, or requests a redo	Explicit, process-oriented
Execution errors	Non-zero exit code from a shell command, error stack traces	Objective, no semantic judgment needed

Execution error signals are especially valuable: they do not depend on semantic understanding, are cheap to judge, and have low noise. They directly reflect the agent’s stability across the tool-calling chain. Correction commands, by contrast, capture an easily overlooked capability: steerability, or whether the model can quickly converge back to the user’s intended direction after being told something is wrong.

An illustrative extraction logic looks like this:

def extract_signals(session):
    """Extract evaluation signals from one agent session (illustrative)"""
    signals = {
        "confirmed": False,   # whether the user explicitly confirmed the result
        "praise": 0,          # number of positive feedback instances
        "complaint": 0,       # number of negative feedback instances
        "steer_cmds": 0,      # number of correction or rework commands
        "shell_errors": 0,    # number of command execution failures
    }
    for turn in session.turns:
        if turn.role == "user":
            if matches(turn.text, CONFIRM_PATTERNS):
                signals["confirmed"] = True
            elif matches(turn.text, STEER_PATTERNS):
                signals["steer_cmds"] += 1
            signals["praise"] += count_praise(turn.text)
            signals["complaint"] += count_complaint(turn.text)
        elif turn.is_tool_call and turn.exit_code != 0:
            signals["shell_errors"] += 1
    return signals

In practice, the judgment rules are usually implemented with a mix of rule templates and semantic classification models, with manual spot checks on boundary samples to control the false positive rate.

4. Causal Tracking: Using Randomization Inference for Attribution

Extracting signals is only the first step. An immediate question follows: A model may show a higher session completion rate simply because it was assigned easier tasks or more experienced users. Correlation is not causation.

The “causal tracking” approach addresses this by bringing randomized controlled experiments into conversation evaluation:

1. Random assignment: Without the user’s awareness, session requests are randomly routed to different models so that task difficulty and user profile become statistically balanced across groups.
2. Unified baseline: Use the average performance of all participating models as the reference rather than a fixed baseline model, avoiding baseline drift.
3. Net improvement rate: Calculate how much better a target model performs relative to the average, using aggregated metrics such as task success rate or the number of correction turns as the dependent variable.
4. Randomization inference: Use resampling or permutation tests to estimate confidence intervals, avoiding inflated rankings caused by small-sample fluctuations.

Formally, the net improvement rate can be approximated as:

net_rate(m) = (metric(m) - metric(average)) / metric(average)

An illustrative implementation looks like this:

def net_improvement_rate(sessions, model_id, n_boot=1000):
    """Estimate a model's net improvement rate over the average (illustrative)"""
    def rate(sample):
        avg = aggregate_metric(sample)                 # overall average across models
        sub = aggregate_metric([s for s in sample
                                if s.model_id == model_id])
        return (sub - avg) / avg
    point = rate(sessions)
    boots = [rate(resample(sessions)) for _ in range(n_boot)]
    return point, percentile_interval(boots, 0.025, 0.975)

It is important to emphasize that randomization does not remove all confounders. The task assignment policy itself, or user adaptation to a particular interaction style, can still introduce bias. A mature evaluation system therefore also uses stratified randomization, covariate adjustment, or similar techniques for further control.

5. Multi-Dimensional Metrics: Do Not Look Only at a Single Total Score

Even with reliable attribution, a single aggregated score can hide important differences. A well-designed agent leaderboard typically includes at least the following dimensions:

• Overall performance: The net improvement rate aggregated across all scenarios, reflecting the general level.
• Tool execution reliability: Error rates in command execution and API calls, which directly affect trust in automated tasks.
• Steerability: After receiving a correction command, how many turns are needed to get back on track, and whether the model repeatedly reworks the same issue.
• Scenario breakdown: Separate rankings across task types such as office collaboration, software development, and everyday conversation.

An important practical lesson is that the same model can diverge significantly across dimensions. A model with strong execution stability may not be good at understanding ambiguous requirements, while a pleasant conversational style may hide a high error rate in the tool chain. Developers should first clarify their own task profile, then compare models against the relevant sub-metrics rather than simply looking at the top name on the leaderboard.

6. Putting It into Practice: Building Your Own Evaluation Loop

Public leaderboards solve the problem of horizontal comparison. Teams, however, also need an internal evaluation system for vertical iteration. A layered architecture can serve as a reference:

1. Data layer: Collect agent session logs, perform anonymization and structuring, and store them in object storage or a log service.
2. Signal layer: Maintain judgment rules and annotation models for confirmations, corrections, and errors, turning logs into signal streams.
3. Metric layer: Define core metrics such as net improvement rate, task success rate, and average correction turns, computed separately by scenario.
4. Experiment layer: Before rolling out a new model or version, run it in a canary environment with random assignment for online comparison, combined with offline replay validation.
5. Monitoring layer: Connect core metrics to dashboards and alerting so that regressions are detected and rolled back in time.

On the engineering side, teams can use common cloud platform capabilities to build the pipeline quickly. For example, a log service can handle collection and retrieval, object storage can hold session snapshots, a container platform can run evaluation jobs, and monitoring and alerting can guard production metrics. Mainstream cloud providers such as Baidu AI Cloud offer these foundational components, so teams do not need to build everything from scratch.

7. Limitations and Caveats

Evaluating on real conversations is not a silver bullet. The following issues require attention during implementation:

• Selection bias: Session data comes from users who are willing to use the agent, so tasks may concentrate on technical scenarios such as programming and operations. Extrapolating results to other domains requires caution.
• Signal noise: Natural language feedback includes sarcasm, omission, and context dependence. Pure rule-based judgment can easily misclassify such cases, so a manual spot-check channel should be retained.
• Privacy and compliance: Conversations may contain sensitive information. Anonymization and access control must be designed in from the start rather than applied after the fact.
• Metric gaming: Over-optimizing a single metric, such as minimizing command error rate, may push the model to avoid complex operations and damage the real user experience.
• Timeliness: Model versions and user task distributions evolve continuously. Metric conclusions are time-sensitive and need rolling re-evaluation rather than a one-time judgment.

8. Conclusion

Agent evaluation is undergoing a methodological upgrade: from static exams and manual voting toward real conversations, signal extraction, and causal inference. For developers, the value of a leaderboard is not in memorizing rankings but in understanding the capability dimensions behind the metrics: whether tool execution is reliable, whether the model can converge quickly after being corrected, and whether it remains stable under a realistic task distribution. For engineering teams, the closed loop of collection, extraction, attribution, and monitoring can be internalized as evaluation infrastructure, giving every model upgrade a trustworthy data foundation.