A New Paradigm in AI Agent Evaluation: Real Conversation Signals and Causal Inference

1. Why Agent Evaluation is Harder Than Chatbot Evaluation

Large language model (LLM) evaluation is undergoing a quiet paradigm shift. Over the past few years, the industry has primarily measured model capabilities using two methods: static benchmarks (scoring on fixed datasets) and human preference voting (having users compare responses from two models side-by-side). While these methods were effective in the “chatbot” era, they show clear limitations in agent scenarios.

1. Three Limitations of Static Benchmarks

• Data Contamination: High-quality test questions are likely already in the training corpus. Scores may reflect memorization rather than actual problem-solving ability.
• Capability Saturation: Top-tier models are approaching score ceilings, reducing differentiation. A fraction of a point difference might just be noise.
• Scenario Mismatch: Single-turn Q&A tests knowledge retrieval, whereas agent tasks require multi-step planning, tool calling, and environmental feedback loops. The correlation between the two is limited.

2. Specifics of Agent Tasks
Agent tasks are typically long-horizon. A real-world development task might span dozens of interaction rounds, involving file read/write operations, command execution, dependency installation, and result verification. The model must not only “say the right thing” but also “do the right thing”—the correctness of each tool call is directly verified by the environment. More importantly, errors are inevitable. Task success often depends on the model’s self-repair capabilities after a failure, which is exactly what static benchmarks struggle to evaluate.

3. Limitations of Human Voting
Pairwise voting mechanisms inherently favor “perceived quality.” Users are more easily impressed by tone, formatting, and verbosity than by verifying whether the code actually runs. Furthermore, voting scenarios are mostly short conversations that fail to cover long-horizon tasks. The proportion of users willing to complete voting is small, making the sample size insufficient to support fine-grained, scenario-specific conclusions.

2. Data Source Evolution: From Voting to Real Conversations

The core idea of next-generation agent evaluation is to stop initiating separate comparison votes and instead use real online agent conversations directly as evaluation data. Users’ real behaviors—confirming task completion, mid-task corrections, or praising/complaining about results—serve as the most credible labels. Users are “voting with their feet.”

This paradigm brings three benefits:

1. Authentic Task Distribution: Conversations originate from users’ spontaneous needs, naturally covering long-tail scenarios and avoiding the distortion of “creating tests for the sake of testing.”
2. Near-Zero Labeling Cost: Signals are automatically extracted from conversation logs without additional manual annotation, allowing for continuous, large-scale accumulation.
3. Longitudinal Comparison: A consistent signal system can depict the true magnitude of progress between model generations.

The trade-off is a significant increase in engineering complexity. Conversation data involves privacy compliance, requiring desensitization and authorization mechanisms. Signals are noisy, demanding rigorous extraction and filtering pipelines. More importantly, different models serve different user groups with varying task difficulties. Confounding variables must be controlled using causal inference methods—which is exactly what “causal tracing” aims to solve.

3. Signal System: Four Automatically Extractable Key Signals

Real conversations are semi-structured text. The first step in an evaluation system is transforming them into computable structured signals. In industry practice, the focus is typically on the following four categories:

1. Explicit Confirmation (Strong Positive Signal): The user explicitly states at the end of a task, “Done” or “This is exactly what I wanted.” This signal has high credibility, but it requires distinguishing between “polite closings” and “actual acceptance,” usually judged in combination with subsequent behavior (whether they ask follow-up questions or start a new task).

2. Praise and Complaints (Sentiment Polarity): Positive and negative feedback within the conversation. Complaint signals are particularly valuable—they often pinpoint the exact point of failure, such as “You broke the file I told you not to touch.”

3. Correction Commands (Correctability): Instructions where the user changes requirements mid-task or pulls back the direction, such as “Try another approach” or “Do not modify this config.” The density of correction commands and the subsequent convergence speed directly measure a model’s “correctability.”

4. Command Execution Errors (Tool Reliability): This is a hard signal unique to agent scenarios. Non-zero exit codes from shell commands, error stacks, and retry failures can be deterministically extracted from execution logs with almost no subjective noise.

The extraction process typically uses a dual-channel approach of “rules + models”: structured information like exit codes and exception stacks are extracted using rules; natural language signals like confirmations, complaints, and corrections are classified using discriminative models, followed by confidence filtering and manual sampling calibration to control the false positive rate.

4. Causal Tracing: Estimating Net Improvement Rate

With signals in place, how do we answer “How much better is Model A than Model B?” Directly comparing the mean of conversation signals is flawed because traffic compositions differ across models. Some models handle more high-difficulty coding tasks, while others are used for simple Q&A. Task difficulty, user demographics, and domain distribution are all confounding variables.

The causal tracing approach is inspired by online experiments: user requests are distributed to different models based on a randomized strategy, similar to an A/B test. Randomization ensures that the task difficulty distribution and user groups for each model converge on a large-number scale. Confounding variables are “averaged out,” meaning group differences can be attributed to the model itself.

Based on this, a “Net Improvement Rate” can be estimated for each model: the mean of the model’s conversation signals minus the average level of all participating models. The interpretation is intuitive—if a model has a net improvement rate of +5%, it means that under an equivalent task distribution, it generated about 5 percentage points more net positive results than the “average model.” If it is 0, it is on par with the average. Compared to rankings that only show positions, the net improvement rate answers “how much better,” providing much more information.

A simplified estimation example is as follows:

import numpy as np
def net_improvement(model_scores, n_boot=2000, seed=0):
    """
    model_scores: dict[model_id -> array of signal scores for that model's sessions]
    Score encoding example: +1 user explicitly confirms completion; 0 no explicit positive/negative signal;
                 -0.5 correction command appears; -1 complaint or critical command execution failure
    Returns: Net improvement rate relative to the overall average for each model and its 95% confidence interval
    """
    rng = np.random.default_rng(seed)
    grand_mean = np.concatenate(list(model_scores.values())).mean()
    def estimate(scores):
        return {m: s.mean() - grand_mean for m, s in scores.items()}
    point = estimate(model_scores)
    boots = []
    for _ in range(n_boot):
        resample = {
            m: rng.choice(s, size=len(s), replace=True)
            for m, s in model_scores.items()
        }
        boots.append(estimate(resample))
    ci = {
        m: (np.percentile([b[m] for b in boots], 2.5),
            np.percentile([b[m] for b in boots], 97.5))
        for m in model_scores
    }
    return point, ci

In engineering, several statistical details require attention: First, confidence intervals must be reported. If a net improvement rate’s interval crosses zero, it cannot be claimed as significantly better than average. Second, sample size and task variance jointly determine statistical power. Long-horizon tasks have high signal variance, typically requiring thousands to tens of thousands of conversations. Third, when comparing multiple scenarios and metrics simultaneously, multiple comparison corrections are necessary to avoid cherry-picking favorable metrics.

5. Key Evaluation Dimensions

1. Overall Net Improvement Rate: The baseline water level, suitable for generational comparisons and model coarse ranking.

2. Scenario-Specific Net Improvement Rate: Statistics categorized by scenarios like work, coding, and chatting. A model performing excellently in chat might be below average in coding, and vice versa. Aggregate scores can mask these structural differences.

3. Shell Command Execution Reliability: Core observations include the non-zero exit code rate, first-time execution success rate, retry-and-repair success rate after errors, and the identification and confirmation of high-risk commands. This is the “hardest” metric in agent deployment—if the command doesn’t run, everything else is zero.

4. Correctability: Upon receiving a correction command, how many rounds it takes for the model to converge to the user’s intent, and whether it repeats similar errors. A model with poor correctability traps users in a “tug-of-war,” resulting in an actual experience far worse than benchmark scores suggest.

5. Long-Horizon Consistency: Compares signal density in the first half versus the second half of a task, measuring the model’s state retention and goal drift within long contexts.

6. Practical Implications for Developers and Enterprises

For Builders: Design tool calls as first-class citizens. Command execution should have timeout, retry, and idempotency protections. When encountering errors, the model should parse the stack trace, locate the cause, and generate a repair plan rather than blindly retrying. Additionally, maintain clear task states to ensure smooth direction switches without losing completed work when users issue mid-task corrections.

For Evaluators: Teams are advised to build an internal conversational evaluation loop—embed tracking in agent applications to collect conversation and execution logs, run signal extraction pipelines, and output scenario-specific net improvement rate dashboards. Static benchmarks can be downgraded to regression testing tools for quick smoke tests during version iterations. Go-live decisions should be based on real conversation metrics.

For Selection: When evaluating models, prioritize net improvement rates, command reliability, and correctability in real task conversations rather than single benchmark scores. Open-weight models and managed services have their trade-offs: the former facilitates private deployment and domain fine-tuning for data-sensitive scenarios; the latter eliminates operational overhead for rapid validation. During deployment, platforms like Baidu AI Cloud provide general capabilities such as model services, containers, and monitoring alerts. Utilizing such platforms can help build a complete pipeline from deployment and stress testing to evaluation and observation, creating a positive feedback loop of “evaluate, improve, re-evaluate.”

7. Conclusion

Agent evaluation is shifting from “asking if the model knows” to “seeing if the model gets it done.” Using real conversations as the data source, four key signals as observations, and causal tracing’s net improvement rate as the metric, this methodology answers what developers truly care about: under a real task distribution, how many more problems did the model solve compared to the average, and how many fewer errors did it make? Understanding and reusing this methodology holds far more long-term value than memorizing any leaderboard ranking.