How to Evaluate AI Agents: Real Conversations and Causal Inference in Practice

    This article explains a practical approach to AI agent evaluation that goes beyond human-preference voting. It walks through extracting attributable signals from real long-running conversations, applying randomized inference to control for confounding variables, and aggregating results into interpretable metrics such as command-execution reliability, self-correction ability, scenario capability profiles, and session-level unattributable rate. It also covers the data pipeline, metric formulas with edge-case handling, evaluation infrastructure design, and a pre-launch checklist of common pitfalls.

    TL;DR: The Takeaways First

    • Evaluation goal: Not simply “which model is better,” but estimating an agent’s capability difference relative to the average using real conversations.
    • Data pipeline: Real conversation collection → privacy redaction → signal extraction → action attribution → randomized inference → metric aggregation.
    • Core metrics: Command-execution reliability, self-correction ability, scenario capability profiles, session-level unattributable rate.
    • Applicability boundaries: Smooth the metric when the baseline success rate is near 0; feedback without a clear target cannot be attributed to the most recent action by default; sensitive data must be redacted first.
    • Common pitfalls: Looking only at absolute success rate, broken randomization, incorrect attribution, uncalibrated subjective signals, evaluation data not versioned.

    1. Why Traditional Voting Falls Short

    Traditional agent evaluation often relies on human-preference evaluation or pairwise voting: evaluators pick the better of two model responses to the same prompt, and a win rate is computed. This is acceptable for single-turn Q&A, but its limitations become concentrated once the object of evaluation is an agent.

    Agents are typically characterized by long task cycles, many steps, and the need to call tools—for example, executing shell commands, reading and writing files, and calling APIs. Their output is often not a piece of text but a result of “whether the task was actually completed.” Therefore, traditional voting-based evaluation has at least three problems:

    1. A single vote cannot reflect process quality. Two models may both give seemingly reasonable answers, but one may call the wrong command at step three, causing the rest of the task to fail.
    2. Static question banks drift from the real distribution. Carefully constructed test questions can hardly cover the diverse needs of real users in long-running tasks.
    3. Manual annotation is costly and subjective. Having humans fully reproduce a multi-step task and then score it is extremely expensive and hard to scale.

    Therefore, a more credible path is: automatically extract signals with clear meaning from real conversations between users and agents, then use statistical inference to estimate the relative difference in model capability.

    2. FAQ: How Do You Evaluate Agents, Choose Samples, Quantify Self-Correction, and Interpret Net Improvement Rate?

    Q1: How do you evaluate agents?
    The core pipeline is “real conversations → attributable signals → randomized inference → stratified metrics.” Don’t jump from subjective voting straight to conclusions. First define which behaviors count as success, which count as failure, and which feedback can be attributed to a specific action, then compare models using relative metrics.

    Q2: How do you choose real conversation samples?
    Samples should come from real user conversations rather than hand-constructed question banks. When selecting, focus on three points: whether it covers the main task types; whether it can be randomly split across different models; and whether redaction and access control have already been completed. For different scenarios such as programming, operations, office work, and customer service, sampling should be stratified to avoid a single scenario dominating the conclusion.

    Q3: How do you quantify self-correction ability?
    Self-correction ability can be measured by metrics such as “post-correction fix rate,” “average number of correction rounds,” “post-correction confirmation rate,” and “error recovery latency.” The prerequisite is that the user’s correction signal can be attributed to a specific action. For example, if a user says “no, you should delete that field” and the agent’s previous action can be located, you can then measure whether the model correctly fixed it after the correction.

    Q4: How do you interpret net improvement rate?
    Net improvement rate is a relative metric that expresses how much a model pushes task success rate relative to the overall average. It is not absolute success rate. If the net improvement rate rises from +0.9% to +5.1%, this means the net improvement rate difference relative to the average is about 4.2 percentage points; it cannot be equated directly with a 4-percentage-point increase in absolute success rate.

    3. Extracting Attributable Signals from Real Conversations

    The evaluation data source is not a constructed question bank but long-running conversations between real users and agents. Real conversations naturally carry users’ feedback on “whether the task succeeded.” The key is not to directly ask users “which model do you think is better,” but to mine high-confidence, attributable signals from the evolution of the conversation.

    Common signals fall into four categories:

    Signal type Meaning Example Characteristics
    Explicit task confirmation User explicitly approves the result “Yes, that’s it” / “Done” Strong positive signal
    Praise and complaints User expresses emotion “Nice” / “That’s wrong” / “Wrong again” Needs context-based attribution
    Correction instruction User interrupts and gives a correction request “You should delete that field” Good for evaluating self-correction
    Command execution error Tool returns a non-zero exit code or error stack Shell command failure Objective, no subjective judgment needed

    An illustrative pipeline:

    1. def extract_signals(session):
    2. signals = []
    3. turns = session.turns
    4. for i, turn in enumerate(turns):
    5. # Objective signal: tool execution error
    6. if turn.tool_result and turn.tool_result.exit_code != 0:
    7. signals.append({
    8. "type": "command_error",
    9. "turn_index": i,
    10. "turn_id": turn.id,
    11. "severity": classify_error(turn.tool_result.stderr),
    12. })
    13. # Subjective signals: confirm, praise, complain, correct
    14. intent = classify_user_intent(turn.user_text)
    15. if intent in ("confirm", "praise", "complain", "correct"):
    16. signals.append({
    17. "type": intent,
    18. "turn_index": i,
    19. "target_turn": resolve_target(turns, i),
    20. "confidence": turn.confidence
    21. })
    22. return signals

    The most important part here is the attribution rule of resolve_target. A common mistake must be avoided: when the user does not clearly indicate a target, the system attributes feedback to the most recent action by default. This creates systematic bias, making metrics look more accurate while actually being distorted.

    Recommended rules:

    1. Attribution when the target is explicit: When user feedback contains clear references such as “step X,” “this command,” “this field,” or “this file,” and maps uniquely to an action, attribute it to that action.
    2. Multiple candidates / ambiguity: If the feedback can correspond to multiple actions, mark it as ambiguous and send it to manual review.
    3. No explicit target: If the user only says “the whole thing is wrong” or “still not working” and no specific action can be located, mark it as session_level_unattributable (session-level unattributable).
    4. Manual review threshold: The following samples are recommended for manual review: intent classification confidence below 0.80; attribution result of ambiguous or session_level_unattributable; serious complaints; critical task failures; consecutive user corrections after a command error. For daily operation, sampling review of at least 5% or 1,000 items per day is recommended, raised to 20% when metrics drift.

    4. From Signals to Trustworthy Metrics

    Signals alone are not enough. Real conversations contain many confounding variables: task difficulty varies, user preferences vary, and the timing of conversations varies. Without handling them, treating “a model’s conversation success rate is high” as “the model is stronger” makes the conclusion unreliable.

    A more robust method is to introduce randomized inference. Its core idea is to introduce randomness at the traffic-distribution stage so that different models are as similarly distributed as possible across dimensions such as task difficulty, user profile, and scenario type. This way, the observed success-rate differences between models can be attributed to the models themselves to a greater degree rather than to external interference.

    Net improvement rate can be abstracted as:

    1. Net improvement rate = (model weighted success rate - overall weighted success rate) / overall weighted success rate

    Here “weighted” means different signals have different confidences. For example:

    1. weight = {
    2. "confirm": +1.0,
    3. "praise": +0.5,
    4. "complain": -0.5,
    5. "correct": -0.8,
    6. "command_error": -1.0,
    7. }

    But the boundaries of the formula must be respected. When the overall baseline success rate baseline approaches or equals 0, using division directly creates division-by-zero risk or numerical instability. The following handling is recommended:

    1. Prefer reporting the absolute difference: Δp = model weighted success rate - baseline weighted success rate, with a confidence interval attached.
    2. Use Laplace smoothing: For example p' = (success + 1) / (total + 2), for small-sample or low-baseline scenarios.
    3. Use a Bayesian posterior: Constructing a posterior mean based on the Beta distribution is better suited to small samples and sparse signals.
    4. Engineering division-by-zero protection: If the baseline still approaches 0 after smoothing, you can set denom = max(baseline_smooth, 1e-6), but this is only to prevent program errors; in this case conclusions should mainly present the absolute difference, confidence interval, and sample size.

    For example: if a model’s net improvement rate rises from +0.9% to +5.1%, it should be expressed as “the net improvement rate difference relative to the average is about 4.2 percentage points, which cannot be equated directly with a 4-percentage-point increase in absolute success rate.”

    To reduce misjudgment, the randomized experiment design should include at least the following checklist:

    • Split ratios: Define each model’s traffic share, avoiding a high-traffic model dominating others due to excessive samples.
    • Stratified randomization: Stratify by task type, user type, and scenario priority to avoid a certain class of tasks concentrating on a single model.
    • Sample size: Estimate based on minimum detectable effect, confidence level, and statistical power.
    • Confidence intervals: Prefer Wilson intervals, bootstrap intervals, or Bayesian posterior intervals.
    • Significance testing: Proportion tests, mixed-effects models, or stratified estimation are all viable, but avoid repeated testing that causes false positives.
    • Random seeds and snapshots: Record the random seed, data snapshot version, and metric computation version to ensure reproducibility.

    5. Breaking Down Core Metrics

    A mature evaluation system should not have a single total score but should be broken into several interpretable sub-metrics. The following metrics are especially critical in agent scenarios:

    Metric Definition Formula Data source Applicable scenario Common errors
    Command-execution reliability Proportion of tool calls that succeed on the first try without retry Successful first calls / calls that should be executed Tool logs, command return codes Shell, API, file operations Environment permission differences, timeout misjudgment
    Self-correction ability Whether the agent can quickly return to the correct path after user correction Post-correction successful fixes / explicit corrections Correction signals, subsequent confirmation signals Long-running tasks, human-agent collaboration Feedback without an explicit target, incorrect attribution
    Scenario capability profile Relative capability broken down by scenario such as work, programming, chat Per-scenario net improvement rate or success rate Conversation labels, task classification Model selection, scenario optimization Imbalanced category samples
    Session-level unattributable rate Proportion of feedback that cannot be attributed to a specific action Unattributable signals / total feedback signals Attribution module output Data quality monitoring Attribution rules too loose or too strict
    Subjective signal calibration error Degree of inconsistency between intent classification and manual review Inconsistent count / manual review sample count Manual review records Classifier maintenance Annotation standard drift

    The value of per-dimension statistics lies in the fact that enterprises usually do not need a model that ranks first in everything, but rather a model that is strong enough in a specific scenario. For example, a model that leads in code generation and debugging but is weaker in open-ended chat benefits from per-dimension rankings, which make selection more precise.

    6. Building Evaluation Infrastructure

    An agent evaluation system that runs over the long term is essentially a high-concurrency online service that needs to accumulate massive conversation data. It can be built from general cloud capabilities:

    • Object storage: Stores raw conversations, tool-call logs, and evaluation result snapshots. Conversation data redaction, sensitive-field filtering, least-privilege access, access auditing, and retention periods must be added to avoid credentials, file contents, and personal information entering the evaluation pipeline.
    • Message queue: Decouples “conversation collection” from “signal extraction,” handles traffic peaks, and supports offline batch recomputation.
    • Container platform: Hosts computation tasks such as signal extraction, intent classification, and metric aggregation, enabling elastic scaling.
    • Logging service: Uniformly collects conversation logs and command execution results. Log fields should be structured and sensitive fields masked; access control, audit records, and retention periods should also be configured.
    • Monitoring and alerting: Set alerts for the success rate, latency, data backlog, and attribution drift of evaluation tasks to ensure the evaluation pipeline itself remains available.

    Privacy and compliance should be designed up front. Raw conversations may contain sensitive content such as user questions, file paths, command arguments, account information, and keys. The recommended principles are: redact at collection, encrypt at storage, least-privilege access, full audit trails, short retention of details, long retention of aggregates.

    7. Pre-Launch Checklist and Common Pitfalls

    Before launch, check each item:

    1. Randomization validation: Check whether conversations across different models are balanced in task difficulty, scenario type, and user profile.
    2. Attribution sampling review: Confirm that resolve_target does not attribute targetless feedback to the most recent action by default, and that the manual review process is executable.
    3. Consistent baseline definition: Define the net improvement rate denominator, smoothing method, small-sample handling, and division-by-zero strategy.
    4. Subjective signal calibration: Periodically check intent classification accuracy, complaint recognition rate, and correction recognition rate.
    5. Data snapshot versioning: The data version, code version, and random seed that metric computation depends on must all be traceable.
    6. Privacy and compliance check: Confirm that redaction, access control, auditing, and retention periods are in effect.
    7. Degradation mechanism: When the evaluation pipeline fails, avoid contaminating online conclusions or affecting the main business.

    Common pitfalls include:

    • Don’t treat absolute success rate as the only standard; it must be combined with a baseline for relative comparison.
    • Don’t cut corners on randomization, or confounding variables will mix back in.
    • Don’t force attribution on ambiguous feedback; incorrect attribution systematically distorts metrics.
    • Don’t treat subjective signals and objective signals equally; command execution errors should be treated as hard metrics.
    • Don’t make the evaluation system itself non-reproducible, or conclusions cannot be reviewed.

    8. Summary

    Agent evaluation is moving from “manual voting + static question banks” to “real conversations + causal inference.” Its core chain can be summarized in three steps: extract attributable signals from real long-running conversations; reduce confounding variables such as task difficulty and user preferences through randomized inference; then break results down into interpretable metrics such as command-execution reliability, self-correction ability, and scenario profiles.

    For teams that want to objectively measure agent capability, the key is not just choosing a model, but building a decoupled, observable, reproducible evaluation system. Only by incorporating data redaction, attribution rules, metric definitions, randomized experiments, and version management into the infrastructure together can evaluation conclusions remain credible over the long term.