How to Build an AI Agent Evaluation System Based on Real Conversations

    This guide explains how to build an AI agent evaluation system grounded in real conversation data rather than static benchmarks or user-vote popularity. It covers extracting computable signals from session logs, distinguishing normal completion from failed or abandoned sessions, measuring reliability and controllability (command execution, tool calls, long-context consistency, and self-correction), and calculating a net improvement rate with clear baselines and confidence intervals. You will also find an end-to-end error-and-correction walkthrough, a comparison of static leaderboards against real-conversation evaluation, a closed-loop iteration process, and practical guidance on compliance, credential scrubbing, and a three-layer enterprise architecture.

    Key Takeaways

    1. Agent evaluation should shift from “is the answer good” to “is the task completed, is the process reliable, and can it self-correct after errors.”
    2. Real-conversation evaluation should be based on observable behavior signals, not a single user-satisfaction vote.
    3. Session termination must distinguish “leaving after normal completion” from “terminating after failure or abandonment.”
    4. The net improvement rate needs a clear definition, baseline, confidence interval, and sample-size requirement to avoid small-sample misjudgment.
    5. Reliability and controllability must be measured separately, especially command execution, tool calls, long-context constraints, and correction ability.
    6. Before using real conversation data for evaluation, you must complete informed consent, privacy de-identification, credential scrubbing, and retention/deletion mechanisms.

    Contents

    1. Evaluation Background: Agents Moving from Generating Text to Completing Tasks
    2. Limitations of Fixed Question Banks and Human Voting
    3. Core Idea of Real-Conversation Evaluation
    4. Signal System Design: From Behavior to Computable Metrics
    5. Causal Tracking: Avoiding “Whoever Uses More Scores Higher”
    6. Net Improvement Rate: Definition, Calculation, and Confidence Intervals
    7. Static Leaderboards vs. Real-Conversation Evaluation
    8. Key Reliability and Controllability Metrics
    9. Engineering the Evaluation Pipeline
    10. End-to-End Case: Evaluating an Error and Correction
    11. From Evaluation to Iteration: Closed-Loop Optimization
    12. Risk Governance: Compliance, Credential Scrubbing, and Data Governance
    13. Enterprise Implementation: A Three-Layer Evaluation Architecture
    14. FAQ
    15. Conclusion

    1. Evaluation Background: Agents Moving from Generating Text to Completing Tasks

    As large language models have grown more capable, agents have expanded from single-turn question answering into complex scenarios such as multi-turn collaboration, tool invocation, code execution, document processing, and data analysis. Traditional evaluation often focuses on whether “the answer is reasonable” or “the answer is correct.” In real business contexts, however, users care more about whether the agent can reliably complete tasks, correct itself promptly after errors, and invoke the necessary tools to return usable results in line with the user’s intent.

    For this reason, agent evaluation needs to shift from static questions to real tasks, and from subjective human scoring to observable behavior signals. The value of real-conversation evaluation is that it reflects how an agent performs across long-horizon, multi-turn, dynamically changing tasks—much closer to the user experience and business value seen in production.

    2. Limitations of Fixed Question Banks and Human Voting

    Traditional agent evaluation often relies on fixed question sets, expert review, or user-preference voting. These methods have some value in early model stages, but they expose two major problems in real agent scenarios.

    First, static questions struggle to cover the diversity of real tasks. Real tasks usually involve ambiguous requirements, implicit constraints, multi-step dependencies, and unexpected anomalies. A fixed question bank typically covers only standardized problems and cannot measure an agent’s robustness or its ability to execute long chains of steps.

    Second, human voting and single-preference signals are hard to scale and easy to distort. Agent tasks take a long time, and a single session may involve many steps, repeated tool calls, and multiple corrections. Judging every session by hand is costly and slow. At the same time, users may give positive feedback because a response reads smoothly, even though the task was never actually completed; conversely, they may give negative feedback because of the interface or wait time, even though the model made no clear error. Evaluation therefore needs finer-grained behavior signals and more rigorous statistical methods.

    3. Core Idea of Real-Conversation Evaluation

    The basic idea of real-conversation evaluation is to extract computable signals directly from the interaction between users and the agent, rather than constructing a separate, artificial scoring task. Its focus is on observing the real behavioral reactions users show while a task is being executed.

    Common observable signals include:

    1. Explicit confirmation: The user says things like “that works,” “continue,” “done,” or “this is exactly what I wanted,” indicating that the task direction and result have been accepted.
    2. Praise and complaints: Users expressing satisfaction or gratitude, or raising dissatisfaction, doubts, or failure feedback, can serve as positive or negative signals.
    3. Correction instructions: The user restates a requirement, adds constraints, points out an error, and asks for a retry—signaling that the agent may have had a misunderstanding or execution error in a prior step.
    4. Command or script execution errors: When the agent fails to generate a command, pass parameters, or call an external capability—for example, wrong parameters, insufficient permissions, timeouts, a non-existent path, or invalid output format—these should be treated as negative reliability signals.
    5. Repeated follow-up questions and abandonment: A user repeatedly asking about the same thing may indicate that the agent is failing to move the task forward effectively.

    One point needs particular correction: the definition of “session termination.” Session termination should not be treated as failure by default, but should be split into two cases:

    • Normal ending: The final turn contains explicit confirmation, or the last N turns contain no corrections and the task result has been delivered—record this as a success or neutral signal.
    • Abnormal termination: No confirmation after a correction, a command error, an execution timeout, the user explicitly giving up, or the user leaving after a run of consecutive failures—count this as a negative signal.

    This distinction directly affects how downstream metrics such as abandonment rate, task completion rate, and reliability are calculated.

    4. Signal System Design: From Behavior to Computable Metrics

    When building a real-conversation evaluation system, the first step is to build a signal-extraction layer. This layer is responsible for distilling structured metrics from session logs, tool-call records, command-execution results, and user feedback.

    Signals fall into four categories:

    1. Result signals: Determine whether the task was ultimately completed—for example, whether the user confirmed a usable result, moved on to a follow-up step, or ended the task on their own without further correction.
    2. Process signals: Determine whether the agent advanced the task efficiently, including the number of steps, repeated-call counts, the proportion of ineffective attempts, and the frequency of context switching.
    3. Correction signals: Measure whether the agent can accept user feedback and fix issues promptly—for example, after the user says “don’t write it that way,” “the parameters are wrong,” or “run it again,” whether the agent correctly understands the correction intent and avoids repeating the same mistake in later steps.
    4. Execution-reliability signals: Focus on the stability of tool calls, command generation, and code execution, including whether a command succeeds on the first try, whether error messages are understood, whether the recovery path after failure is reasonable, and whether the same error is produced repeatedly.

    In code, you need to specify for each signal whether it is a boolean or a count, and state the statistical window. For example, explicit_confirm indicates whether explicit confirmation appears in the current session; correction_count indicates the number of user corrections in the current session; repeated_correction_count indicates corrections that recur after the same type of error; and abandon indicates whether the session is an abnormal abandonment. A correction score should not simply deduct points for every correction, but should be scored by correction success rate or repeated-error rate.

    Sample code:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class SessionSignals:
    4. # Boolean signals: whether present in the current session
    5. explicit_confirm: bool = False
    6. praise: bool = False
    7. complaint: bool = False
    8. abandon: bool = False
    9. # Count signals: cumulative count within the current session
    10. correction_count: int = 0
    11. repeated_correction_count: int = 0
    12. execution_error_count: int = 0
    13. repeated_execution_error_count: int = 0
    14. def session_score(s: SessionSignals) -> float:
    15. # Correction scoring: only penalize repeated same-type corrections,
    16. # to avoid over-penalizing the user's act of requesting a fix
    17. correction_penalty = min(
    18. s.repeated_correction_count,
    19. s.correction_count
    20. ) * 1.0
    21. # Reliability scoring: command or tool errors, and repeated errors
    22. reliability_penalty = (
    23. s.execution_error_count * 0.8
    24. + s.repeated_execution_error_count * 1.2
    25. )
    26. score = 0.0
    27. score += 2.0 if s.explicit_confirm else 0.0
    28. score += 1.0 if s.praise else 0.0
    29. score -= 1.5 if s.complaint else 0.0
    30. score -= correction_penalty
    31. score -= reliability_penalty
    32. score -= 2.0 if s.abandon else 0.0
    33. return max(-5.0, min(5.0, score))

    This example only illustrates the signal-weighting idea. In a real system, weights must be recalibrated for each business scenario. The recommended approach is to fit weights on a small-traffic, human-labeled set using logistic regression, Bayesian calibration, or rule-based constraints, and to set confidence intervals and anomaly filters. Weights should undergo versioned backtesting to avoid frequent manual tuning.

    5. Causal Tracking: Avoiding “Whoever Uses More Scores Higher”

    A key challenge in real-conversation evaluation is that users do not choose models at random. Users, tasks, and scenarios differ inherently. Simply comparing the satisfaction of one model’s sessions can mistake task difficulty and user-group differences for differences in model capability.

    To address this, you need to introduce inference methods similar to causal tracking. The goal is not simply to count how much praise a model received, but to estimate a model’s net improvement relative to an average baseline.

    Common implementation approaches include:

    1. Randomized assignment: When conditions allow, randomly assign different models to the same or similar tasks and observe differences in session outcomes and feedback.
    2. Stratified sampling: Stratify by task type, user type, session length, and tool-call complexity to prevent any single sample group from overly influencing the overall conclusion.
    3. Baseline comparison: Set a stable baseline model or historical version as a control, and compute the new model’s change relative to the baseline.
    4. Propensity scoring or weighted estimation: When full randomization is impossible, use statistical methods to balance sample characteristics and reduce selection bias. Propensity-score methods can draw on the matching idea proposed by Rosenbaum and Rubin (1983), and can be combined with inverse probability weighting, stratified estimation, or difference-in-differences.

    When applying these methods, pay attention to the conditions for causal identification:

    • First check whether propensity scores overlap. Comparison is only meaningful on the task or user strata jointly covered by both the model group and the baseline group.
    • Judge significance by clustering at the session or user level, not by treating each turn message as independent. Multiple turns in one session and multiple requests from one user are not independent.
    • Users not providing feedback is non-random missingness. Report the no-feedback rate and run a worst-case sensitivity analysis. For example, if a large share of non-responding users actually had a worse experience, the model’s advantage could be overstated.
    • For high-value negative samples, retain manual attribution and review to avoid misinterpretation by automatic metrics.

    6. Net Improvement Rate: Definition, Calculation, and Confidence Intervals

    The net improvement rate can be understood in English as uplift or relative lift; its core is the outcome advantage relative to the baseline. It is closer to version-iteration decisions than a single success rate, but it requires a clear definition.

    Let the outcome metric for the model group be Y and for the baseline group be Y0. Y can be task success rate, average task score, user confirmation rate, pairwise win rate, or correction success rate. There are two common forms of net improvement rate:

    1. Absolute net improvement = Model group metric − Baseline group metric
    2. Relative net improvement = (Model group metric − Baseline group metric) / Baseline group metric

    Direction convention: a positive value means the model group outperforms the baseline group. If a metric is an inverse indicator such as error rate, complaint rate, or average correction rounds, you must first convert it to a positive indicator or use a reversed definition.

    For example, if the baseline task success rate is 62% and the new version is 66%, the absolute net improvement is +4 percentage points; computed as a relative lift, it is (66 − 62) / 62 ≈ 6.5%. If the confidence interval crosses 0, you cannot conclude that the new version is better based on the mean alone.

    Recommended decision rules for engineering use:

    1. Compute by task type, not just the overall mean.
    2. Bootstrap at the session level to produce a 95% confidence interval.
    3. Before launch, estimate the minimum detectable effect (MDE) and the required sample size.
    4. In small-sample scenarios, avoid over-interpreting short-term fluctuations.
    5. The net improvement rate is a supporting decision metric; it cannot replace hard constraints such as safety, compliance, and cost.

    The value of the net improvement rate is that it turns “seems more popular” into “whether it truly delivers task-level improvement.” For engineering teams, metrics like this suit version canary releases, model routing, and continuous regression—not a blanket replacement for all offline evaluation.

    7. Static Leaderboards vs. Real-Conversation Evaluation

    Dimension Static leaderboards Real-conversation evaluation
    Task coverage Mostly standardized questions Closer to real long-horizon tasks and multi-turn interaction
    Evaluation cost Lower upfront cost Requires collection, de-identification, signal extraction, and statistical engineering
    Main bias Question overfitting, task distortion Selection bias, feedback noise, uneven samples
    Suitable decisions Coarse screening, basic capability comparison Version iteration, canary release, routing strategy
    Risk control Easy to misjudge real experience Requires causal correction and compliance governance

    8. Key Reliability and Controllability Metrics

    In agent tasks, what users are most sensitive to is often not whether the model “talks nicely,” but whether it “does things reliably.” The evaluation system therefore needs separately designed reliability and controllability metrics.

    1. Command execution reliability
      Command execution reliability concerns how correctly the agent generates commands, parameters, paths, and call arguments. Common problems include command syntax errors, missing parameters or type mismatches, non-existent paths or resources, insufficient permissions, error output that goes unread, and re-running the same wrong command after failure. Observable metrics include first-try command success rate, tool-call success rate, error-recognition rate, and repeated-error rate.

    2. Direction adjustment and correction ability
      In real tasks, users frequently change requirements mid-course. Direction-adjustment ability assesses whether the agent can accurately understand newly added constraints and stay consistent in later steps. For example, when a user says “don’t delete the original file,” “access read-only only,” or “keep the old version,” whether the agent adheres to these constraints in subsequent operations is an important reflection of controllability. Metrics can include correction success rate, average recovery steps, and repeated-correction rate.

    3. Long-context consistency
      In long-horizon tasks, the agent must remember early goals, constraints, and completed steps. If the model forgets earlier constraints in the latter half, it forces the user to repeat corrections. Evaluation should therefore focus on context-retention ability, not just the quality of a single-step answer.

    4. Tool-call appropriateness
      Whether, when, which, and how the agent combines tool calls all affect task results. Evaluation needs to distinguish two types of problems: “not calling when it should” and “over-calling when it should not.”

    5. Session termination and abandonment rate
      Normal ending rate, abnormal termination rate, and user abandonment rate must be tracked separately. Normal endings should not count as failures; only abnormal abandonment counts as negative experience. A high abandonment rate usually means the agent lacks the ability to advance tasks, or that a critical step repeatedly fails.

    9. Engineering the Evaluation Pipeline

    A sustainable real-conversation evaluation system should include a complete engineering pipeline.

    1. Session collection and de-identification
      The system must capture the full interaction between users and the agent, including inputs, outputs, tool-call records, execution status, error messages, and user feedback. De-identification is mandatory to prevent sensitive-data leakage.

    2. Sample cleaning and anomaly filtering
      Not every session is suitable for evaluation. Filter out test traffic, bot requests, abnormal traffic inflation, duplicate invalid sessions, and clearly leading inputs to prevent metric contamination.

    3. Signal extraction and labeling
      Use a combination of rules, classification models, and human sampling to convert raw sessions into structured signals. For high-value negative samples, further manual attribution can be performed.

    4. Model routing and control-group design
      If multiple models or versions are online simultaneously, design a sound routing strategy. Randomized assignment is ideal; if constrained by business needs, use stratified sampling with statistical correction.

    5. Metric calculation and significance testing
      Do not look at the mean alone when producing metrics; also examine confidence intervals, sample size, and trends. For multiple turns in one session or multiple requests from one user, use session-level bootstrapping or cluster-robust estimation to avoid overstating significance.

    6. Regression monitoring and alerting
      The evaluation system should be wired into continuous monitoring, with thresholds set for key metrics such as net improvement rate, execution success rate, correction success rate, complaint rate, and abandonment rate. When a metric fluctuates abnormally, trigger an alert and trace back the relevant session samples.

    10. End-to-End Case: Evaluating an Error and Correction

    Suppose a user asks the agent to generate a log-aggregation script. The session proceeds as follows:

    1. The user makes a request: aggregate logs by hour and generate a command.
    2. The agent generates the command and attempts to execute it.
    3. The tool returns an error: insufficient permissions.
    4. The user hints: switch to a read-only path.
    5. The agent adjusts the parameters and re-runs.
    6. Execution succeeds, and the user replies “that works.”

    Signal-extraction example:

    • explicit_confirm = True
    • praise = False
    • complaint = False
    • correction_count = 1
    • repeated_correction_count = 0
    • execution_error_count = 1
    • repeated_execution_error_count = 0
    • abandon = False

    This session shows one execution failure, but the agent successfully understood the error and completed the correction, with the user ultimately confirming. Therefore, the whole turn should not be judged a failure simply because one command error occurred. A more reasonable approach is to record one execution error while also granting a positive signal for a successful correction.

    11. From Evaluation to Iteration: Closed-Loop Optimization

    The ultimate purpose of real-conversation evaluation is not to produce rankings, but to drive product improvement. Evaluation results should feed back into R&D and operations.

    First, classify failed sessions. Common categories include intent misunderstanding, missed requirement constraints, wrong tool selection, parameter-generation errors, insufficient error-recovery ability, insufficient long-context consistency, and improper handling of safety boundaries.

    Second, distill high-frequency failure samples into a regression test set. Whenever the model or prompt strategy is updated, prioritize verifying whether these historical failure cases have been fixed and whether new regressions have been introduced.

    Third, adopt canary releases and online experiments. Before a new version launches, run a comparison experiment on a small traffic slice to observe whether net improvement rate, correction success rate, and execution reliability have improved, then gradually expand traffic.

    Finally, establish a user-feedback loop. Regularly review user complaints, correction instructions, and failed sessions. If a particular error keeps recurring, it signals that the current model capability or product interaction design still has room for improvement.

    12. Risk Governance: Compliance, Credential Scrubbing, and Data Governance

    Real-conversation evaluation depends on large volumes of user-interaction data, so data governance and security compliance are critical.

    1. Privacy protection and informed consent
      Sessions may contain user personal information, business data, or sensitive content. Using real conversations for evaluation should clearly state the purpose, scope, and evaluation intent through the privacy policy, terms of service, or product notice, and should follow the principle of data minimization. Define retention periods, access permissions, deletion mechanisms, and user opt-out options clearly, to avoid secondary use of session data that could trigger complaints or compliance risk.

    2. Credential and sensitive-information scrubbing
      In the collection and de-identification stage, you must automatically detect and scrub credentials such as API keys, tokens, passwords, connection strings, and access tokens, as well as personal information such as phone numbers, national ID numbers, emails, and addresses. For command-execution logs, additionally prevent secrets from entering the evaluation set in plaintext.

    3. Feedback noise
      User feedback may be influenced by emotion, interface experience, or wait time, and cannot all be attributed to model capability. Combine behavior signals with human sampling review.

    4. Sample bias
      Different user groups and task difficulties can distort evaluation results; correct for this through stratification, weighting, propensity scoring, or cluster estimation.

    5. Gaming and adversarial behavior
      If evaluation results affect resource allocation or business decisions, there may be abnormal score inflation, induced praise, or malicious complaints. Build anti-cheating mechanisms such as abnormal-traffic detection, behavior-consistency checks, and sampling audits.

    6. Metric explainability
      Teams need to be able to explain the reasons behind metric changes; otherwise results cannot support decisions. Retain metric definitions, sample scope, confidence intervals, and anomaly-traceback capability.

    7. Human review mechanism
      For complex cases that automated evaluation cannot judge, retain sampling-based human review to improve result credibility.

    13. Enterprise Implementation: A Three-Layer Evaluation Architecture

    For enterprises that want to build agent evaluation capability in their own products, a three-layer architecture works well.

    1. Observation layer
      The observation layer records sessions, tool calls, execution results, and user behavior. It can be built on general cloud capabilities such as logging services, monitoring and alerting, object storage, and message queues to construct the data pipeline. The focus is on ensuring data is complete, traceable, and queryable, while meeting de-identification and retention policies.

    2. Evaluation layer
      The evaluation layer handles signal extraction, metric calculation, model comparison, and experiment analysis. This layer should align evaluation metrics with business goals, such as task completion rate, user complaint rate, average correction rounds, execution success rate, and abandonment rate.

    3. Governance layer
      The governance layer handles access control, de-identification policies, credential scrubbing, anomaly detection, model version management, and evaluation auditing. This layer ensures the evaluation system runs on a safe, compliant, and trustworthy basis over the long term.

    14. FAQ

    Q: How do you evaluate agents?

    A: Start from real conversations. First build session collection and de-identification capability, then extract signals such as explicit confirmation, correction instructions, execution errors, repeated follow-ups, and abandonment. Next, define result, process, correction, and reliability metrics, and judge whether a version is effective by combining a baseline with confidence intervals.

    Q: What should you watch out for in real-conversation evaluation?

    A: Focus on four points: first, distinguish normal endings from abnormal abandonment; second, avoid mistaking task difficulty for model capability; third, preserve the original signal definitions and calculation scripts; and fourth, properly implement privacy de-identification, credential scrubbing, and retention/deletion mechanisms.

    Q: How do you avoid misjudgment from static leaderboards?

    A: Static leaderboards are good for coarse screening, but should not directly decide the launch of a complex agent. Use real-conversation evaluation to supplement metrics such as task completion rate, correction success rate, execution reliability, and abandonment rate, and control selection bias through stratification, randomization, or propensity scoring.

    15. Conclusion

    The competitive focus of agents is shifting from “single-response quality” to “long-term task execution ability.” An excellent evaluation system cannot only compare whether a model speaks well; it must answer whether the model is reliable, whether it is controllable, and whether it can consistently help users complete tasks in real environments.

    By building an evaluation system from real user conversations—using behavior signals, causal correction, and metrics such as the net improvement rate—teams can more accurately understand the boundaries of model capability. For developers and enterprises, this is not just an evaluation method, but critical infrastructure for the continuous iteration, quality assurance, and business growth of agent products.