AI Agent Evaluation: Causal Tracing Methods and Net Improvement Rate

    This article explains how to evaluate AI agents in real-world sessions rather than relying only on static benchmarks or subjective votes. It covers session-based evaluation design, automatic extraction of behavioral signals, session scoring, causal inference methods, net improvement rate calculation, sub-dimensions such as shell reliability and steerability, platform engineering practices, and how to interpret agent leaderboards.

    1. Introduction: Why Agent Evaluation Is Difficult

    Agents differ from traditional conversational models because they must complete real tasks through persistent, multi-step interactions: understanding goals, breaking down steps, calling tools, and adjusting direction after errors. This long-chain, multi-turn nature makes agent evaluation much more complex than single-turn question answering.

    Common evaluation approaches each have trade-offs:

    • Static benchmarks: Fixed tasks and reproducible results, but they face risks of data contamination and test-oriented overfitting, and they often struggle to cover the long-tail distribution of real tasks.
    • Preference-based voting: Users compare two responses and choose the preferred one. It is easy to implement, but it mainly reflects subjective preference. It cannot reliably answer whether a task was truly completed, nor can it capture errors and recovery during multi-turn execution.
    • Human-annotation evaluation: High quality, but expensive and slow, making it difficult to update continuously as models iterate rapidly.

    A more convincing direction is therefore to assign evaluated models randomly within real user sessions, automatically extract key signals closely related to task success, and use causal inference methods to estimate each model’s net improvement relative to the average. This article systematically explains the design and interpretation of such an evaluation system.

    2. Three Design Principles for Agent Evaluation

    2.1 Authenticity: Evaluation from real tasks

    Evaluation data can come directly from real user agent sessions. Task types, difficulty distribution, and user behavior are close to production conditions, which helps reduce the risk of fixed test sets being “gamed.” However, real sessions often span multiple turns or even multiple working days. Therefore, evaluation and analysis should use the session as the basic unit, not a single question-answer pair.

    2.2 Objectivity: Replace subjective voting with verifiable signals

    The system can automatically capture the following signals during sessions:

    • Explicit user confirmation that a task is complete;
    • Praise or complaints;
    • Correction instructions, where the user asks the agent to change its approach;
    • Tool-call failures such as shell command execution errors.

    These signals are directly related to task success or user experience. When evaluating a candidate model, the key question is whether it leads to fewer corrections, fewer complaints, and clearer confirmations of task completion.

    2.3 Causality: Randomization removes confounding factors

    If models are routed according to a fixed policy in real traffic—for example, based on task type, user level, or system load—observed performance differences may be caused by those factors rather than by the models’ intrinsic capabilities. A safer approach is:

    • Randomly decide which candidate model handles a session when the session is created;
    • Let that model handle all subsequent interactions within the session.

    Randomization ensures that the tasks and user distributions faced by each model are statistically comparable, so differences can be attributed to the models themselves. This premise holds only when all models are served fairly. If candidate models differ in queueing latency, context length, or tool permissions, the measured results will reflect more than model capability.

    3. Key Signal Extraction: Turning User Behavior into Computable Labels

    The quality of signal extraction directly determines the credibility of the evaluation. Signals can be processed in three main categories:

    Signal type Detection basis Role in scoring
    Task confirmation User explicitly says things like “that works,” “it’s solved,” or “that’s what I meant” Positive; core success signal
    Praise Positive feedback such as “great” or “very convenient” Positive; reflects improved experience
    Complaint or correction Expressing dissatisfaction or asking the agent to change its approach Negative; reflects communication cost
    Shell execution failure Tool call returns a non-zero exit code, times out, command not found, etc. Negative; reflects execution reliability

    3.1 Delayed feedback and confirmation

    User confirmation often lags behind the conversation itself. Some confirmations appear only after the task has actually been completed, such as when a build passes, tests succeed, or a deployment goes live. Therefore, each session should have an observation window, and additional feedback after the session ends should also be included in the analysis.

    3.2 Ambiguity and misclassification control

    Natural-language confirmation and negation can be ambiguous. A phrase like “okay” may mean approval or reluctant closure, while “fine, I guess” may imply dissatisfaction. In practice, a combination of rule matching, semantic classification, and confidence thresholds is commonly used. Low-confidence samples should be manually reviewed, and the extractor’s precision and recall should be evaluated periodically. For cases where multiple users share the same session, such as in customer-service scenarios, feedback should be attributed to the user who actually issued the task.

    3.3 Session scoring function

    Based on these signals, a composite score can be defined for each session. Confirmations and praise can be treated as positive signals, while complaints, corrections, and shell failures can be treated as negative signals. Each item should be capped to prevent a single extreme session from dominating the result:

    1. # The following code is illustrative and only shows the calculation logic.
    2. def session_score(signals, w):
    3. return (w["confirmed"] * signals["confirmed"]
    4. + w["praised"] * min(signals["praised"], 3)
    5. - w["complaint"] * min(signals["complaint"], 3)
    6. - w["corrected"] * min(signals["corrected"], 5)
    7. - w["cmd_fail"] * signals["cmd_fail_rate"])
    8. def evaluate_model(sessions, model_id, w):
    9. """Estimate the net improvement rate of a model using all sessions as the baseline."""
    10. treated = [s for s in sessions if s.model_id == model_id]
    11. control = [s for s in sessions if s.model_id != model_id]
    12. return (mean(session_score(s, w) for s in treated)
    13. - mean(session_score(s, w) for s in control))

    The weights should be set according to business goals. For example, online delivery scenarios may increase the weights of shell failures and corrections. Weights should remain consistent across multiple evaluations to keep results comparable over time.

    4. Causal Tracing: Using Randomization to Infer Model Contribution

    4.1 Why causal inference is needed

    Random assignment removes systematic confounding at the user and task levels, but natural variation at the session level still exists. Tasks differ in difficulty, users differ in how clearly they express requirements, and long-running tasks are more likely to encounter random fluctuations. A causal inference framework is therefore needed. This is the role of causal-tracing methods: they do not trace the source of a piece of information, but instead use experimental design to trace the actual impact of model replacement on task outcomes.

    4.2 Overall approach

    1. Session-level random assignment: Treat “handled by the target model” as a random treatment. Sessions handled by the target model form the treatment group, while other sessions form the control group.
    2. Stratified comparison: Stratify by task type such as work, coding, or chat, by difficulty, or by user type. Within each stratum, calculate the difference in average scores between target-model sessions and non-target-model sessions.
    3. Weighted aggregation: Use each stratum’s sample proportion as the weight, aggregate the stratum-level differences, and obtain the target model’s net improvement relative to the overall average.
    4. Uncertainty quantification: Use bootstrap resampling to estimate confidence intervals and determine whether the difference is statistically robust.

    The net improvement rate answers the question: under comparable conditions, how much has a model improved task outcomes relative to the average? For example, if a model’s overall net improvement rate is +5.1%, while the previous generation was +0.9%, this indicates an improvement of about 4.2 percentage points between generations.

    4.3 Illustrative implementation

    1. def stratified_delta(sessions, model_id, w, strata_fn):
    2. """Calculate the score difference between the target model and control by strata."""
    3. treated, control, deltas = [], [], []
    4. for stratum, group in group_by(sessions, strata_fn):
    5. t = [s for s in group if s.model_id == model_id]
    6. c = [s for s in group if s.model_id != model_id]
    7. if not t or not c:
    8. continue # Skip this stratum due to insufficient samples.
    9. treated += t
    10. control += c
    11. deltas.append((len(group),
    12. mean(session_score(s, w) for s in t)
    13. - mean(session_score(s, w) for s in c)))
    14. return treated, control, deltas
    15. def bootstrap_delta(sessions, model_id, w, strata_fn,
    16. n_boot=1000, alpha=0.05):
    17. """Estimate the point estimate and confidence interval of the net improvement rate."""
    18. treated, control, deltas = stratified_delta(sessions, model_id, w, strata_fn)
    19. total = sum(n for n, _ in deltas)
    20. point = sum(n * d for n, d in deltas) / total
    21. samples = []
    22. for _ in range(n_boot):
    23. # Perform resampling with replacement for the treatment and control groups separately.
    24. t_boot = resample(treated)
    25. c_boot = resample(control)
    26. boot = [s for s in t_boot + c_boot]
    27. _, _, d = stratified_delta(boot, model_id, w, strata_fn)
    28. samples.append(sum(n * v for n, v in d) / sum(n for n, _ in d))
    29. samples.sort()
    30. lo = samples[int(n_boot * alpha / 2)]
    31. hi = samples[int(n_boot * (1 - alpha / 2))]
    32. return point, (lo, hi)

    In real systems, complex weighted estimation is often unnecessary. A common approach is to randomly assign a fixed proportion of traffic to candidate models and then calculate the net improvement rate using difference-in-means: the treatment-group mean minus the control-group mean. This is an unbiased estimator and a standard method for randomized experiments. The code above further illustrates the idea of stratification and interval estimation.

    4.4 Implementation considerations

    • Fairness: Candidate models should be consistent in context length, tool permissions, rate limits, and other service conditions.
    • Stratification and quotas: Task types can differ significantly in difficulty. Historical data should be used to determine the minimum sample size for each stratum to avoid excessive fluctuation in fine-grained metrics.
    • Robustness checks: After randomization, variables such as user type and task type should be roughly balanced across groups. This can be used to verify whether randomization is working as intended.
    • Time windows: When models are continuously updated on the serving side, long statistical periods may mix multiple versions. Overlapping time windows are recommended to align comparisons, and results should be annotated with the corresponding time range.

    5. Sub-dimensions: Shell Reliability and Steerability

    A composite score is useful for ranking models, but it is not sufficient for identifying weaknesses. Sub-dimensional metrics are therefore needed.

    5.1 Shell execution reliability

    In coding and operations tasks, agents rely heavily on command-line tools to perform builds, tests, deployments, and other operations. Shell execution reliability measures the correctness of the commands generated by the model: whether the commands can be executed properly and whether they ultimately help complete the task.

    It is useful to distinguish the following cases:

    • Model-side errors: Command not found, parameter spelling mistakes, syntax errors, insufficient permissions, and other errors that can be directly attributed to the model.
    • Environment-side errors: Missing dependencies, network issues, unavailable external services, and other problems that are not the model’s fault.

    In practice, one can track the total failure rate, the model-side failure rate, and the distribution of failure types. It is also useful to observe how a model recovers after consecutive failures. Excluding environment-related failures makes differences between models more indicative of their actual capabilities.

    5.2 Steerability

    Real tasks are rarely completed correctly on the first attempt, and users often need to provide corrections midway. Steerability measures whether an agent can quickly understand a new intent, adjust its approach, and return to the correct track after receiving a correction instruction. It can be measured from three angles:

    • Recovery turns: The number of interaction turns required from the user’s correction instruction to confirmed task completion;
    • Success rate after correction: Among sessions with correction instructions, the proportion that eventually receive user confirmation;
    • Repeated correction rate: The proportion of sessions in which the user issues correction instructions multiple times.
    1. def steerability_metrics(sessions):
    2. corrected = [s for s in sessions if s.signals["corrected"] > 0]
    3. if not corrected:
    4. return {}
    5. resolved = [s for s in corrected if s.signals["confirmed"] > 0]
    6. return {
    7. "resolution_rate": len(resolved) / len(corrected), # Success rate after correction
    8. "median_recovery_turns": median(s.recovery_turns for s in corrected), # Median number of turns to converge
    9. "repeat_rate": sum(s.signals["corrected"] > 1 for s in corrected) / len(corrected), # Repeated correction rate
    10. }

    Steerability is especially important for long-running tasks. If users are willing to continue giving correction instructions, it means the task still has a chance of completion. Repeated corrections, however, often indicate that the agent has not truly understood the user’s intent.

    5.3 Domain-level and process metrics

    Public leaderboards often rank models by task domain. This is essentially a form of stratified statistics:

    • Work tasks: Document writing, information retrieval and organization, workflow automation, and similar tasks;
    • Coding tasks: Code writing, debugging, build and deployment tasks, and similar tasks;
    • Chat tasks: Open-ended question answering, consultation, advice, and similar tasks.

    Domain-level rankings can reveal a model’s capability profile. For example, a model may show strong shell reliability in coding tasks but only average performance in chat tasks. In that case, model selection should be based on the actual business scenario.

    In addition to user-feedback metrics, process metrics can also be introduced based on session trajectories:

    • Tool-call success rate: Number of successful tool calls divided by total tool calls;
    • Invalid step ratio: Number of rolled-back, redone, or undone steps divided by total steps;
    • Retries and turns to completion: Used to measure execution efficiency.
    1. def process_metrics(trajectory):
    2. """Calculate process-level metrics based on a session trajectory (illustrative)."""
    3. calls = trajectory.tool_calls
    4. success = sum(c["succeeded"] for c in calls)
    5. wasted = sum(step["status"] in ("rolled_back", "undone")
    6. for step in trajectory.steps)
    7. return {
    8. "tool_success_rate": success / len(calls) if calls else None,
    9. "waste_ratio": wasted / len(trajectory.steps) if trajectory.steps else None,
    10. }

    6. Engineering Implementation: Building a Session-Driven Evaluation Platform

    To put these methods into practice, a continuously running evaluation platform is needed. A typical data flow is:

    Session collection → signal extraction → net improvement rate calculation → visualization dashboard and alerts

    Stage Key work Common cloud infrastructure categories
    Session collection Record prompts, tool calls, and user feedback after desensitization; obtain user authorization before collection Logging services, object storage
    Signal extraction Asynchronously process session data in batches or streams; generate structured labels and scores Message queues, container platforms
    Estimation and display Calculate net improvement rate and confidence intervals; maintain leaderboards and trend dashboards Monitoring and alerting, data warehouses
    Quality control Manual sampling, extractor version management, and retraining Object storage, container platforms

    Several engineering considerations are important:

    • Sampling and privacy: Real sessions may contain sensitive information. Only collect fields necessary for evaluation, apply desensitization and access auditing, and comply with relevant laws, regulations, and platform privacy policies.
    • Cold start: When feedback is sparse in the early stage, objective signals such as shell failures can be prioritized, supplemented by small-scale manual annotation.
    • Asynchronous updates: Leaderboards can first be aggregated daily and then updated in a rolling manner, balancing timeliness and statistical stability.
    • Cost control: Set a budget for evaluation traffic and dynamically decide the sampling volume based on confidence-interval width, avoiding high inference costs for very small differences.

    7. How to Correctly Interpret Leaderboards

    • Check time and scale: Prioritize leaderboards with recent statistical periods and large numbers of evaluated sessions. When the sample size is too small, confidence intervals are wide, and top rankings may have limited significance.
    • Check sub-dimensions: Beyond the composite score, pay attention to shell reliability, steerability, and rankings in target task domains.
    • Look at intervals, not just point estimates: When adjacent rankings have substantially overlapping confidence intervals, the actual difference may not be significant.
    • Check task profiles: If the evaluation sessions are mostly coding tasks, the conclusions should not be directly applied to scenarios such as long-document writing.
    • Build your own validation: Teams can run small-percentage randomized canary tests on their own business traffic, re-test using the same session scoring and net improvement rate methods, and then make selection decisions.

    8. Conclusion

    Session-driven agent evaluation shifts the focus from “how good the answer is” to “whether the task is truly completed.” Its core ideas can be summarized as follows:

    1. Automatically extract key signals such as confirmation, praise, complaints, corrections, and shell failures from real sessions;
    2. Use causal-tracing and other randomized inference methods to estimate each model’s net improvement rate relative to the average;
    3. Combine sub-dimensional metrics such as shell reliability and steerability to form a complete capability profile of models.

    For R&D teams, this approach is also suitable for internal model selection and iteration validation. When building an evaluation system, teams can use general public-cloud infrastructure such as logging services, object storage, message queues, container platforms, data warehouses, and monitoring and alerting to build a sustainable evaluation pipeline. For example, application development teams using Baidu AI Cloud can compose these capabilities as needed to form an evaluation loop that fits their own business.