Causal Inference Methods for Evaluating AI Agents on Real-World Tasks

1. Shifting Evaluation Paradigms: From Preference Voting to Behavioral Signals

As AI agents transition from demonstration phases to production environments, the core contradiction in their evaluation is shifting. Early model evaluations mostly relied on human preference voting on paired responses. While this approach works for single-turn Q&A scenarios, it exposes significant flaws in long-horizon, multi-step agent tasks involving tool calls:

• Disconnect between voters and real users: Annotators often lack the business context of a task, making it difficult to judge whether a executed Shell command or a file modification actually solved the problem.
• Single-turn preferences fail to capture process quality: The value of agent tasks largely lies in intermediate steps—whether the agent correctly explored the environment, self-corrected after an error, or adhered to user constraints.
• High cost and long cycles of human evaluation: These methods struggle to support the continuous feedback required for rapid model iteration.

Consequently, a new evaluation approach is taking shape: mining quality signals directly from real user-agent sessions and estimating model capabilities using statistical inference methods. This type of evaluation does not require users to bear additional annotation costs; instead, it treats the “behavioral traces” left by users during natural interactions as implicit feedback. This article explores this paradigm, discussing its signal system, causal inference framework, metric design, and engineering implementation essentials.

2. Signal System: Key Evidence Extractable from Real Sessions

Real sessions contain signals that form the data foundation for evaluation. When designing a signal system, two conditions must be met: first, strong correlation with task success; second, automatic and stable extraction across most sessions. Common high-value signals include the following categories.

2.1 Explicit Confirmation and Positive Feedback

Explicit user confirmation of task results is the strongest positive signal, such as “Yes, that’s right,” “Okay, thanks,” or “This version is fine.” Such expressions typically appear at the task closing stage and can reliably mark a successful task loop. It is important to note that user silence does not equate to satisfaction; therefore, positive signals should only capture explicit expressions, avoiding misinterpreting “no complaints” as success.

2.2 Complaints and Negative Expressions

In contrast to confirmations, complaints, negations, and repeated requests for rework (“No,” “You got it wrong again,” “I meant the other file”) are strong negative signals. The value of these signals lies in their frequent appearance after multiple rounds of error correction, reflecting systematic deficiencies in the model’s understanding of intent or execution details.

2.3 Correction Commands

Correction commands fall between positive and negative but carry high information value. When a user provides directional corrections (“Don’t use recursion, change to iteration,” “Back up before deleting”), it indicates a deficiency in the model’s steerability, though the task may still be salvageable. Treating correction commands as a separate metric category helps distinguish between the capability profiles of “getting it right the first time” and “getting it right after correction.”

2.4 Tool Execution Errors

In agent scenarios equipped with command execution capabilities, Shell command failures, non-zero exit codes, permission errors, and non-existent paths are objective, easy-to-collect negative signals. These signals do not rely on subjective user expressions, feature low noise, and are highly reproducible, making them ideal as the primary source for execution reliability (e.g., Bash Reliability) metrics.

2.5 Task Completion and Turn Structure

Beyond the semantic signals above, the structural characteristics of a session also hold value: whether the task converged within a reasonable number of turns, whether there were repeated rollbacks, or whether the user abandoned the task midway. These features can be cross-validated with semantic signals, reducing the risk of misjudging a single signal.

In engineering implementation, signal extraction typically employs a combination of “rules + lightweight classifiers”: rules handle high-confidence pattern matching (such as command exit codes and explicit confirmation phrases), while classifiers categorize semantically ambiguous expressions. All extraction results should retain original snippets and confidence scores for subsequent manual spot-checking and metric calibration.

3. Causal Tracking: Estimating Net Improvement Rate via Randomized Inference

Signals alone are not enough; the key is deriving comparable estimates of model capabilities from them. Real sessions contain numerous confounding factors: task difficulty varies greatly among different users, usage habits of the same user differ across time periods, and the proportion of models routed to different user groups may be unbalanced. If we directly compare the raw success rates of various models, the conclusions are easily polluted by these factors.

The core concept of “causal tracking” is to transform evaluation into a randomized controlled experiment: randomly assigning models within real traffic (or leveraging existing randomized routing), and then comparing the differences of each model relative to the overall average level under the same task distribution. Its basic framework can be summarized as follows:

1. Define treatment and control: Treat “sessions involving a specific model” as the treatment group, and sessions involving other models during the same period as the control pool.
2. Stratification and covariate control: Stratify by task type (work, coding, chat), session length, and tool invocation density to reduce bias caused by task difficulty differences.
3. Aggregate signals into scores: Synthesize multiple signal types with weights into a utility score for a single session, with weights calibrated through offline annotated data.
4. Estimate net improvement rate: Calculate the difference in average utility between the treatment group and the control pool within each stratum, then weight and aggregate by traffic proportion to obtain the model’s net improvement rate relative to the average level.

The skeleton of this estimation process is expressed in pseudocode:

def net_improvement(sessions, model_id, strata):
    total_num, total_den = 0.0, 0.0
    for stratum in strata:
        treated = [s for s in sessions
                   if s.model == model_id and s.stratum == stratum]
        control = [s for s in sessions
                   if s.model != model_id and s.stratum == stratum]
        if not treated or not control:
            continue
        delta = mean_utility(treated) - mean_utility(control)
        weight = len(treated) / len(sessions_in(model_id))
        total_num += weight * delta
        total_den += weight
    return total_num / total_den if total_den else 0.0

A key property of this method is that the reference point for the net improvement rate is the “average level” rather than a fixed baseline. Therefore, when the set of evaluated models changes, the metric remains internally comparable. Simultaneously, randomization ensures that the treatment group and control pool are identically distributed in expectation, allowing the difference to be interpreted as the net effect brought by the model, rather than a projection of user group differences.

In practice, three points require attention: first, the choice of randomization unit (by session or by user) affects variance; for long-horizon tasks, session-level randomization supplemented with user-level clustered robust standard errors is recommended. Second, sample size requirements are high, as valid samples for long-horizon tasks are often far fewer than the total number of sessions, necessitating prior power analysis. Third, prevent intra-stratum imbalance caused by model routing strategies correlating with task types, introducing propensity score weighting if necessary.

4. Metric Design: Integrating Net Improvement Rate with Granular Capability Dimensions

A single composite score is insufficient to guide optimization; an evaluation system typically adopts a “one composite metric + multiple granular metrics” structure.

The composite net improvement rate measures the model’s relative performance across the overall task distribution, serving as the primary leaderboard metric. Granular metrics delineate specific capability dimensions, for example:

• Execution Reliability: Centered on tool invocation failure rates and command retry counts, reflecting the agent’s ability to stably complete tasks in real-world environments.
• Steerability: Centered on the frequency of correction commands and the task convergence rate post-correction, reflecting the model’s efficiency in understanding user intent modifications.
• Task Convergence Efficiency: Centered on the number of turns and tool invocations required to reach confirmation, reflecting the proportion of “getting it right the first time.”

The value of granular metrics lies in interpretability: when composite scores are close, granular dimensions reveal differences in the model’s capability structure, helping users select the right model for specific scenarios. Meanwhile, granular metrics must maintain a causal inference framework consistent with the composite metric, avoiding methodological inconsistencies where “the composite score uses randomization, but granular scores use raw means.”

5. Data Scale, Bias Control, and Implementation Essentials

The credibility of this evaluation heavily depends on data scale and quality control. Taking long-horizon agent tasks as an example, a single session might contain dozens of interaction turns and massive tool calls, meaning valid samples accumulate much slower than in short Q&A evaluations. Therefore, engineering efforts should focus on:

1. Sampling Strategy: Prioritize a balanced task type distribution to prevent high-frequency tasks from dominating the overall score; oversample low-frequency but high-value task types and restore weights during aggregation.
2. Signal Calibration: Regularly sample and manually review signals from each category to estimate precision and recall, adjusting weights accordingly to prevent systematic drift caused by extractor degradation.
3. Privacy and Compliance: Session data involves real user work content; the extraction process should occur post-desensitization. Signal storage should only retain necessary structured fields and short snippets, keeping original sessions based on the principle of minimum necessity.
4. Version Comparability: When evaluation criteria (signal definitions, stratification methods, weights) change, historical versions should be retained and breakpoints marked to avoid misreading direct cross-version comparisons.
5. Confidence Interval Disclosure: The net improvement rate should simultaneously provide confidence intervals and valid sample sizes, allowing readers to judge whether differences are significant rather than just looking at decimal rankings.

6. Conclusion

Agent evaluation is shifting from “human voting” to “behavioral inference.” An evaluation method utilizing real sessions as the data source, multiple behavioral signals as evidence, and randomized causal tracking as the inference framework can estimate the net improvement rate relative to the average level without drastically increasing annotation costs, while delineating capability structures through granular metrics. For developers, understanding the signal system and bias control essentials of this method not only aids in more accurately interpreting leaderboard results but also allows the application of the same approach to evaluate agent performance within their own business operations. After all, the evaluation data closest to real tasks always comes from the real tasks themselves.