Causal Inference for Evaluating AI Agents in Real-World Tasks

I. Why Traditional Evaluation Fails in Agent Scenarios

Evaluating large language models has long relied on static Q&A benchmarks and human preference voting. While these methods work reasonably well for single-turn Q&A, their limitations become glaringly obvious once applied to AI Agent scenarios.

The core characteristics of agent tasks are long horizons, multi-step processes, and heavy interaction. A complete session might involve a dozen stages: requirement clarification, solution planning, tool calling, command execution, error recovery, and result confirmation. In this context:

• Static benchmarks cannot cover real-world distributions: Pre-defined questions fail to replicate real user behaviors under conditions like ambiguous requirements, environment differences, or mid-task requirement changes.
• Human voting is costly and distorted: Asking evaluators to compare two long sessions line-by-line is expensive and highly susceptible to biases such as presentation order and length preference.
• Single-point answer correctness does not equal task completion: An agent might write incorrect commands or misunderstand constraints in intermediate steps, yet luckily produce a seemingly reasonable final conclusion. Traditional evaluation struggles to capture these issues.

Consequently, the industry is gradually shifting toward an approach closer to production environments: instead of manually constructing evaluation sets, extract signals directly from real user agent sessions and use statistical inference methods to estimate differences in model capabilities.

II. Core Concept: Mining Implicit Feedback from Real Sessions

When collaborating with agents, real users unconsciously generate a massive amount of high-quality feedback signals. These signals require no additional user annotation, yet they are more fine-grained and reliable than simple “thumbs up/down” ratings.

2.1 Key Extractable Signal Types

Common signals that can be automated for extraction include:

1. Explicit Confirmation: Users reply with phrases like “okay,” “that’s it,” or “go with this,” indicating task completion or solution acceptance.
2. Praise and Complaints: Explicit emotional expressions, such as “this approach is great” or “you messed up again.”
3. Corrective Instructions: Users request a change in direction, such as “No, I want incremental updates, not a full rebuild.”
4. Tool Execution Errors: Observable failures such as Shell commands returning non-zero exit codes, timeouts, or permission denials.
5. Repeated Inquiries: The same question being asked multiple times, often implying that previous answers did not solve the actual problem.
6. Premature Session Termination: Users abandoning the task midway, usually a negative signal, though it requires contextual judgment.

The commonality among these signals is: they are byproducts of the task execution process, not an additional cost. This allows evaluation to cover an extremely large sample size, far exceeding what human annotation can achieve.

2.2 Mapping Signals to Scores

Transforming signals into comparable scores requires solving two problems: directionality and weighting.

• Directionality: Confirmations and praise are positive; complaints, corrections, and errors are negative; repeated inquiries and session terminations need contextual judgment.
• Weighting: Different signals have different confidence levels. For example, an “explicit confirmation” is more reliable than a “silent ending,” and a “command error” is more objective than a “user complaint.”

A common practice is to define signed scores for each signal type, then perform a weighted aggregation at the session level to obtain a “net signal score” for that session. However, models cannot be directly compared at this stage—because the task difficulty, user habits, and environment complexity vary drastically across different sessions.

III. Causal Tracing: How Randomized Inference Eliminates Confounders

Directly comparing the average signal scores of different models is unfair, as the session distributions faced by the models may differ: some models might be assigned more simple tasks, while others handle high-difficulty scenarios. This is a classic confounding variable problem.

The core of the “causal tracing” method is to eliminate such biases through randomization and relative estimation.

3.1 The Role of Randomization

In a real-world product, user requests are randomly routed to different models (or randomly switched within the same session). This randomization ensures:

• Variables like task difficulty, user profiles, and time distribution are approximately balanced across models.
• The differences between models can be attributed closer to a “causal effect” rather than a mere “correlation.”

This aligns with the underlying logic of A/B testing, except the experimental unit shifts from “users” to “sessions” or “task segments.”

3.2 Net Lift Relative to the Average

Rather than estimating the absolute capability of each model, it is better to estimate its net lift relative to the overall average level. Formally, if the expected net signal score generated by a model under randomized conditions is $E[m]$, and the mean of all models is $E[\bar{m}]$, the net lift can be defined as:

net_lift(m) = E[m] - E[mean]

The benefits of this relative definition include:

• Eliminating the impact of overall task difficulty fluctuations.
• Allowing direct comparison of evaluation results across different batches and time periods.
• Being more robust to anomalous sessions, as extreme values are averaged out.

In practical implementation, techniques like stratified sampling and inverse probability weighting are often combined to further control covariates such as session length and task type.

IV. Metric System Design: From Overall Score to Fine-Grained Capabilities

A single overall score is insufficient to guide optimization. A usable evaluation system should simultaneously provide overall rankings and fine-grained dimensions.

4.1 Overall Net Lift

The overall score represents the weighted performance of all positive and negative signals, making it suitable as the primary metric for leaderboards. However, it can easily be dominated by high-frequency signals, so it must be interpreted alongside fine-grained metrics.

4.2 Typical Fine-Grained Dimensions

• Shell Command Execution Reliability: Tracks command success rate, retry counts, and error recovery capabilities. This is a critical capability for agents deployed in real environments.
• Direction Adjustment and Error Correction: Measures whether the model can quickly and accurately adjust its subsequent behavior upon receiving corrective instructions, rather than repeating mistakes.
• Multi-step Task Completion: Calculates the final confirmation rate on a per-task basis.
• Interaction Efficiency: The average number of turns required to reach a confirmation, reflecting whether the model “gets it right the first time.”

4.3 Engineering Implementation Details

# Pseudo-code: Session-level net signal score calculation
def session_score(session):
    score = 0.0
    for event in session.events:
        if event.type == "explicit_confirm":
            score += W_CONFIRM
        elif event.type == "praise":
            score += W_PRAISE
        elif event.type == "complaint":
            score -= W_COMPLAINT
        elif event.type == "correction":
            score -= W_CORRECTION
        elif event.type == "tool_error":
            score -= W_TOOL_ERROR
    return normalize(score, session.length)

Key engineering constraints:

1. Signal extraction must be explainable: Every score should be traceable to specific events, facilitating auditing and parameter tuning.
2. Weights require continuous calibration: The consistency between automated signals and human judgment can be validated using a small-scale human-annotated dataset.
3. Prevent metric gaming: If a model tends to induce user confirmations, this must be suppressed in the signal definition—for example, by down-weighting sessions with “confirmations without substantive execution.”

V. Practical Insights and Implementation Recommendations

For teams looking to build or utilize such an evaluation system, consider the following:

1. Prioritize randomization quality. Without reliable random routing, all subsequent inferences are untrustworthy. Ensure complete assignment logs are recorded at the routing layer.
2. Be conservative with signal definitions. It is better to miss weak signals than to introduce high-noise ones, which would dilute the signal-to-noise ratio of overall metrics.
3. Value both overall and fine-grained scores. The overall score is for horizontal comparison, while fine-grained scores locate issues. For instance, a model with an average overall score but high command reliability ranking is well-suited for automated operations scenarios.
4. Continuously monitor distribution drift. Real user task distributions change over time; the evaluation system requires periodic recalibration to prevent historical conclusions from becoming invalid.
5. Feed evaluation results back into training and products. Low-scoring items in fine-grained dimensions often indicate the most explicit direction for the next optimization cycle.

VI. Conclusion

Agent evaluation is transitioning from “constructed questions + human voting” to “real sessions + causal inference.” The core value of this shift lies in replacing expensive and easily distorted human evaluation with the feedback loop of the production environment itself. By eliminating confounders through randomized routing, ensuring comparability through relative net lift, and guiding optimization through fine-grained metrics, evaluation truly becomes the infrastructure for agent capability iteration. For developers and enterprise users, understanding this methodology also helps in rationally interpreting leaderboard numbers and translating them into selection and optimization criteria for their own scenarios.