Evaluating AI Agents on Real-World Tasks: Causal Tracing and Net Improvement Rate

1. Why Agent Evaluation Needs a New Approach

Traditional model evaluation primarily relies on two methods: static benchmark datasets and human or user preference voting on paired responses. While effective in the era of conversational models, these approaches show significant limitations in Agent scenarios.

The core characteristics of agent tasks are long-term execution, multi-step processes, and high interactivity. A single task might span dozens of dialogue turns, involving file read/write operations, command execution, environment exploration, and repeated trial and error. In such scenarios, voting-based judgments on “which response is better” largely fail. Users rarely have the opportunity to view the complete execution trajectories of two models simultaneously, and even if they do, it is difficult to provide a reliable preference regarding the overall quality of dozens of operational steps. More importantly, voting reflects subjective feelings, whereas the success or failure of agent tasks is determined by objective facts: whether commands execute successfully, whether users repeatedly correct the same errors, and whether the task is explicitly confirmed as complete.

Therefore, a new evaluation paradigm is taking shape: instead of interrupting users or relying on votes, it automatically mines behavioral signals from real-world usage processes and converts them into comparable capability metrics using statistical inference methods. This article explores this approach by breaking down its signal system, inference methods, and metric interpretation, and discusses its practical significance for agent research and development.

2. Signal System: Extracting Reliable Evidence from Natural Conversations

Real-world conversations contain abundant clues about “how well a model performs.” The key lies in how to define and extract them. Usable signals can be categorized into four types:

1. Explicit Confirmation Signals
Users proactively indicate task completion or that results meet expectations, such as confirming that a specific output is usable or a problem is solved. This is the strongest positive signal because it represents the actual achievement of the task goal from the user’s perspective.

2. Praise and Complaint Signals
Users express satisfaction or dissatisfaction with intermediate results. Praise usually points to the high quality of a specific step, while complaints often point to comprehension deviations, missed constraints, or incorrect decisions. These signals are more fine-grained than confirmation signals but also noisier, requiring contextual judgment.

3. Steering/Correction Signals
Users request the model to change direction, correct its approach, or re-execute, such as pointing out “not this file” or “you should check dependencies before installing.” Corrections are highly valuable information in agent scenarios: they directly expose the model’s performance in steerability—whether it can accurately understand correction intents and execute them stably, rather than repeatedly making mistakes or drifting from the goal.

4. Execution Error Signals
In agents capable of command execution, non-zero status codes from Shell commands, script errors, and non-existent paths are all objective errors. These signals do not rely on subjective user expressions; they are hard evidence that can be automatically collected, directly reflecting command execution reliability.

The combination of these four signals covers multiple levels, from “whether the task is completed” to “whether the process is robust.” It is important to emphasize that no single signal is sufficient to evaluate a model: a model might receive few complaints simply because users gave up early; or it might have a low command error rate but an equally low task completion rate. Therefore, signals must be aggregated into a unified metric with bias correction.

3. Causal Tracing: Estimating Net Improvement Rate via Randomized Inference

Once the signals are collected, the next question is: how do we convert them into a measure of “how much better this model is than the average”?

Directly calculating the signal ratios of different models encounters severe confounding factors. Sessions from different models come from distinct user groups, task types, and time windows. User expression habits, patience levels, and task difficulties are constantly changing. If raw proportions are compared directly, the conclusions will likely be contaminated by these factors.

The core practice of causal tracing introduces randomization: introducing controllable randomness during session allocation or signal attribution to break the association between models and tasks, users, and time periods. This isolates “model capability” from confounding factors. Based on this, through controlled experiment-style inference, we can estimate the Net Improvement Rate (NIR) of each model relative to the overall average level.

In a simplified form, the Net Improvement Rate for model m can be expressed as:

NIR(m) = Observed Signal Score(m) − Expected Signal Score(Average Level)
          ↑ Unbiased estimate after randomization correction

The “expected score” here is not a simple mean, but a counterfactual estimate controlling for task and user distribution under a randomized design. The values obtained this way are comparable across models: a positive value indicates performance above the average, a negative value indicates below average, and the magnitude reflects the scale of improvement.

Key points of this method include:

• Randomization must be genuinely effective, otherwise the correction is merely formal.
• Signals need to be weighted. Confirmations and execution errors obviously carry different weights. Weight settings should be based on the actual impact on task success or failure.
• The sample size must be sufficient. Long-term tasks inherently have high variance. Under small samples, the NIR fluctuates significantly, requiring confidence intervals rather than single-point estimates.

4. Metric Interpretation: From Overall Scores to Granular Capabilities

Aggregated metrics provide an overall ranking, but in engineering practice, teams care more about “where the gap lies.” Therefore, evaluations typically break down into granular dimensions, such as:

• Command Execution Reliability: First-time success rate of Shell commands, error recovery capabilities, and correct environment exploration.
• Steerability: Whether the model can stably converge to the correct direction after user correction, and whether the same error repeats.
• Task Completion: The proportion of explicit confirmation signals and the convergence of long-term tasks.
• Collaboration Efficiency: The number of turns and tool calls required to reach confirmation.

Consider a specific scenario to illustrate the value of granular metrics. Suppose an open-weight model achieves a comprehensive NIR of +5.1%, a 4.2 percentage point increase from the +0.9% of the previous generation in the same series. Simultaneously, it ranks among the top open-weight models in the command execution reliability and steerability dimensions. The implication of this data set is not just a “rise in ranking,” but it indicates that the improvement primarily stems from execution-level robustness and correction response capability, rather than pure conversational quality. For the R&D team, this directly points to optimization directions: strengthening error handling and retry strategies for tool calls, improving the parsing of correction intents, and maintaining clearer state tracking during long tasks.

It is worth reminding that rankings must be interpreted in the context of the sample background. If an evaluation is based on over 17,500 real long-term sessions from global users, its statistical power is strong, but attention must still be paid to whether the task distribution leans toward specific scenarios. If the evaluation focuses on work, coding, and conversational tasks, the conclusions primarily apply to these scenarios, and cross-domain generalization should be extrapolated with caution.

5. Engineering Implementation: Building a Lightweight Internal Evaluation

For most teams, replicating a comprehensive leaderboard is unnecessary, but the methodology can be adopted to build an internal evaluation system. The following steps are recommended:

Step 1: Define Signal Patterns. Annotate the four types of events—confirmation, complaint, correction, and execution error—within session logs. This can be implemented using rule matching combined with a lightweight classification model:

def classify_signal(turn, prev_context):
    if is_explicit_confirmation(turn):
        return "confirm"
    if is_complaint(turn):
        return "complaint"
    if is_correction(turn, prev_context):
        return "steer"
    if has_command_failure(turn):
        return "exec_error"
    return None

Step 2: Design Randomization. If there are multiple candidate models internally, the same task can be assigned to different models randomly, recording the full trajectory. If there is only one model, random stratified sampling can be performed on similar tasks to control the task difficulty distribution.

Step 3: Calculate NIR and Provide Intervals. Calculate the weighted signal score for each model, subtract the stratified expected score, and use bootstrap sampling to estimate the confidence interval.

Step 4: Break Down by Dimensions and Review. Conduct manual spot checks on task types where negative signals are concentrated to determine whether they are comprehension errors, tool invocation errors, or state management errors.

Step 5: Closed-loop Validation. Translate evaluation conclusions into specific changes (such as retry logic, correction parsing, context compression strategies), then run another round of evaluation to verify changes in the Net Improvement Rate.

6. Method Boundaries and Considerations

Any evaluation method has applicable boundaries, and users should remain clear-headed when applying them:

• Signals do not equal ground truth. A lack of user complaints does not mean the task succeeded; the user might have just given up. An explicit confirmation might also stem from a low tolerance standard. Signals are proxy metrics and need to be calibrated with manual spot checks.
• Randomization is costly. Randomizing real traffic might affect user experience and should be carefully implemented within a canary release range, with a rollback mechanism in place.
• Long-tail tasks are masked by averages. The aggregated NIR will dilute severe failures on a few highly difficult tasks. Granular dimensions and a failure case library are equally important.
• Cross-version comparisons require caution. Task distributions and user compositions drift over time. Cross-period comparisons should use the same time window or apply distribution correction.

7. Conclusion

The value of evaluating real-world agent tasks lies not in producing a ranking number, but in establishing a reliable pipeline from natural behavior to comparable metrics: using multi-dimensional signals to characterize process quality, randomized inference to strip away confounding factors, the Net Improvement Rate to provide a quantitative expression of relative performance, and granular dimensions to guide engineering optimization. For agent R&D, this method is both an evaluation tool and a requirements specification—it tells the team what users truly care about: whether tasks can be completed stably, whether corrections can be accurately executed, and whether commands can run reliably. Solidifying these dimensions holds more long-term significance than competing for a top spot on a single leaderboard.