Evaluating AI Agents on Real-World Tasks: Causal Tracing and Net Improvement Rate Explained

Agent capability assessment is shifting from “scoring dialogue quality” to “measuring real task completion.” Traditional evaluation often relies on human preference voting between two responses. While this works for short Q&A sessions, it becomes insufficient for long-horizon tasks spanning dozens of steps, file edits, and command executions. In such scenarios, voting signals are sparse, subjective, and disconnected from actual production experience. This article details an evaluation approach that uses real user sessions as the data source: automatically mining implicit feedback signals from user behavior and applying randomization-based causal inference to estimate each model’s net improvement rate relative to the average baseline, producing a ranking closer to real-world practice.

1. Why Preference Voting Fails for Agent Scenarios

Agent tasks exhibit three characteristics that make preference voting inadequate:

1. Long processes with delayed outcomes. A single task may involve dozens of tool calls, and success depends on the correctness of the final steps. Voters typically only see fragments.
2. Multiple independent evaluation dimensions. In the same task, instruction following, command execution reliability, and error-correction responsiveness are independent dimensions. A simple binary choice cannot distinguish between them.
3. Real user signals are richer. Users confirm results, express praise or complaints, and issue corrective instructions during sessions. These behaviors are themselves high-quality labels.

Therefore, evaluation should shift from “asking users to score” to “extracting signals from what users already do.”

2. Signal Extraction: Turning User Behavior into Computable Labels

The evaluation system analyzes long-horizon agent sessions from real users and automatically extracts several key signals. These fall into four categories:

• Explicit confirmation: User statements accepting the task outcome, indicating the task was judged complete.
• Praise and complaints: Explicit emotional expressions, stronger than ordinary confirmation, used for weighting positive and negative samples.
• Corrective instructions: User requests to change direction, redo steps, or point out errors, reflecting direction adjustment and correction capability.
• Command execution errors: Shell or tool invocations returning non-zero exit codes or abnormal output, directly reflecting execution reliability.

In practice, these signals are jointly validated using rules and lightweight models, then denoised to make them executable. Common denoising rules include:

• Environment error exclusion: If a Shell return code is non-zero and logs contain keywords like “permission denied” or “network unreachable,” mark it as an environment fault rather than a model error, and exclude it from the shell_error count.
• Speech slip threshold: For negative constructs like “no, that’s not right,” use a lightweight NLP classifier (e.g., rule-based modal verb matching) with a 0.7 confidence threshold. Signals below the threshold are excluded from correction signals.
• Duplicate signal merging: Similar signals within the same session (e.g., multiple confirmations) are merged into a single count within a 30-second window to avoid over-weighting a single session.

Signals are mapped into session-level or task-level scores, then aggregated at the model level. Specific weights should be tuned via grid search or Bayesian optimization on a small labeled sample set, rather than assigned arbitrarily.

# Example: session signal aggregation (pseudocode)
# Parameters: w1-w5 are weights fitted from labeled samples; signals are denoised signal objects
signals = extract_signals(session)   # confirmation/praise/complaint/correction/execution_error (denoised)
score = (
    w1 * signals.confirm
  + w2 * signals.praise
  - w3 * signals.complaint
  - w4 * signals.correction
  - w5 * signals.shell_error
)

3. Causal Tracing: Estimating Net Improvement Rate Relative to the Average

Raw scores alone are insufficient because task difficulty varies significantly, and directly comparing means is confounded by task distribution. To address this, a randomization-based causal inference method is introduced: on the same set of tasks corresponding to real sessions, each model’s performance is compared against the average of all participating models to estimate its net improvement rate—that is, the net positive improvement proportion the model brings relative to the average baseline.

The core idea is to treat “model identity” as the intervention variable and estimate the treatment effect while controlling for task difficulty distribution. Randomization ensures independence between tasks and models, thereby avoiding selection bias.

Randomization prerequisites and implementation: Independent assignment between tasks and models can be achieved through two routes: first, A/B split, where models are randomly assigned when users initiate tasks; second, post-hoc random relabeling, where permutation resampling is applied to model labels of already completed sessions. Both routes require stratification at the sampling stage to ensure even coverage of tasks across difficulty levels.

# Example: randomized estimation of net improvement rate
# Parameters: N_RESAMPLES should be >= 1000; the mean effect is used as the point estimate; the median can serve as a robustness check
effects = []
for _ in range(N_RESAMPLES):
    shuffled = randomize_model_labels(sessions)  # permute model labels
    effect = mean_score(shuffled, model) - mean_score(shuffled, all_models)
    effects.append(effect)
net_gain = mean(effects)               # mean effect as point estimate
robust_gain = percentile(effects, 50)  # median as robustness check

A positive net improvement rate indicates the model performs above average under the same task distribution; the larger the value, the stronger the advantage. Because randomization resampling is used, confidence intervals can be reported, avoiding ranking instability caused by small samples. The required sample size depends on the task distribution and statistical power requirements, and the measurement basis should be noted in the report.

4. Metric Dimensions: Reliability, Controllability, and Overall Performance

Evaluation is typically broken into multiple sub-dimensions to locate specific capability gaps. The table below shows the core formula and suggested weight for each dimension:

Dimension	Core Formula	Suggested Weight	Notes
Command execution reliability	Successful invocations / total invocations	0.40	Includes error recovery capability
Corrective responsiveness	Inverse of steps to convergence after correction	0.30	Faster convergence yields higher score
User sentiment positivity rate	(praise + confirmation) / (praise + confirmation + complaint + correction)	0.20	Reflects overall experience
Overall net improvement rate	Randomized mean effect	0.10	Used for final ranking

Ranking separately by task type (e.g., workflow processing, coding, daily conversation) avoids a single aggregate score masking real differences.

5. Practical Recommendations and Considerations

1. Prioritize signal quality over model quantity. It is better to reduce the number of evaluated models than to compromise signal extraction accuracy, otherwise ranking will be dominated by noise.
2. Control task distribution. Randomization inference requires that the task set is fair to every model, so stratification must be applied at the sampling stage.
3. Report confidence intervals. Point estimates of net improvement rate should be accompanied by intervals to avoid readers misinterpreting small differences.
4. Keep dimensions interpretable. Rankings should be accompanied by sub-metrics so developers can identify whether the issue lies in execution reliability or correction capability.
5. Continuously regress. Real session distributions drift as the product evolves; evaluation sets need periodic refreshment and baseline recomputation.
6. Compliance note. Signal extraction involves user behavior data and must comply with regulations such as GDPR and applicable data protection laws. Anonymization and authorization management should be implemented during collection and storage.

Conclusion

Core takeaways: First, real-session behavioral signals can replace subjective voting as a reliable label source for long-horizon agent evaluation. Second, randomization inference controls task distribution and provides reproducible net improvement rate point estimates with confidence intervals. Third, sub-dimension rankings guide targeted optimization better than a single aggregate score.

Using real sessions as the data source, behavioral signals as labels, and randomized inference as the statistical tool, this evaluation approach moves agent assessment from subjective voting toward reproducible effect estimation. For developers, understanding net improvement rate and sub-dimension meanings is more valuable than memorizing a specific rank—it guides targeted improvements in areas like command execution reliability and corrective responsiveness, enabling agents to deliver consistently in long-horizon real-world tasks.