Evaluating AI Agents: Quantifying Real Task Capabilities with Causal Inference

1. The Core Dilemma in AI Agent Evaluation

Traditional large language model (LLM) evaluations typically use a “double-answer comparison + user voting” approach. The platform displays answers from two models simultaneously, and users decide which is better. While this method has some reference value for conversational models, directly applying it to agent task evaluation exposes significant flaws.

First, the completion process for agent tasks is much longer than a single-turn response. A real-world agent task might span dozens or even hundreds of interaction rounds, involving tool calls, command execution, file read/write operations, and error recovery. Simply comparing two “final answers” fails to reflect capability differences during the process.

Second, evaluation criteria for agent tasks are ambiguous. Conversational tasks can be subjectively judged by fluency and relevance, but whether an agent task is “truly completed” must be determined by objective signals, such as explicit user confirmation, correction commands, or successful Shell command execution.

Third, user voting introduces systematic bias. Users have vastly different tolerance levels for “usability.” Some users give high scores simply because the model “seems to understand the intent” rather than actually solving the problem. This makes voting-based leaderboards imprecise in distinguishing a model’s net capability in real tasks.

Therefore, agent evaluation systems must transition from “subjective comparison” to “objective signal extraction + causal effect estimation.” This is the core method discussed in this article.

2. Evaluation Signal Collection: Extracting Measurable Features from Real Sessions

To conduct reliable agent evaluations, we must first define “what data objectively reflects task execution quality.” From an engineering perspective, the extractable signals mainly fall into four categories.

2.1 Explicit Task Confirmation

When a user replies with “Done,” “Looks good,” “No problem,” or ends the session without adding further requests, it serves as a positive signal of task success. These signals directly correspond to user acceptance of the result and have high confidence.

However, positive confirmation alone is insufficient. The difficulty of tasks varies greatly; a simple file renaming task should not carry the same weight as a multi-step data analysis task. Therefore, the evaluation system needs to normalize confirmation signals using contextual factors such as task trajectory length, tool call frequency, and error occurrence rate.

2.2 Correction Instructions and User Dissatisfaction

Commands like “No, I wanted…”, “Start over,” or “You misunderstood” are strong signals that the model has deviated from the correct direction. Detecting these signals can be achieved through a combination of rule matching and semantic classification models. The rule layer captures high-frequency negative expressions, while the semantic classification layer identifies more subtle expressions of dissatisfaction, such as “This result isn’t quite what I expected.”

During quantification, correction instructions must be graded by severity. A mild correction indicates the model’s direction is generally correct but flawed in details; a severe correction indicates a fundamental deviation in understanding, requiring core steps to be re-executed. Different levels correspond to different capability penalty weights.

2.3 Shell Command Execution Errors

For agent tasks involving command-line operations, the execution result of Shell commands is one of the most objective signals. Non-zero exit codes, standard error outputs, or timeouts are all failure events that can be automatically collected.

The advantage of these signals lies in their complete objectivity and scalability for statistical analysis. A model’s Shell command success rate across a large volume of tasks directly reflects its reliability at the environment operation level. The evaluation system can further categorize errors into syntax errors, path errors, permission errors, and missing dependencies, thereby pinpointing the model’s specific weak points.

2.4 Completeness of the Task Execution Path

Beyond the final result, the completion path itself contains a wealth of capability information. For example: Did the model perform reasonable intermediate validations? Did it proactively retry after discovering an error? Did it ask the user for confirmation when uncertain rather than executing blindly?

These behavioral features can be extracted into structured features through trajectory analysis. For instance, a robust agent should check if a file exists after writing it, and check the exit code after executing a command. Missing these validation steps implies lower reliability, even if the final result happens to be correct.

3. Causal Tracing: Estimating Net Improvement Rate via Randomized Inference

Once signals are collected, the core question becomes: How do we estimate a model’s net capability performance relative to the average level from mixed observational data?

3.1 Why Direct Comparison is Insufficient

Directly calculating a model’s success rate across all sessions is heavily influenced by task assignment bias. For instance, one model might be assigned more simple tasks, while another handles complex ones, making their raw success rates incomparable. Even within the same task type, the difficulty of specific instances can vary significantly.

3.2 Causal Inference under Randomization

The evaluation system introduces a randomization mechanism when assigning tasks to models: tasks of the same category are randomly routed among multiple models. Randomization ensures that the task difficulty distribution assigned to different models converges in expectation, thereby severing the confounding relationship between “task difficulty” and “model selection.”

Under this setting, differences in model performance can be attributed to the models’ inherent capabilities rather than uneven task distribution. More specifically, the causal tracing method uses the “average level of all participating models” as a reference baseline to calculate each model’s net improvement rate relative to that benchmark.

3.3 Formalizing the Net Improvement Rate

The net improvement rate can be understood as the expected change in the probability of user task success when using a specific model compared to randomly selecting an average-level model, after controlling for the task difficulty distribution. A positive value indicates the model brings a measurable capability gain; a negative value indicates it is significantly inferior to the industry average.

The advantage of this formulation is that it avoids the incomparability of absolute scores. Between different model versions, the change in the net improvement rate directly indicates whether “capabilities have genuinely improved,” unaffected by evaluation batches or changes in the task pool.

4. Multi-Dimensional Evaluation Framework

A comprehensive agent evaluation should not merely provide a total score but decompose into actionable capability dimensions. Based on the aforementioned signal types, the following dimensions can be constructed:

Dimension	Corresponding Signal	Interpretation
Task Completion Rate	Explicit confirmation signals	Whether the user’s task is genuinely completed
Shell Reliability	Command exit codes, standard errors	Stability of environment operations
Controllability	Behavioral changes after correction instructions	Ability to accept user feedback and correct direction
Error Recovery	Retry and repair trajectories after errors	Robustness when facing failures
Process Rationality	Intermediate validation steps	Whether a robust operational strategy is adopted

An independent net improvement rate is calculated for each dimension and finally aggregated into a composite metric via weighting. Weights can be adjusted based on application scenarios. For example, in automated IT operations, Shell reliability should carry more weight than conversational fluency; in interactive data analysis, the weight of controllability should be increased.

This decomposition ensures the evaluation results answer not just “which is better” but “in what aspect it is better,” providing direct guidance for model selection and optimization.

5. Practical Implications for Long-Horizon Task Sessions

The sample size for agent evaluation is not a case of “bigger is better”; the key lies in the time span and task depth of the samples. Short-horizon tasks might be completed in a few dozen rounds, but long-horizon tasks (e.g., complex operations spanning hours across multiple environments) are necessary to fully expose capability differences in memory retention, context management, and long-term planning.

From a practical standpoint, the evaluation system should:

First, include a sufficient proportion of long-horizon tasks. In long-horizon tasks, models must maintain cross-round state consistency. Whether it forgets intermediate variables, repeats completed steps, or loses initial user constraints in a long context—these are issues unobservable in short-horizon tasks.

Second, record complete execution trajectories rather than just results. A model that produces correct results through a chaotic process will incur maintenance and audit costs in production environments. Trajectory data is crucial for post-mortem analysis and provides a foundation for subsequent capability optimization.

Third, apply the same signal standards to open-weight and closed-source models. Open-weight models allow deployers to conduct their own tests, but the evaluation system must ensure consistency in signal collection and scoring logic, otherwise comparisons become meaningless. The same task routing, signal extraction logic, and causal estimation methods are prerequisites for fairness.

6. Key Technical Implementation Points

If you intend to build similar evaluation capabilities within your own environment, focus on the following technical points:

Task Routing Randomization. The routing layer’s randomization must be decoupled from user and session identifiers to ensure stable model identity within a single session, preventing signal contamination caused by mid-session model switching. Additionally, randomization should be stratified—for instance, executing stratified random assignment based on task type and estimated difficulty layers to improve statistical efficiency.

Signal Extraction Latency and Consistency. Signal extraction must not impact the real-time nature of task execution. In practice, an asynchronous bypass approach can be adopted: the execution pipeline proceeds normally, while the signal collection module consumes trajectory data from the log stream to perform classification and aggregation in the background. This ensures user experience while obtaining complete evaluation data.

Computational Stability of Causal Estimation. When sample sizes for certain models are insufficient, the estimation variance of the net improvement rate increases significantly. In such cases, shrinkage estimation or Bayesian hierarchical models should be employed to shrink the performance of small-sample models toward the overall mean, preventing extreme values from misleading decisions.

A schematic implementation framework is as follows:

# Pseudocode: Estimating net improvement rate based on randomized task routing
def estimate_net_gain(sessions, model_id, baseline_mean):
    # Filter sessions for the target model
    model_sessions = [s for s in sessions 
                      if s.assigned_model == model_id]
    # Calculate task success signals
    success_rate = sum(s.task_confirmed for s in model_sessions) \
                   / len(model_sessions)
    # Under randomized conditions, directly compare the difference with the baseline
    net_gain = success_rate - baseline_mean
    # Optional: Apply shrinkage estimation to reduce small-sample variance
    shrinkage = 1.0 / (1.0 + var_estimate / len(model_sessions))
    adjusted_gain = net_gain * shrinkage
    return adjusted_gain

The core assumption of this logic is that tasks have been randomly assigned among models, so the difference in success rates can be directly treated as the causal effect of model capability. Actual systems must also handle complex scenarios like session truncation, user churn, and task difficulty covariates, but the overall approach remains the same.

7. Conclusion

The evaluation of AI agent capabilities is transitioning from “intuition-based” to “evidence-based.” Evaluation methods grounded in real task signals and leveraging randomization and causal inference can more robustly measure a model’s true gains in complex environments. For engineering teams, the focus should not rest on rank changes in a specific leaderboard, but rather on deeply understanding the signal definitions, statistical methods, and capability decomposition logic, integrating these concepts into their own model evaluation and selection workflows. Only evaluation conclusions built on objective signals and rigorous inference can genuinely guide the iterative direction of agent systems.