Evaluating AI Agents in Real-World Tasks: A Causal Inference Approach

1. Why Traditional Evaluation Methods Fall Short

As AI Agents transition from lab demonstrations to production environments, evaluation becomes particularly challenging. Traditional evaluation paradigms generally fall into two categories: static benchmarks using fixed datasets for scoring, and human preference voting where users make A/B choices between responses. Both approaches have significant flaws.

The problem with static benchmarks is “overfitting.” Once a benchmark becomes an industry-standard metric, training data may skew toward it, resulting in high scores but mediocre real-world performance. More importantly, static queries struggle to cover the open-ended nature of agentic tasks. A real-world programming task might involve multi-turn file read/write operations, command executions, and error recovery—none of which can be simulated by a single-turn Q&A.

Human preference voting suffers from “sampling bias.” Users willing to participate in voting are often power users or specific demographic groups whose preferences do not represent the broader user base. Furthermore, voting scenarios typically involve short conversations, disconnecting them from the actual experience of long-horizon tasks. What users truly care about is not “which response is more pleasing,” but “which agent actually got the job done.”

Therefore, evaluation methods require a paradigm shift: moving from “letting users judge” to “automatically extracting signals from real-world usage.”

2. Signal Mining in Real Conversations

This new evaluation approach is built on a simple observation: when collaborating with an agent, users unconsciously leave behind a wealth of feedback. This feedback is not deliberate scoring, but a natural byproduct of the task progression process. These can be categorized into several key signals:

1. Explicit Confirmation Signals
When a user says “Yes, exactly,” “That works,” or “No problem,” it means the agent’s previous output was accepted. This is the strongest positive signal because it directly correlates with the achievement of the task’s objective.

2. Praise and Complaint Signals
Phrases like “Great approach” or “Why did you mess this up again?” are emotional expressions. While noisy, when combined with the task context, they can effectively distinguish between “done right” and “done wrong.”

3. Correction Instruction Signals
When a user says “No, I meant delete first, then rebuild” or “Don’t use that library, switch to another,” it indicates a deviation in the previous output. This is a clear negative signal that also reflects the agent’s “steerability.”

4. Command Execution Error Signals
In tasks involving terminal operations, command errors, timeouts, or permission denials executed by either the user or the agent are negative signals that can be automatically captured. These signals require no subjective user expression; they are purely objective facts.

By aggregating these signals across the conversation dimension, we can generate a “performance score” for each model on each task. The key takeaway is that these signals originate from real tasks rather than artificial queries, significantly enhancing their external validity.

3. Causal Tracing: Bridging Correlation and Causation

With the signals in place, the next question is: how do we fairly compare different models?

Directly comparing average scores is problematic because different models may face different task distributions—one model might simply be assigned easier tasks. A more subtle issue is endogeneity: a model’s output influences subsequent user behavior (e.g., a user is more willing to continue the conversation if the model answers well).

The core idea of the “causal tracing” method is randomized inference. Specifically:

• In real traffic, randomly assign tasks of the same category, allowing different models to take over the conversation with a certain probability.
• Record each model’s net improvement rate on each task (relative to the average performance of all models on that task).
• Use randomization to ensure that task assignment is independent of model capabilities, thereby isolating “model differences” from “task differences.”

Expressed as a formula, for task $t$ and model $m$, the net improvement rate is defined as:

NetImprovement(m, t) = Score(m, t) - Mean_over_models(Score(·, t))

By calculating a weighted average across all tasks, we obtain the global net improvement rate for model $m$. Because tasks are randomly assigned, this estimator is unbiased. Compared to simple average score comparisons, it effectively eliminates the confounding factor of task difficulty.

Another advantage of this method is its “relativity.” The net improvement rate is relative to the average level, so when a new model joins the evaluation, historical model scores do not need to be recalculated—as long as the change in the average baseline is correctly tracked. This makes the leaderboard sustainable over time.

4. Evaluation Dimensions: Beyond “Getting It Right”

Evaluating real-world agentic tasks cannot focus solely on the final result; it must also assess process quality. Several key sub-dimensions deserve attention:

Bash Reliability
In tasks involving command-line operations, the success rate of command execution, error recovery capabilities, and the avoidance of destructive operations are hard metrics. An agent that casually runs rm -rf or triggers timeouts is untrustworthy, even if it eventually completes the task.

Steerability
When a user issues a correction, can the agent quickly adjust its direction? Some models get stuck in a loop, repeatedly outputting the same erroneous solution, while others correct their course in a single step. This dimension directly dictates collaboration efficiency.

Long-Horizon Consistency
In tasks spanning dozens of interaction turns, does the agent remember the initial goal? Does it go off-track midway? The completion rate of long-horizon tasks is a critical watershed for distinguishing model capabilities.

Task Completion
Did the task actually get completed, or does it just “look completed”? This requires a comprehensive judgment combining user confirmation signals and objective execution results.

5. Engineering Implications

For developers and enterprise users, this evaluation methodology offers several practical insights:

First, prioritize real-world task metrics over benchmark scores during model selection. A model that scores high on static benchmarks but exhibits a near-zero net improvement rate in real-world, long-horizon tasks offers limited production value. Prioritize models that demonstrate consistently positive performance in real-session evaluations.

Second, consider the cost-effectiveness of open-weight models. Open-weight models have reached competitive levels in dimensions like steerability and command execution reliability. For scenarios involving sensitive data or requiring private deployment, they are viable options worth serious evaluation.

Third, feed evaluation signals back into your own applications. Confirmation, correction, and error signals within conversations can be used not only for model selection but also for internal product quality monitoring. Establishing a similar signal collection pipeline enables teams to continuously track model performance drift in actual business operations.

Fourth, do not blindly trust a single metric. The net improvement rate is an aggregate score, but specific business use cases may prioritize certain sub-dimensions. Programming scenarios emphasize command reliability, while customer service scenarios prioritize steerability. Model selection should apply secondary weighting based on business priorities.

6. Conclusion

AI agent evaluation is shifting from “artificial exam rooms” to “real battlefields.” The methodology combining real-session signal mining and causal tracing with randomized inference provides the industry with a more credible evaluation path. It does not pursue absolute score rankings; instead, it answers a more pragmatic question: In long-horizon tasks with real users, how much better is this model compared to the average?

For developers, the value of understanding this methodology lies not in memorizing a specific ranking, but in establishing an evaluation mindset: using real data, causal methods, and a multi-dimensional perspective to determine whether an agent can truly get the job done. This is the critical judgment required for model selection and engineering implementation.