Agent Evaluation on Real-World Tasks: From Conversational Signals to Causal Inference

1. Why Traditional Evaluation Methods Are Failing

For the past few years, large language model evaluation has relied mainly on two approaches: static benchmarks and human preference voting. The former measures accuracy on a fixed question set, while the latter asks users to choose between two responses. But as agents take on increasingly complex tasks—cross-file code refactoring, multi-turn tool calling, long-horizon environment interaction—both approaches show clear limitations.

The problem with static benchmarks is training set contamination and scenario distortion. Once questions are public, they risk being absorbed into training data. More importantly, real tasks often have no standard answer; they are a process of trial, error, and gradual convergence, which is fundamentally different from the single-choice logic of benchmark evaluation.

The problem with human voting is surface-level preference. Users are drawn to fluent phrasing and polished formatting, but these surface features correlate weakly with actual task completion quality. One response may be elegantly worded yet miss a critical boundary condition, while another may be plain and direct but genuinely solves the problem—voting mechanisms struggle to capture that difference.

The industry therefore needs a new evaluation paradigm: one that does not rely on subjective judgment but automatically mines objective signals from real usage scenarios.

2. Signal Mining from Real Sessions

The core idea behind a new generation of agent leaderboards is to stop constructing manual evaluation sets and instead analyze agent session logs from real users working on long-horizon tasks. These sessions naturally contain a wealth of extractable quality signals; the key is how to define and capture them.

2.1 Four Categories of Key Signals

Evaluation systems typically extract signals automatically along the following dimensions:

(1) Explicit confirmation signals

When a user says things like “that’s right,” “exactly,” or “that solved it,” the output of the current turn has been accepted. This is the most direct positive signal. Note that confirmation must target the task itself rather than being a social courtesy, so contextual judgment is required.

(2) Praise and complaint signals

Compared with confirmation, praise (e.g., “that’s a good approach”) and complaints (e.g., “you got it wrong again,” “this isn’t what I asked for”) carry stronger emotional coloring and usually correspond to clearer positive or negative judgments. The system needs fine-grained classification of such expressions, distinguishing evaluations of task results from evaluations of interaction experience.

(3) Correction instruction signals

When a user asks the agent to change direction, redo a step, or points out a specific error, this constitutes a strong negative signal. For example:

User: No, I said keep the existing interface signature and only change the internal implementation.

The value of correction instructions is that they pinpoint the agent’s deviation in understanding or execution, carrying more information than a vague “not satisfied.”

(4) Execution error signals

In tasks involving command-line operations, runtime errors such as failed Shell commands, timeouts, and permission denials are hard negative signals. These signals are fully objective and require no language understanding, making them the most reliable of all.

2.2 The Challenge of Signal Aggregation

Individual signals are often noisy. A user may complain because of a bad mood, or issue repeated corrections because the task itself is unsolvable. Evaluation systems therefore need to aggregate multiple signals into a session-level assessment and then further into model-level metrics.

A common approach is to compute a “net signal score” for each session and then combine scores with weighting. Weights should reflect the reliability of each signal type—execution errors typically carry more weight than verbal confirmation, since the former does not depend on semantic judgment.

3. Causal Tracing: Inferring Causation from Correlation

Simply counting signal frequencies is not enough. A model might happen to be assigned easier tasks and thus achieve a higher confirmation rate. To produce a fair model ranking, task difficulty must be stripped out as a confounding factor.

3.1 The Basic Idea of Randomized Inference

Causal Tracing is an inference method based on randomized experiments. Its core assumption is that if tasks are randomly assigned across models, then differences in model performance can be attributed to the models themselves rather than to differences in task distribution.

In real scenarios, task assignment is often not fully random—users select models based on their own expectations. However, by stratifying session features (such as task type, session length, and number of tool calls), randomization can be approximated within each stratum, yielding more reliable estimates.

3.2 Computing the Net Improvement Rate

The net improvement rate is defined as a model’s improvement over the average of all evaluated models. Formally, if model $m$ scores $s_{m,t}$ on task $t$ and the global average is $\bar{s}_t$, then:

Net improvement rate(m) = E_t[ s_{m,t} - \bar{s}_t ]

In practice, task difficulty must be adjusted for, and confidence intervals reported. Only when the confidence interval of the improvement rate excludes 0 can the model be considered genuinely better than average.

The advantage of this method is that it does not require absolute scores, only relative comparisons, while the randomized design effectively controls for confounding variables.

4. Evaluation Dimensions and Sub-Metrics

The composite net improvement rate is only an overview metric. Understanding model capability in depth requires looking at sub-dimensions. Common categories include:

4.1 Task Type Dimension

• Work tasks: document processing, information organization, schedule coordination, and similar
• Programming tasks: code generation, debugging, refactoring, test writing, and similar
• Chat tasks: open-ended dialogue, knowledge Q&A, creative collaboration, and similar

Models can differ widely across task types, so category-specific rankings are more informative than overall rankings.

4.2 Capability Dimension

• Shell command execution reliability: the proportion of commands that succeed on the first attempt, reflecting how well the model understands environment constraints
• Steerability: whether the model can accurately understand and act on user correction instructions, reflecting robustness in instruction following
• Long-horizon consistency: whether the model keeps its goal from drifting across multi-turn interactions

These sub-metrics often reveal a model’s real weaknesses better than an overall score. For example, a model may score high on programming tasks but low on Shell reliability, indicating that its generated code is logically correct but poorly adapted to the environment.

5. Practical Takeaways for Developers

Understanding this evaluation mechanism offers several direct benefits for developers:

First, prioritize sub-metrics when choosing a model. A high overall score does not mean the model fits your scenario. If your work is mainly code refactoring, focus on programming and steerability metrics.

Second, pay attention to long-horizon task performance. High scores on short tasks may come from pattern matching, whereas long-horizon tasks expose genuine understanding and planning ability. Evaluations built on more than ten thousand real long-horizon sessions offer more reference value than short benchmarks.

Third, understand the limitations of evaluation. Signal mining depends on language understanding and may misjudge; causal inference depends on randomization assumptions that are hard to fully satisfy in real scenarios. Any leaderboard should therefore be treated as a reference, not the sole basis for decisions.

Fourth, apply the evaluation approach to your own projects. You can borrow this method to extract signals such as user confirmations, corrections, and execution errors from your own production logs, build an internal evaluation system, and continuously monitor model performance.

6. Conclusion

Agent evaluation based on real sessions represents an important shift in evaluation paradigms: from manual construction to production mining, and from subjective voting to causal inference. By automatically extracting multi-dimensional signals—confirmations, praise, complaints, corrections, and execution errors—and combining them with randomized inference methods, it estimates a model’s net improvement rate relative to the average.

The value of this mechanism lies not only in producing rankings but in offering a reusable methodology: how to measure agent capability continuously and objectively from real usage data without disturbing users. For any team deploying agents in production, this is an approach worth understanding and adopting.