Agent Evaluation Methodology for Real-World Tasks: From Session Signals to Causal Inference

1. Why Traditional Evaluation Is Failing

As large models iterate rapidly, objectively measuring how a model performs on real-world tasks has become one of the toughest problems for developers and enterprises when selecting models.

Traditional evaluation approaches fall into two broad categories:

• Static benchmarks: Running fixed question sets for scores. The advantage is comparability; the drawback is a severe disconnect from real usage. A model may score high on a benchmark but fall apart once it enters multi-turn interaction, tool calling, or long-horizon tasks.
• Human blind voting: Asking users to pick one of two responses. The advantage is closeness to subjective experience; the drawbacks are high cost, large sample bias, and an inability to capture the “process quality” that matters most in agent tasks.

The problem is that Agent tasks are fundamentally different from ordinary Q&A. An agent task may span dozens of interaction turns, involving file reads and writes, command execution, error recovery, and direction adjustments. A user finally saying “okay” does not mean the intermediate process was fine; conversely, a user interrupting mid-way to correct the model does not mean overall performance was poor.

Therefore, the evaluation system needs a new methodology: do not look at user votes on results, but mine signals from real sessions; do not rely on simple statistics, but use randomized inference to estimate net improvement.

2. Core Idea: Extracting Key Signals from Real Sessions

The first step of this methodology is to abandon the “ask users to evaluate actively” approach and shift to “passively mining behavioral signals in real sessions.”

In a typical agent session, users naturally produce a large number of observable behaviors, and these behaviors themselves are the most authentic feedback on model performance. They can be grouped into the following categories:

2.1 Explicit Confirmation Signals

The user explicitly states that the task is complete or the result is correct, such as “that works”, “no problem”, or “that’s it.” These signals represent positive feedback, but one must distinguish “grudging acceptance” from “genuine satisfaction.”

2.2 Praise and Complaint Signals

Praise such as “that approach is great” or “faster than I expected,” and complaints such as “you got it wrong again” or “this isn’t what I wanted.” These signals carry high emotional intensity and strong discriminative value, but the samples are relatively sparse.

2.3 Correction Instruction Signals

The user restates requirements, adds constraints, or points out the wrong direction. This is the most common and most valuable negative signal in agent tasks, because it directly reflects a deviation in the model’s understanding, planning, or execution.

2.4 Execution Error Signals

In tasks involving command-line or script execution, non-zero exit codes, exception stack traces, and timeout interruptions are all objective, hard error signals that do not depend on subjective user judgment.

2.5 Interaction Pattern Signals

For example, whether the user repeatedly retries the same instruction, interrupts multiple times within a short window, or degrades from a long task to short Q&A. These patterns reflect the model’s stability and controllability.

After structuring and extracting these signals, each session can be mapped to a weighted “performance vector” rather than a simple thumbs-up/thumbs-down.

3. Causal Tracing: Estimating Net Improvement Rate with Randomized Inference

With signals in hand, the next question is: how do we turn signals into comparable model rankings?

Directly averaging signal strength across models introduces serious bias, because session distributions differ across models—some models are used for more complex tasks, and some serve pickier user populations. To address this, randomized inference methods are needed, often referred to in the industry as “causal tracing.”

The basic logic is as follows:

1. Randomized exposure: In real traffic, randomly assign user sessions to different models according to a certain strategy, ensuring that confounding factors such as task difficulty and user profile are roughly balanced across models.
2. Signal alignment: For the same task category, extract standardized signals from each model’s sessions.
3. Net improvement rate estimation: Using the average level of all participating models as the baseline, estimate each model’s net improvement rate relative to that baseline, rather than an absolute score.

This process can be expressed with simplified pseudocode:

def net_improvement_rate(model_sessions, baseline_sessions):
    # Extract standardized signals
    model_signals = extract_signals(model_sessions)
    baseline_signals = extract_signals(baseline_sessions)
    # Align task difficulty distributions
    aligned = align_by_task_difficulty(model_signals, baseline_signals)
    # Estimate relative improvement
    improvement = estimate_relative_gain(aligned)
    return improvement

The meaning of the net improvement rate is: after controlling for task difficulty and user distribution, how much positive increment does this model bring relative to the average level. It can be positive or negative. A positive value indicates above-average performance; a negative value indicates below-average performance.

The advantages of this method are:

• Bias-resistant: Randomization eliminates systematic errors caused by uneven task assignment.
• Comparable: Different models share the same baseline, so rankings are consistent.
• Close to reality: Signals come from real long-horizon tasks, not artificial test sets.

4. Granular Metrics: Why a Single Overall Score Is Not Enough

Agent tasks cover an extremely wide range, from chat and coding to operations automation, and a single overall score cannot reflect a model’s actual usability. Therefore, evaluation systems usually break results down into multiple granular dimensions:

• Task completion: Whether the user ultimately confirms the task is achieved.
• Direction adjustment capability: When the user corrects course, whether the model can quickly understand and adjust strategy rather than stubbornly persisting.
• Execution reliability: In tasks involving command execution and script running, whether the model can stably produce runnable results and reduce abnormal exits.
• Long-horizon stability: In sessions exceeding a certain number of turns, whether the model can still maintain context consistency and goal focus.
• Interaction efficiency: The number of interaction turns and user interventions required to reach the same result.

Together, these dimensions form a capability profile. A model may not stand out on the overall score but lead on execution reliability; another model may offer a good chat experience but drift off on long-horizon tasks. For developers, granular metrics are often more informative than the overall score.

5. Engineering Implementation: Building a Usable Evaluation Pipeline

To implement a similar methodology in your own team, you can follow these steps:

5.1 Data Collection and De-identification

Collect real session logs, including user inputs, model outputs, tool call records, and execution results. De-identification and compliance handling must be done properly, removing personally identifiable information and sensitive content.

5.2 Signal Extraction Layer

Design rules and lightweight models for different signal types:

• Rule matching: For high-frequency patterns such as explicit confirmation and correction instructions.
• Classification models: For semantic judgments such as praise and complaints.
• Execution log parsing: For command exit codes and exception capture.

5.3 Randomization and Control

When traffic allows, perform random grouping at the session level to ensure the task distributions faced by different models are comparable. If online randomization is not possible, offline replay plus stratified sampling can approximate it.

5.4 Statistical Inference Layer

Use net improvement rate estimation against a relative baseline, and provide confidence intervals. Avoid using point estimates for absolute rankings.

5.5 Visualization and Iteration

Present the overall score and granular metrics in a dashboard, supporting drill-down by task type, user group, and time window. The evaluation system itself also needs continuous iteration, adjusting signal weights as task distributions change.

6. Common Pitfalls in Practice

When building such an evaluation system, there are several easily overlooked traps:

• Treating user silence as satisfaction: A user not responding does not equal satisfaction—it may mean giving up. This must be judged together with session termination patterns.
• Ignoring task difficulty differences: Comparing models without controlling task distribution is like comparing scores between easy and hard questions.
• Over-relying on a single signal: Looking only at complaint rates underestimates models that users cannot be bothered to complain about but that actually perform poorly.
• Ignoring long-horizon decay: Many models perform excellently in the first few turns but collapse once turns pile up, so long-session performance must be measured separately.
• Treating evaluation as a one-off task: Models change, users change, and tasks change, so the evaluation system must run and calibrate continuously.

7. Summary

The core challenge of agent evaluation is not “how to score,” but “how to objectively measure the net value a model brings in real, complex, long-horizon tasks.” Mining signals from real sessions, using randomized inference to estimate the net improvement rate, and supplementing this with granular dimension profiles is a proven feasible path.

For developers, the value of understanding this methodology lies not only in reading leaderboards, but also in transferring its ideas into their own business evaluations: replacing guesswork and single-point benchmarking with real data, controlled experiments, and relative baselines, so as to make more reliable technical selection and optimization decisions.