A Methodology for Evaluating AI Agents in Real-World Tasks: From Conversational Signals to Net Improvement Rate

1. Why Agent Evaluation Requires a New Approach

Evaluating Large Language Models (LLMs) has traditionally relied on two methods: static benchmarks and human preference voting. The former uses fixed question sets for scoring, but once the questions are public, they face data contamination risks. Furthermore, static benchmarks struggle to cover real-world agent scenarios like multi-turn interactions, tool calling, and dynamic environment states. The latter asks users to choose between two responses, which is costly, subjective, and inherently misaligned with actual usage contexts. A user’s judgment when facing a side-by-side comparison box does not perfectly correlate with their actual satisfaction during a real task.

Agent tasks are characterized by long life cycles, multi-step processes, and heavy reliance on the environment. A single session might include dozens of dialogue turns, multiple command executions, and several direction adjustments. Evaluating an isolated response is meaningless. What truly matters is: Throughout the complete task workflow, does the model make the user more efficient, reduce their need for corrections, and minimize rework? This introduces a new evaluation paradigm—inferring model quality from the “behavioral traces” of real sessions, rather than relying on explicit user votes.

2. Evaluation Targets and Data Foundation

This evaluation methodology targets real-world agent tasks, covering primary scenarios such as professional workflows, software development, and daily conversations. Data is sourced from tens of thousands of real, long-cycle agent sessions generated globally (e.g., over 17,500 sessions), rather than synthetic tasks constructed in a lab.

Using real sessions as evaluation data provides three key benefits:

1. Realistic task distribution: Tasks originate from actual user needs, naturally covering long-tail scenarios and eliminating the “benchmark leakage” problem.
2. Naturally generated signals: In real tasks, users spontaneously confirm results, express dissatisfaction, and issue corrective commands. These serve as high-quality quality signals.
3. Authentic environments: Tool calls, command executions, and file operations occur in real environments, ensuring failure modes align with production environments.

The trade-off is that the data is noisy, signals are sparse, and distributions are imbalanced. Therefore, the core challenge of this methodology is not simply “data collection,” but rather “how to reliably extract comparable signals from the noise.”

3. Key Signals: Translating User Behavior into Quality Metrics

The evaluation system automatically extracts multiple signal categories from sessions, each corresponding to an explainable capability dimension:

Signal Type	Behavioral Manifestation	Corresponding Capability Dimension
Explicit Confirmation	User acknowledges task completion or adopts the output	Task Completion
Praise	User voluntarily provides positive feedback	Output Quality
Complaints	User expresses dissatisfaction or requests a redo	Correctness & Stability
Corrective Commands	User points out deviations and requests direction changes	Controllability (Direction Adjustment & Correction)
Command Execution Errors	Shell command execution failures or errors	Execution Reliability

Two design principles here deserve further elaboration:

First, signals must be tied to “task progression.” Simple “likes” are easily skewed, but “whether a user needs to re-explain the same thing” is hard to fake. Corrective commands and repeated instructions represent the highest-cost negative signals—if a user is willing to type additional text to correct the trajectory, it strongly indicates that the previous output deviated from their intent.

Second, execution signals must be modeled independently. The fundamental difference between an agent and a traditional conversational model is that an agent takes action. Command execution failures directly break the task chain. Therefore, execution reliability (e.g., the success rate and error recovery capability of Shell commands) must be treated as an independent metric, rather than being mixed into a generic “response quality” score.

4. Causal Tracing: From Correlation to Net Improvement Rate

Once signals are established, the next step is attribution. Directly comparing the “praise rates” of different models introduces obvious biases: different models face varying session types, task difficulties, and user demographics. Raw ratios are not directly comparable.

The “causal tracing” method mitigates this issue through randomized inference. Its basic logic is:

1. Construct comparable units within sessions, treating interactions under the same task and contextual conditions as matched samples.
2. Randomize model identities or leverage natural random assignment to weaken confounding factors like task difficulty and user preferences.
3. Estimate each model’s Net Improvement Rate relative to the “overall average level”—specifically, how much the model increases the probability of task success compared to the baseline after controlling for confounders.

A simplified pseudocode illustrates this estimation process:

# Conceptual illustration: Randomized estimation of Net Improvement Rate
def net_improvement(sessions, model_id):
    treated, control = [], []
    for s in sessions:
        # Construct control groups via randomization or condition matching
        if s.model == model_id:
            treated.append(s.signal_score)
        elif s.is_matched_control(model_id):
            control.append(s.signal_score)
    baseline = mean(control)                 # Expected score of the average level
    observed = mean(treated)                 # Actual score of this model
    return (observed - baseline) / baseline  # Relative Net Improvement Rate

The actual implementation is far more complex, requiring handling of sample matching, variance estimation, confidence intervals, and multiple comparisons. However, the core concept is clear: translate “whether a model is good” into “how much a model improves over the average line under identical conditions,” thereby yielding a Net Improvement Rate that is comparable across models.

The advantages of this metric include:

• Explainability: The positive or negative sign directly indicates whether performance is above or below average, while the magnitude represents the scale of improvement.
• Confounding resistance: Randomization and matching weaken biases introduced by task difficulty and user demographics.
• Granularity: The same methodology can be applied independently to sub-metrics like execution reliability and controllability, forming a capability profile rather than a single holistic score.

5. Interpreting Metrics: How to Read an Agent Evaluation Report

When reviewing an evaluation report of this nature, the following interpretation sequence is recommended:

Step 1: Examine the Overall Net Improvement Rate. This answers the question, “Is it better than average overall?” For instance, if an open-weight model achieves an overall NIR of +5.1%, marking a 4.2 percentage point increase from its previous generation’s +0.9%, it indicates that the generational improvement is substantive rather than just statistical noise.

Step 2: Analyze Sub-metric Distributions. An aggregate score can mask capability shortcomings. Sub-metrics like execution reliability and direction adjustment/correction capabilities better reflect a model’s weak points in a real pipeline. A model with a mediocre overall score might rank highly in execution reliability, which holds greater value for users focused on automated scripting.

Step 3: Review the Ranking Context. A model’s position might vary significantly between an “open-weight models” leaderboard and an “all evaluated models” leaderboard (e.g., first among open-weight, thirteenth overall). This indicates that while top proprietary models still hold an advantage, the pace at which the open-weight tier is catching up is noteworthy. Additionally, pay attention to scenario-specific leaderboards—capability rankings across workflow, coding, and chat tasks are rarely identical.

Step 4: Consider Sample Size. Conclusions backed by tens of thousands of real sessions are vastly more reliable than those drawn from hundreds of samples. However, confidence intervals must still be considered to avoid treating minute decimal-point differences as definitive conclusions.

6. Engineering Practices: Applying this Methodology to Your Own System

This methodology is not just for reading leaderboards; it can guide the construction of your own evaluation systems. Here are a few recommendations for implementation:

1. Prioritize Telemetry over Surveys. Log events like user confirmations, repeated questions, corrective commands, and tool execution failures within the session pipeline to form structured logs. These logs are far more reliable than after-the-fact questionnaires.
2. Establish Control Mechanisms. Canary releases, A/B routing, and shadow execution of multiple models on the same task are low-cost ways to construct control groups. Without controls, you only have correlation, not causation.
3. Layer Your Metrics. Separate “task completion,” “execution reliability,” and “interaction controllability” into independent monitored metrics to prevent a single aggregate score from masking underlying issues.
4. Value Long-Cycle Sessions. Agent failures often only emerge after a dozen or more turns. The evaluation window must be sufficiently long; otherwise, capabilities will be systematically overestimated.
5. Continuous Calibration. Signal weights and baselines will drift as user demographics change. They require regular re-estimation to prevent metric distortion.

7. Conclusion

The value of evaluating real-world agent tasks lies in three fundamental shifts: shifting the evaluation target from “a single response” to “a complete task workflow,” shifting the evaluation method from “user voting” to “behavioral signals,” and shifting attribution from “raw ratio comparison” to “Net Improvement Rate under randomized inference.” Together, these transitions solve the noise, bias, and explainability challenges inherent in real-world scenario evaluations.

For developers, understanding this methodology means more than just reading a leaderboard. It provides the foundation to design your own effectiveness measurement system. Ultimately, the ability to reliably articulate “exactly how much this iteration improved” is a critical engineering capability in itself.