Decoding AI Agent Evaluation: From Real-World Session Signals to Net Improvement Rates

1. Why Traditional Evaluation Falls Short for AI Agents

In the realm of Large Language Model (LLM) evaluation, early mainstream approaches relied heavily on A/B testing where users voted between two model responses, or automated scoring using static question banks. While these methods are adequate for measuring text generation quality, their limitations become glaringly obvious when applied to “agent tasks”—long-horizon missions requiring tool invocation, command execution, and multi-turn interactions.

There are three primary issues with traditional methods:

1. Long Task Horizons and Multi-Step Processes: A real-world agent task might involve dozens of interaction turns, encompassing file read/write operations, command executions, and error retries. The correctness of the final answer simply does not reflect the reliability of the intermediate steps.
2. High Noise in Subjective Voting: Users typically only see the final output and cannot judge whether the model made correct decisions during intermediate steps. Voting results are easily skewed by superficial fluency.
3. Static Question Banks vs. Real-World Scenarios: Constructed test cases struggle to cover the diverse range of user intent expressions and environmental constraints found in production environments.

Consequently, the industry is exploring a new path: abandoning manual voting in favor of mining signals directly from real user agent sessions and estimating model capabilities using statistical inference.

2. Core Evaluation Logic: Real Sessions + Signal Extraction + Causal Inference

2.1 Data Source: Real Long-Horizon Sessions

The foundational data for this evaluation comes from agent sessions generated by users in actual production environments. These sessions inherently possess several advantages:

• Authentic Tasks: Users arrive with clear objectives, meaning the task distribution closely mirrors production environments.
• Complex Environments: Interactions involve real file systems, command-line interfaces, and dependency version constraints.
• Natural Feedback: Users unconsciously provide signals during the session, such as confirmations, corrections, and complaints.

For instance, one evaluation cycle utilized a sample size of over 17,500 real long-horizon agent task sessions, covering various task types including work, programming, and general chat. This scale ensures the statistical stability of the inferences.

2.2 Key Signal Extraction

From these sessions, the evaluation system automatically extracts several categories of key signals:

Signal Type	Description	Capability Dimension
Explicit Confirmation	The user explicitly states the task was completed correctly	Task Completion
Praise	The user expresses satisfaction	Overall Experience
Complaint	The user expresses dissatisfaction or points out errors	Output Quality
Correction Command	The user asks the model to change direction or amend its approach	Steerability
Command Execution Error	A shell command returns a non-zero exit code or error	Execution Reliability

The common characteristic of these signals is that they are not manually annotated; rather, they are byproducts of natural user interaction. This avoids annotation bias and ensures the evaluation results closely align with real-world user experiences.

2.3 Causal Tracing: Randomized Inference Method

Signals alone are insufficient; a method is required to translate them into comparable model capability metrics. The core methodology employed here is randomized inference, akin to “causal tracing.”

The basic logic can be simplified as follows:

1. In real sessions, different models are randomly assigned to different users (or different sessions).
2. Because the assignment is random, the differences between models can be approximated as causal effects rather than user selection bias.
3. For each model, the system calculates its net improvement rate relative to the “average level.”

The core concept is represented in the following pseudocode:

# Pseudocode: Net Improvement Rate Estimation
for model in models:
    # Count positive and negative signals in the model's sessions
    positive = count_signals(model, ["confirm", "praise"])
    negative = count_signals(model, ["complaint", "correction", "shell_error"])
    # Calculate net improvement relative to the average level
    net_improvement = (positive - negative) / total_sessions
    # Obtain confidence intervals through randomized inference
    ci = randomized_inference(net_improvement, baseline="average")

The advantage of this method is that it does not rely on absolute scores; instead, it measures the “net improvement relative to the average level,” thereby circumventing the scaling issues caused by varying task difficulties.

3. Understanding the Evaluation Metric System

3.1 Comprehensive Net Improvement Rate

The comprehensive net improvement rate is the core metric, reflecting the model’s overall performance across all task types. For example, an open-weight model achieved a +5.1% comprehensive net improvement rate in an evaluation, representing a 4.2 percentage point increase over its previous generation’s +0.9%. This improvement is statistically significant, indicating a substantial leap in the model’s reliability for real-world tasks.

3.2 Granular Metrics

Beyond the comprehensive metric, the evaluation also focuses on several granular dimensions:

• Bash Reliability (Shell Command Execution Reliability): Measures whether the commands generated by the model execute correctly without frequent errors. This is the most fundamental reliability requirement for agent tasks.
• Steerability (Direction Adjustment and Correction): Measures how quickly the model can understand and adjust its behavior when the user requests a change in direction.
• Task Type Ranking: Models are ranked separately across different categories (e.g., work, programming, chat) to help users choose the right model for their specific scenarios.

3.3 Ranking Dimensions

The evaluation results provide rankings across two dimensions:

• Open-Weight Model Ranking: Compares models whose weights are publicly available.
• Overall Participating Model Ranking: Compares against all participating models, including closed-source models.

For instance, a specific model might rank first among open-weight models but 13th overall, while still securing the top spot among open models in work, programming, and chat categories. This multi-dimensional ranking allows users with different requirements to find a reliable reference point.

4. Engineering Practice: Reproducing a Similar Evaluation Framework

If a team wishes to establish a similar evaluation mechanism within their own context, the following steps can serve as a guide:

4.1 Data Collection

• Implement instrumentation in the real product to record the complete trajectory of agent sessions (inputs, tool calls, command execution results, user feedback).
• Ensure randomized assignment: Different models must be invoked randomly across the same task distribution to avoid selection bias.

4.2 Signal Definition and Extraction

• Define positive and negative signal dictionaries, capturing phrases like “confirmed,” “thanks,” “incorrect,” or “start over.”
• For command execution results, directly parse exit codes and standard error outputs.
• Utilize a combination of rule-based systems and lightweight models to extract signals, thereby reducing manual labor costs.

4.3 Statistical Inference

• Use randomized inference or causal inference methods to estimate the net effect of each model relative to the baseline.
• For models with smaller sample sizes, provide confidence intervals rather than single-point estimates.

4.4 Continuous Iteration

• Regularly update the signal dictionaries and task classifications.
• Monitor the stability of model rankings to prevent misjudgments caused by sample fluctuations.

5. Interpreting Results and Model Selection Advice

For developers and enterprise users navigating these evaluation results, it is recommended to interpret the data from the following perspectives:

1. Review the Comprehensive Net Improvement Rate: This reflects overall reliability, but it must be assessed alongside confidence intervals to determine statistical significance.
2. Review Granular Metrics: If your business relies heavily on command-line operations, focus on execution reliability metrics. If your workflows require frequent direction changes, prioritize steerability.
3. Review Task Type Rankings: Select models that rank highly within the category corresponding to your specific business use case.
4. Review Open-Weight vs. Closed-Source: Open-weight models facilitate private deployment and secondary fine-tuning, making them ideal for enterprises with strict data security requirements.

6. Conclusion

Real-world task evaluation for AI agents is shifting from “manual voting” toward a data-driven paradigm of “real sessions + signal extraction + causal inference.” Its core values lie in:

• Authenticity: Based on long-horizon tasks from real users rather than constructed question banks.
• Objectivity: Signals are derived from natural user behavior, minimizing subjective bias.
• Comparability: The net improvement rate provides a unified scale for comparison.

For agent developers, understanding this evaluation mechanism is not only crucial for model selection but also serves as a reverse-engineering guide for model optimization. For example, improving command execution reliability and enhancing steerability are key to achieving a better net improvement rate in real-world tasks. As agent application scenarios continue to expand, this evaluation methodology will become a critical infrastructure for measuring the engineering capabilities of AI models.