From Real-World Conversations to Net Improvement Rate: A New Methodology for Evaluating AI Agents
As large language models evolve into AI agents, traditional benchmarks increasingly fail to reflect real-world performance. This article explains how real user conversations, automatically extracted quality signals, and randomized causal analysis can be combined to measure an agent’s net improvement rate. It also examines key evaluation dimensions and shows how development teams can apply the methodology to model selection, monitoring, and iterative improvement.
From Real-World Conversations to Net Improvement Rate: A New Methodology for Evaluating AI Agents
1. Why Traditional Evaluation Methods Are Being Redefined
As large language models enter the AI agent era, evaluation faces a fundamental challenge: the gap between laboratory benchmark scores and real-world user experience is growing.
Traditional pairwise blind testing—asking users to compare two model responses and vote—works well in chat scenarios. However, it exposes several limitations when applied to AI agents:
- Tasks often span long workflows: An agent task may involve dozens of tool calls, file edits, and command executions. It is difficult for users to make a reliable judgment from a single comparison.
- Results depend heavily on the environment: Agent performance is closely tied to context configuration, toolchains, and permission settings. Artificial evaluation environments are difficult to align with real-world task distributions.
- Preference does not equal correctness: Users may prefer a response that appears fluent, while the core value of an agent lies in whether it actually completes the task. Voting signals can therefore be misleading.
As a result, a new evaluation approach has emerged: instead of actively initiating controlled comparisons, systems passively collect real user-agent conversations and automatically extract quality signals from the interaction traces. These evaluation leaderboards focus on performance in real agent tasks and provide an increasingly important reference for measuring practical model capability.
2. The Core Mechanism: Automatically Extracting Evaluation Signals from Conversation Traces
The key idea is that evaluation signals are not obtained by directly asking users to rate a model. Instead, they are mined from the interaction between users and agents. Common signals include the following.
1. Explicit User Confirmation
When a user indicates at the end of a task that it is “done” or “good enough,” the statement can serve as a strong signal that the task was completed successfully. Semantic analysis can automatically identify these endpoint events.
2. Praise and Complaints
Positive feedback, such as “That solution is elegant,” and negative feedback, such as “This is not the result I wanted,” form a natural source of evaluation data. Compared with forced voting, spontaneous feedback often provides a more authentic reflection of user sentiment.
3. Corrective Instructions
When a user says, “That is not right—use the other approach,” or “You edited the wrong file,” the correction directly reveals a deviation in the agent’s execution. The frequency of corrective instructions is an important measure of steerability and controllability.
4. Shell Command Execution Errors
When an agent executes terminal commands, non-zero exit codes, error traces, and retry behavior provide objective, machine-detectable failure signals. Compared with subjective text feedback, these signals are less noisy and easier to reproduce.
Illustrative signal extraction workflow:Conversation logs → Turn segmentation → Intent/sentiment detection → Signal labeling→ Aggregation by task type (work / coding / chat)→ Causal inference analysis
By aggregating tens of thousands of real, long-running agent task sessions from users worldwide, an evaluation system can cover task distributions across work, coding, chat, and other categories, while also producing more focused leaderboards.
3. Causal Tracking: Estimating the Net Improvement Rate of a Model
Extracting signals is only the first step. The more difficult question is: how can we isolate the contribution of the model itself from conversation data affected by task difficulty, user expertise, and environmental differences?
This is the problem addressed by causal tracking. Its central idea is derived from randomized controlled experiments:
- Random assignment: At the entry point, real sessions are randomly assigned to different models. This ensures that, in statistical terms, the task distributions received by each model are comparable.
- Establishing a baseline: The average performance of all participating models is used as the baseline for average performance.
- Estimating the net improvement rate: A model’s net improvement rate is defined as the difference between its task-success quality and the baseline.
For example, if an open-weight model has an overall net improvement rate of +5.1%, it means that, after controlling for confounding factors such as task distribution, the model performs 5.1 percentage points above the average of all participating models. The metric can also be used for generational comparisons. If a model improves from +0.9% in one generation to +5.1% in the next, the 4.2-point increase provides a direct view of the real benefit of the new version rather than a mechanical rise in a benchmark score.
This design has three major advantages:
- Greater resistance to leaderboard gaming: The data comes from real conversations, making it extremely costly to fabricate voting results.
- No fixed prompt set to leak: Because there is no static evaluation question set, models cannot simply memorize or optimize for the benchmark.
- Dynamic and continuously updated evaluation: Once a new model is launched, it can enter the blind-testing traffic pool and contribute to a continuously updated leaderboard.
4. Granular Metrics: Decomposing Agent Capabilities
In addition to the overall net improvement rate, granular metrics help developers identify a model’s specific strengths and weaknesses. Common dimensions include:
| Metric | What It Measures | Typical Signal Sources |
|---|---|---|
| Shell command execution reliability | One-shot command success rate and error recovery capability | Exit codes and error traces |
| Direction adjustment and error correction | The quality of the agent’s response to user corrections | Task trajectory after correction |
| Task completion confirmation rate | The proportion of tasks that users explicitly confirm as completed | Semantic analysis of conversation endpoints |
| Long-horizon stability | Performance degradation during multi-turn tasks | Segmented success-rate curves |
For developers, this metric system supports more precise model selection. If the primary use case is automated operations, shell reliability should receive greater weight. If the main use case is pair programming, correction and steering capabilities may be more important.
5. Practical Lessons for Developers
Teams deploying AI agents can apply the following practices based on this evaluation methodology.
1. Build an Internal Conversation-Signal Pipeline
Structure and store events such as user confirmations, complaints, corrections, and command failures to create an internal quality dashboard. A simple signal extraction example is shown below:
def extract_signals(session_log):signals = []for turn in session_log.turns:if turn.cmd_exit_code != 0:signals.append(("bash_error", turn.cmd))if detect_correction(turn.user_text):signals.append(("steering", turn.user_text))if detect_confirmation(turn.user_text):signals.append(("task_done", turn.task_id))return aggregate(signals)
2. Use Relative Metrics Instead of Absolute Scores
Metrics such as net improvement rate, which measure performance relative to the average, can answer whether a new version is genuinely better more effectively than an isolated benchmark score.
3. Prioritize the Collection of Failure Signals
Command execution errors are among the lowest-cost and least noisy quality signals. They should be integrated into monitoring and alerting systems, such as centralized logging and observability platforms.
4. Create a Closed Loop Between Evaluation and Iteration
Bring the leaderboard-style evaluation approach into internal development processes. Whenever a model or prompt is upgraded, use a small percentage of real traffic for randomized routing and measure the net effect. This helps prevent the situation where a system improves in the lab but regresses in production.
6. Conclusion
AI agent evaluation is moving from “humans evaluating models” toward data-driven model evaluation. Real conversations provide the raw material, automated signal extraction provides the measurement mechanism, and randomized causal inference provides the analytical framework. Together, they produce net improvement metrics that can directly support model selection and iterative development.
For developers, understanding this methodology makes it easier not only to interpret agent evaluation leaderboards accurately, but also to reproduce the same principles within their own product quality systems—ensuring that every upgrade is supported by real-world evidence.
