A New Paradigm in AI Agent Evaluation: Real Conversation Signals and Net Improvement Rate

1. Why Pairwise Voting Falls Short for Agent Evaluation

Over the past few years, the dominant paradigm for evaluating Large Language Models (LLMs) has been the “arena-style” pairwise comparison. Users are shown responses from two models to the same prompt and vote for the better one. The system then aggregates massive amounts of votes into Elo ratings and rankings. This mechanism works well for open-ended Q&A or copywriting because it only requires a low-cost, subjective judgment from the user.

However, the nature of AI Agent tasks is fundamentally different:

• Long, non-interchangeable processes: A real-world agent session might involve dozens of interaction turns, multiple file reads/writes, and command executions. The results of the first half directly alter the task state of the second half. The sessions of two different models cannot be simply swapped and compared like two static answers.
• Objective outcomes, subjective processes: Whether a task is ultimately completed—whether tests pass, scripts run successfully, or reports are generated—is an objective, verifiable fact. However, asking a bystander to vote on “which process was better” is highly impractical.
• Lack of user motivation: Real users simply want to solve their own problems. Asking them to perform side-by-side comparisons for complex tasks is almost unfeasible. Even if they did, the sample would be heavily skewed toward simpler tasks.

Consequently, a new evaluation approach has emerged in the industry: instead of organizing human judges, treat the real-world agent sessions themselves as evaluation data. The system automatically mines evidence of task success from these sessions and uses statistical inference to convert them into comparable scores.

2. Real Conversations as a Data Source: Representation, Anti-Overfitting, and Compliance

Compared to manually constructed benchmark datasets, real conversation data sources offer three significant advantages:

1. Distribution aligns with real needs: Users bring long-tail, cross-domain, and highly constrained tasks. These are often underrepresented in manual datasets but truly reflect a model’s capability boundaries in production environments.
2. Difficult to game the leaderboard: The evaluation corpus comes from a continuous stream of constantly updating real sessions, drastically reducing the opportunity for targeted optimization against a fixed set of benchmark questions.
3. Sample scales naturally with usage: Statistical conclusions achieve a stable foundation only when accumulated sessions reach the scale of thousands or even tens of thousands of real, long-cycle tasks.

Of course, using real conversations requires strict compliance: sessions must undergo desensitization and authorization processing. Metrics must be published in an aggregated statistical format to prevent leaking any individual user information.

3. Automatic Extraction of Four Key Signals

The core of this evaluation system is the automatic extraction of structured signals from unstructured conversations. A typical signal dictionary includes four categories:

• Explicit Confirmation: The user explicitly states that the task is complete or accepts the result (e.g., “That’s good,” “Send it out as is”).
• Praise and Complaints: Explicit feedback with positive or negative polarity, reflecting user satisfaction with the process and outcome.
• Correction Instructions: The user interrupts the current direction, asking for a different approach or a redo. This is the most direct evidence of an agent “going off track.”
• Tool Execution Errors: Non-zero command-line exit codes, error stack traces, or repeated retries of the same command. These are objective and difficult to fake.

A desensitized signal extraction result might look like this:

{
  "session_id": "anon-8f3a",
  "task_domain": "coding",
  "turns": 23,
  "signals": [
    { "type": "tool_error",  "turn": 12, "tool": "shell", "exit_code": 1 },
    { "type": "correction",  "turn": 15, "summary": "Requested a different implementation approach" },
    { "type": "praise",      "turn": 21, "polarity": "positive" },
    { "type": "confirmation","turn": 22, "polarity": "positive" }
  ]
}

Individually, these signals are noisy—user personalities and expression habits vary wildly. However, when aggregated across massive sessions, they form stable statistical metrics: confirmation rate, correction rate, first-pass command success rate, and self-recovery rate after errors.

4. Causal Tracing: Estimating Net Improvement Rate via Randomized Inference

Having signals is not enough. Directly comparing raw signals falls into the trap of confounding factors: different models handle tasks of varying difficulty, serve different user demographics, and experience different session lengths. A higher confirmation rate for Model A might simply mean it was assigned easier tasks.

A common approach adopted by evaluation platforms to counter this is a randomized inference method known as “Causal Tracing”:

1. Construct Comparable Groups: Leverage platform-side random allocation (or active random routing) of sessions to ensure that the task families entering different models have a consistent distribution in difficulty and type.
2. Estimate Net Effects: Compare the occurrence rates of key signals within the same task family to determine a model’s net improvement margin relative to the average level of all participating models. This is the “Net Improvement Rate.”
3. Aggregation and Testing: Weight and aggregate across task families into a comprehensive Net Improvement Rate, providing confidence intervals and significance levels simultaneously.

A highly simplified illustration:

Average confirmation rate baseline for a task family = 60%
Model M's confirmation rate for this task family    = 63%
Net improvement for this task family ≈ (63% - 60%) / 60% = +5%

(The actual implementation is far more complex, requiring control over covariates like task difficulty, session length, and user profiles. This example merely illustrates the concept of “effect size benchmarked against the average.”)

Compared to pairwise win rates or Elo ratings, the advantage of the Net Improvement Rate lies in its interpretability: it directly answers “how much better is this model than the average,” rather than just providing a relative ordinal ranking. Furthermore, anchored by task outcome signals, it more closely reflects “whether the task was actually accomplished.”

5. Granular Metrics Worth Tracking

Beyond comprehensive rankings, several granular metrics offer greater reference value for engineering decisions:

• Shell Command Execution Reliability: First-pass success rate of commands and self-repair rate after errors. A significant portion of an agent’s actual work involves command-line operations. This metric directly determines whether you can let the agent run autonomously.
• Steerability: After receiving a correction instruction, how many turns it takes to converge to the user’s expected direction, and whether the issue recurs post-correction. In long sessions, this capability often dictates final success more than single-turn “intelligence.”
• Scenario-Specific Performance: Task structures vary greatly across scenarios like office collaboration, software development, and daily chat. Rankings among “open-weight models” versus “all participating models” must be interpreted separately. Granular data like this is precisely characterizing the gap between open-weight models and proprietary flagships.

6. Developer Practices: Reading Leaderboards and Building Evaluations

Four principles for reading leaderboards:

1. Look at sample size and confidence intervals first. Ranking differences with clearly overlapping intervals do not constitute a valid conclusion.
2. Check if the task distribution matches your business scenario. If the evaluation set is heavily skewed toward coding, its reference value for office scenarios is limited.
3. Prioritize granular metrics (reliability, steerability) before consulting the comprehensive score.
4. Monitor the trend of the same model across version iterations. Trends indicate improvement directions better than single-point rankings.

Five steps to build your own evaluation system:

1. Session Logging: Under the premise of desensitization and authorization, sink production environment agent sessions into evaluation assets.
2. Define Signal Dictionary: Use a combination of “rules + model classification” to label the four signal types: confirmation, complaint, correction, and error.
3. Random Routing: Route new model versions to homogeneous tasks via A/B testing to ensure causal interpretability of the comparison.
4. Metrics Dashboard: Track core metrics like Net Improvement Rate, first-pass command success rate, and correction convergence turns. Configure monitoring alerts.
5. Complement with Offline Benchmarks: Online signals reflect real-world performance, while offline benchmarks ensure reproducibility and fast regression testing. Both are indispensable.

The infrastructure requirements for this pipeline are not high. General cloud capabilities such as model invocations, logging services, object storage, and monitoring alerts are sufficient to support it. The key lies in the rigor of signal definition and the randomness of traffic routing.

7. Limitations and Future Outlook

Real conversation evaluation is not perfect:

• Signal Noise: Explicit confirmation is merely a proxy variable for satisfaction. Implicit user dissatisfaction (silently abandoning the task) is hard to capture.
• Long-tail Coverage: Low-frequency task samples are insufficient, leading to wide confidence intervals. Conclusions here must be treated with caution.
• Inference Assumptions: If randomization is incomplete or covariate control is insufficient, the Net Improvement Rate may still be biased.

It is foreseeable that AI agent evaluation will evolve into a hybrid system: “online real-world signals + offline reproducible benchmarks + specialized capability tests.” For developers, understanding the methodologies behind these metrics holds more long-term value than memorizing rankings on any leaderboard. Ultimately, the goal of evaluation is not merely to rank models, but to answer a simple question: If you integrate it into your system, can it actually get the job done?