Agent Evaluation Methodology: From Real Sessions to Net Improvement Rate

1. Why Agent Evaluation Is a New Problem

The focus of large models is shifting from “generating a piece of text” to “completing a task.” Agents, as the core form of this shift, exhibit three distinctive characteristics in how they operate:

• Long-horizon: A single task may span dozens of conversation turns and dozens of tool calls, with a complete loop of planning, execution, observation, and correction in between.
• Strong environment coupling: Final performance depends not only on the model itself but also on toolchain quality, retrieval effectiveness, system prompts, and the stability of the execution environment.
• Outcome-oriented: Users no longer care whether “the answer reads like a human wrote it” but whether “the file was correctly generated, the code runs, and the command executed successfully.”

These three characteristics mean agent evaluation cannot simply inherit the practices of traditional LLM evaluation. The evaluation target has changed from “single-turn text quality” to “long-horizon task outcomes,” and the scoring system must be restructured accordingly.

2. Limitations of Two Mainstream Evaluation Paradigms

2.1 Static Benchmarks: Scores in the Lab

Static benchmarks (fixed question sets plus standard answers or automatic graders) are the most classic evaluation approach, but they face three structural problems in agent scenarios:

• Contamination and overfitting: Once questions are public, they may enter training data, decoupling “leaderboard scores” from real capability.
• Distribution mismatch: Tasks in benchmarks are closed-ended with standard answers, whereas real agent tasks are open-ended, multi-constrained, and evolve with the conversation.
• Update lag: Agent capabilities change rapidly as toolchains iterate, and static benchmarks cannot keep up.

2.2 Pairwise Voting: Preference Is Not Capability

Another popular paradigm asks real users to blind-vote between two models’ responses. This works well for chitchat but largely fails in agent scenarios:

• Long tasks are hard to compare side by side: Users will not patiently watch two complete execution traces just to cast one vote.
• Sample bias: Voting samples naturally skew toward short conversations and lightweight tasks, precisely missing the long-horizon heavy tasks that are most central to agents.
• Style interference: Voting measures “user preference,” and factors like length, format, and tone systematically distort results. Preference is not equivalent to capability.

The conclusion is clear: the industry needs a third path—turn real production sessions themselves into an evaluation data source. Instead of asking users “who is better,” observe what actually happens in real tasks.

3. Mining Real Session Signals: Turning Production Traffic into Evaluation Assets

The core of this path is automatically extracting key signals strongly correlated with task success or failure from raw sessions. Four types of signals have been validated in industry practice:

Signal Type	Polarity	Strength	Source	Description
Explicit task confirmation	Positive	Strong	Subjective	User explicitly acknowledges the result is achieved, e.g., “Yes, that’s it”
Praise / complaint	Positive / Negative	Medium	Subjective	Explicit user sentiment feedback
Correction instruction	Negative	Strong	Subjective	“No, change it to…” indicates the previous action deviated from the goal
Command execution error	Negative	Strong	Objective	Non-zero exit code, error stack trace, from the execution environment

Several engineering notes:

• The value of correction instructions goes beyond “penalty”: The number of turns from when a correction occurs to when the task returns on track is itself first-hand material for measuring “steerability.”
• Objective signals have the lowest noise: Command execution errors do not depend on user judgment and are suitable as hard indicators for the reliability dimension.
• Extraction needs layering: High-confidence patterns are handled by rule matching; borderline samples (e.g., distinguishing complaints from催促) use lightweight intent classification models; correction instruction detection must be compared against adjacent turns to avoid misclassifying normal requirement iteration as error correction.

Illustrative signal extraction code:

def extract_signals(session):
    """Extract evaluation signals from an agent session (illustrative implementation)"""
    signals = []
    for i, turn in enumerate(session.turns):
        text = turn.user_text
        # 1) Task confirmation: user explicitly acknowledges the result
        if intent_match(text, patterns=["that works", "no problem", "just like that"]):
            signals.append({"type": "confirm", "polarity": +1, "turn": i})
        # 2) Sentiment feedback
        s = classify_sentiment(text)     # praise / complaint / neutral
        if s != "neutral":
            signals.append({"type": s, "polarity": 1 if s == "praise" else -1, "turn": i})
        # 3) Correction instruction: compare with previous user request, detect directional override
        if i > 0 and is_correction(text, session.turns[i - 1].user_text):
            signals.append({"type": "correction", "polarity": -1, "turn": i})
        # 4) Command execution error: objective signal from the execution environment
        for call in turn.tool_calls:
            if call.tool == "shell" and call.exit_code != 0:
                signals.append({"type": "shell_error", "polarity": -1, "turn": i})
    return signals

Compliance is an unavoidable prerequisite: session data must be anonymized, collected under the minimization principle, and users must be fully informed of data usage.

4. Net Improvement Rate and Randomization Inference: Making Rankings Survive Statistical Testing

Once signals are obtained, the naive approach is “comparing means”—whoever has more praise and fewer complaints ranks higher. But this mixes model differences with task difficulty differences and user population differences. For example, if a model happens to be heavily adopted by programming power users, its sessions will naturally contain more command execution errors, and directly comparing means will systematically underestimate its reliability—a classic confounding bias that in severe cases triggers Simpson’s paradox.

A more robust approach estimates the net improvement rate: the improvement in the target model’s positive outcome rate relative to the “average level of all evaluated models” while controlling for session-level conditions. For example, a net improvement rate of +5% can be roughly interpreted as: under equivalent task conditions, the model solves about 5% more critical problems than the average.

Its statistical backbone is randomization inference (permutation testing), in four steps:

1. Use the session as the analysis unit, recording (model label, signal score) for each session.
2. Compute the observed statistic: target model score minus the overall average score.
3. Construct the null hypothesis “model label is unrelated to session outcome”—randomly shuffle all model labels, recompute the statistic, and repeat thousands of times to form a reference distribution under the null.
4. The observed value’s improvement relative to the reference distribution is the net improvement rate, and the deviation magnitude also gives the significance level.

Illustrative code:

import random
def observed_gap(sessions, target):
    score_t = mean([s.score for s in sessions if s.model == target])
    score_all = mean([s.score for s in sessions])
    return score_t - score_all
def net_improvement(sessions, target, n_perm=10000):
    observed = observed_gap(sessions, target)
    labels = [s.model for s in sessions]
    null = []
    for _ in range(n_perm):
        random.shuffle(labels)                    # Null hypothesis: labels unrelated to outcomes
        null.append(observed_gap(relabel(sessions, labels), target))
    return observed - mean(null)                  # Net improvement rate

This method has three advantages: it does not rely on distribution assumptions and is more robust for long-tailed task scores; randomizing at the session level naturally balances confounding at the task-type level; and results come with built-in uncertainty measures, preventing noise from being misread as differences. Note that long-horizon agent sessions are far more expensive to obtain than short conversations—a single model typically needs thousands to tens of thousands of sessions to get a stable estimate; slicing by scenario requires proportionally larger sample sizes, otherwise confidence intervals for sub-ranking will be too wide.

5. From Composite Score to Capability Radar: Designing Sub-Metrics

The composite net improvement rate answers “who is better overall,” but enterprise selection needs to answer “who is better in my scenario.” Therefore, two layers of dimensions should be broken out beneath the composite score:

Capability dimensions (horizontal)

• Command execution reliability: Command execution success rate, and the proportion of errors that are autonomously fixed without user intervention.
• Steerability: The minimum number of turns needed for the agent to get back on track after a user issues a correction instruction.
• Result confirmation rate: The proportion of sessions that conclude with explicit user confirmation of completion.

Scenario dimensions (vertical)

• Workflow automation, software development, and general assistant scenarios should be measured separately. The same model’s net improvement rate can differ significantly across scenarios; a top composite ranking does not mean it also leads in your target scenario.

Dashboard design recommendation: display composite score, scenario slices, and capability slices in a three-way cross view, with each score accompanied by a confidence interval rather than just a point estimate—a leaderboard without confidence intervals easily mistakes statistical noise for capability gaps.

6. Engineering Practices for Building an In-House Evaluation System

This methodology is not only for external comparison but can be directly implemented as an enterprise’s internal quality infrastructure. The reference architecture has four layers:

1. Collection layer: Instrument the agent runtime to record session text, tool call parameters and return codes, and task end status, and feed them into a data lake through general cloud capabilities such as log services and message queues. If the agent is deployed on a cloud platform such as Baidu AI Cloud, the platform’s general log collection and monitoring/alerting capabilities can be used to build the pipeline.
2. Signal layer: Rule engines handle high-confidence signals, classification models handle borderline samples, and structured signal records are output uniformly.
3. Analysis layer: Periodically batch-compute the net improvement rate, sub-metrics, and significance for each agent version, with automatic comparison before and after releases.
4. Application layer: Feed back into the R&D process as a version regression gate, a basis for canary release traffic splitting, and a source of quality degradation alerts.

Two engineering red lines must be upheld:

• Prevent contamination: If evaluation session data flows back into the training set, it creates “optimizing for the evaluation” data leakage; evaluation and training sets must be strictly isolated.
• Prevent feedback loops: Do not let leaderboard results directly affect online traffic allocation, otherwise it breaks the “label-independent” assumption that randomization inference relies on, rendering statistical inference invalid.

7. Limitations and Correct Interpretation

Every evaluation system has boundaries, and the real-session signal method is no exception:

• Survivorship bias: If failed sessions where users gave up midway are not fully recorded, positive signals will be overestimated.
• Population differences: Different models face users with different skill levels and task structures; randomization inference can mitigate but not eliminate such bias.
• Subjective signal noise: User confirmation is not strictly equivalent to objective task success; critical business scenarios should overlay objective verification.

Mature teams typically adopt a “three-layer data cross-validation” strategy: real session signals (large scale, close to production) + small-scale controlled task sets (objective results, controllable variables) + periodic manual sampling (calibrating systematic bias). Only when the three corroborate each other can conclusions be reached that have both statistical confidence and business validity.

8. Conclusion

Agent evaluation is undergoing a paradigm shift: from “asking users who is better” to “inferring who is better from real tasks”—using real sessions as the data source, four signal types (task confirmation, sentiment feedback, correction instructions, and command execution errors) as the yardstick, randomization inference as the statistical backbone, and net improvement rate as the unified measurement language. For enterprises, the greater value of this method lies in internalization: letting the gains and losses of every agent version release be faithfully answered by real-world data, turning evaluation from a one-time external report card into a continuously iterating dashboard.