AI Agent Evaluation: A New Paradigm Based on Real Sessions and Net Improvement Rate

    This article introduces a new paradigm for evaluating AI agents—systems that plan, call tools, and interact with their environment to complete long-horizon tasks. It explains why preference voting falls short for agent tasks, then presents two core techniques: mining task-level signals from real user sessions (explicit confirmation, praise and complaints, corrective instructions, and command execution errors) and using randomized assignment to estimate a causal "net improvement rate." It also covers sub-metrics such as Bash Reliability and Steerability, how to read leaderboards beyond raw rankings, the engineering architecture for running such evaluations at scale, and a practical checklist—along with common pitfalls—for enterprises building their own agent evaluation system. All figures are illustrative and refer to no specific product or model.

    1. Introduction: Why Agent Evaluation Needs a New Paradigm

    The capability boundary of large models is expanding from “answering questions” to “completing tasks.” Agents equipped with planning, tool-calling, and environment interaction can autonomously break down requirements, execute commands, read and write files, and iterate through trial and error until they deliver a result.

    This shift poses a fundamental challenge for evaluation. Traditional evaluation measures “single-turn answer quality”; agent evaluation measures “whether a long-horizon task is ultimately accepted.” The former can be judged by human raters, while the latter requires observing the actual execution process.

    Some platforms have built dedicated real-time leaderboards for agents. This article uses that as its main thread to systematically review the core methodology and engineering practices. All data here is illustrative and does not refer to any specific product or model.

    2. The Challenge of Agent Evaluation: Why Preference Voting Falls Short

    2.1 The Changing Shape of Tasks

    Conversational Q&A usually ends within seconds, and the quality of an answer can largely be judged on the spot. Agent tasks, by contrast, often span tens of minutes or even hours, involving dozens or even hundreds of rounds of interaction between the model and its environment, and requiring continuous planning, tool calls, result observation, and course correction based on feedback.

    As tasks grow longer, failure modes become more diverse, for example:

    • Incorrect command parameters;
    • Misreading execution results;
    • Repeatedly trying in the wrong direction;
    • Drifting away from the user’s true intent midway.

    These problems are hard to detect by looking at any single-turn answer.

    2.2 Three Limitations of Preference Voting

    In conversational evaluation, a common approach is to have reviewers compare two models’ answers, vote for the better one, and aggregate a ranking from the votes. When applied to agent tasks, these limitations are amplified.

    1. Sparse coverage. The vast majority of sessions receive no votes, so the score actually reflects only a few sampled sessions and cannot represent a model’s true capability on complex tasks.
    2. Fragmented perspective. Reviewers typically see only the first few rounds of a task, not the full picture of “whether commands executed successfully and whether the user ultimately accepted the result.” More confident, more fluent answers tend to win votes—not necessarily the one that actually solves the problem.
    3. Broken causal chain. Preference voting is essentially correlational evidence: it can only show that “reviewers prefer a certain answer,” not “how much switching to a certain model would net-improve the task success rate.”

    Therefore, agent evaluation needs to shift toward two ideas: first, automatically mining task-level signals from real sessions; second, using randomized designs for causal inference.

    3. Mining Signals from Real Sessions: User Behavior as Evidence

    The core idea is to extract objective evidence of task completion from the behavior users naturally produce when interacting with the agent, rather than requiring users to provide additional ratings. There are four main signal types.

    Signal Type Typical Manifestation Nature Key Implementation Points
    Explicit confirmation User confirms the result meets requirements, or directly adopts the output Positive Rule-based with an intent-classification model as support; preserve context
    Praise and complaints Positive feedback or expressions of dissatisfaction Positive / Negative Distinguish emotional expression from factual judgment; filter out sarcasm and quotes
    Corrective instructions Instructions pointing out deviations, requesting rework, or overturning the original approach Negative Identify “rejection of the previous output,” not an ordinary follow-up question
    Command execution errors Shell commands returning non-zero exit codes, timing out, or being blocked Negative Recorded directly by the execution environment; objective and reproducible

    3.1 Explicit Confirmation: Users Vote with Their Actions

    Unlike preference voting, confirmation signals come from the real outcome at the end of a task. A task is only considered closed-loop once the user confirms the result is usable. This signal is highly resistant to interference, because users have already invested significant waiting and interaction to complete the task, so their conclusion is often more reliable than a casual vote.

    3.2 Praise and Complaints: The Most Direct Evidence of Satisfaction

    Attitudes that users actively express, whether positive or negative, are strong signals. In engineering, emotional expression must be separated from factual statements. For example, “It finally ran through this time” contains both praise and confirmation and can be scored together, avoiding a double-counted reward.

    3.3 Corrective Instructions: The Signal That Best Exposes Capability Gaps

    A correction means the model deviated from the user’s intent, or that the previous round’s result was flawed. It is more granular than “the final answer is wrong”: it can pinpoint exactly which step went wrong, and can measure how quickly a model corrects itself after being told about a problem.

    3.4 Command Execution Errors: Quantifiable, Automatable Hard Signals

    Agents rely heavily on shell commands to operate the environment, and whether a command succeeds is given directly by the execution environment, almost unaffected by subjective factors—making it the most suitable objective metric. Deeper analysis can divide them into several categories:

    • Syntax or usage errors: incorrect command spelling, parameters, or paths;
    • Expected failures: the target file does not exist, insufficient permissions, and so on;
    • Environment anomalies: timeouts, insufficient resources;
    • Blocked dangerous operations: high-risk commands involving deletion, overwriting, and the like.

    Only the first two categories should be attributed primarily to the model. The latter two must be judged against the state of the environment and, when necessary, excluded during attribution.

    3.5 Engineering Essentials for Signal Collection

    1. Separate facts from inferences. Execution errors and explicit user replies are factual signals and can be used directly. Confirmation, praise, complaints, and corrections rely on semantic recognition and should be assigned confidence scores and filtered by threshold, with low-confidence samples reviewed separately.
    2. Ensure complete context. Whether the same “try again” is a correction or an ordinary follow-up depends on the surrounding context, so the classifier input should cover the necessary rounds.
    3. Prevent attribution bias. When users quote model output (for example, restating an error message), they also use words like “error” or “failure.” Only negative expressions directed at the agent’s output or behavior should count as a complaint or a correction.
    4. Unify session granularity. All signals should land on the “session” as the analysis unit and be linked to metadata such as task category and task length for later stratified analysis.

    Illustrative labeling logic:

    1. POSITIVE_WORDS = {"yes", "okay", "succeeded", "done", "ran through", "no problem", "exactly"}
    2. NEGATIVE_WORDS = {"wrong", "incorrect", "not", "didn't succeed", "failed again", "redo"}
    3. def label_session(session):
    4. """Map a single agent session to a task-level outcome label.
    5. Every recognition stage must be replaceable and evaluable."""
    6. labels = []
    7. # 1. Command execution errors: objective records from the execution
    8. # environment, highest confidence
    9. shell_errors = [e for step in session.steps
    10. for e in step.shell_errors
    11. if e.cause in ("syntax", "expected_failure")]
    12. if shell_errors:
    13. labels.append(("shell_error", -1.0))
    14. # 2. User replies: distinguish factual feedback from quoted text
    15. for msg in session.user_messages:
    16. if msg.quotes_agent_output:
    17. continue
    18. intent, confidence = classify_intent(msg, context=session.context(msg))
    19. if confidence < 0.8:
    20. continue # low-confidence samples go to a manual review queue, not scored directly
    21. if intent == "praise":
    22. labels.append(("praise", 1.0))
    23. elif intent == "complaint":
    24. labels.append(("complaint", -1.0))
    25. elif intent == "correction":
    26. labels.append(("correction", -0.8))
    27. elif intent == "confirmation" and is_final_turn(msg):
    28. labels.append(("confirmation", 1.0))
    29. return aggregate_to_task_outcome(labels)

    4. Causal Tracking: Estimating the “Net Improvement Rate” with Randomization

    4.1 Why Randomization Is Needed

    Extracting signals from sessions only yields “task performance under a certain model.” To answer “how much would switching to another model improve things,” one must rule out interference from factors such as task difficulty and user habits.

    In real traffic, model assignment is often non-random—it may be chosen by users themselves or determined by a routing policy. More technically capable users tend to choose more capable models, and more capable models may be preferentially assigned to important scenarios. Directly comparing model performance here confounds model effects with differences in population and tasks.

    The idea behind causal tracking is to borrow from randomized controlled trials: randomly assign the models under evaluation to real tasks, compare the outcomes across groups, and thereby attribute the differences to the models themselves.

    4.2 Design Essentials

    1. Random assignment. Each eligible task session is assigned to a participating model with a certain probability. The more models participate, the less traffic each individual model receives, so a trade-off between traffic cost and statistical precision is needed.
    2. Stratified randomization. Stratify by task category (e.g., work, coding, chat) and by task scale to ensure each model has enough samples in every stratum, avoiding a concentration of certain task types on a few models.
    3. User-level consistency. The same user should, as far as possible, keep using the same model. This both reduces back-and-forth switching in the usage experience and avoids leaking routing patterns between models, which could introduce behavioral biases such as the “Hawthorne effect.”
    4. Stratified mixed reporting. When publishing results, report by stratum or aggregate with fixed weights, so that changes in task composition do not distort the rankings.

    4.3 Calculating the Net Improvement Rate

    First aggregate session-level signals into an outcome score, then compare it against the average across all sessions:

    1. For model m:
    2. Δ(m) = mean( Y_i | T_i = m ) − mean( Y_i )
    3. NetImprovement(m) = Δ(m) / mean( Y_i ) × 100%
    4. Y_i: task outcome score for session i (closer to 1 means more successful)
    5. T_i: the model to which session i was assigned

    A positive Δ(m) indicates that the model’s task outcomes outperform the average. The net improvement rate, expressed as a percentage, means “by what proportion the task performance improved relative to the average.”

    Illustrative calculation:

    1. Model Sessions Avg score Lift over average Net improvement
    2. A 3,200 0.72 +0.08 +12.5%
    3. B 2,800 0.66 +0.02 +3.1%
    4. C 3,100 0.60 −0.04 −6.3%
    5. All 9,100 0.64 — —

    4.4 Significance and Robustness

    1. Interval estimation. Use resampling methods such as bootstrapping to produce confidence intervals for the net improvement rate. Models whose intervals overlap substantially should not have their rankings over-interpreted.
    2. Stratification correction. Use inverse-probability weighting, or an estimator that takes “model mean − stratum mean” and then weighted-averages it, to correct for the bias caused by unequal sample sizes across strata:
    1. Δ_adj(m) = Σ_k w_k × ( mean(Y | T=m, stratum k) − mean(Y | stratum k) )
    2. w_k: the proportion of stratum k among all tasks
    1. Repeated tasks. When conditions allow, have several models each execute some of the same tasks to form paired comparisons, which can significantly improve estimation precision.
    2. Robustness checks. If the conclusion’s direction remains unchanged after changing the score aggregation method, adjusting signal weights, or altering the stratification scheme, the results are robust.

    5. Sub-Metrics: From “Who Is Stronger” to “Stronger Where”

    The composite net improvement rate answers who is better; sub-metrics answer where it is better and where it is worse. The sub-metrics in the source material include Bash Reliability (the reliability of shell command execution), Steerability (the ability to adjust direction and correct course), and others. They can be expanded into a diagnostic system:

    Dimension Core Question Data Source Typical Use
    Command execution reliability Can commands be written correctly on the first try and run stably Execution environment records Distinguish syntax errors from environment anomalies
    Course-correction ability After a problem is pointed out, can it quickly get back on track Corrective instructions and the subsequent trajectory Measure the first-pass rate after corrections
    Task completion Is the task ultimately confirmed by the user Explicit confirmation Serve as the backbone of the outcome score
    Satisfaction Is the user overall satisfied Praise and complaints Capture experience beyond completion
    Long-horizon stability Does performance decay as the task grows longer Per-round statistics Compare short, medium, and long tasks
    Efficiency Is the cost under control Interaction rounds, elapsed time, resource consumption Control cost and response time

    The value of sub-metrics lies in being interpretable and actionable for optimization:

    • Low command execution reliability indicates that the model has an insufficient understanding of the runtime environment, command syntax, or paths, so tool-use and execution-feedback comprehension should be strengthened;
    • Weak course-correction ability indicates that the model tends to cling to the original approach or spin in place when facing negative feedback, and needs improved reflection and re-planning mechanisms;
    • Marked long-horizon stability decay may be related to context management and state retention.

    6. Reading the Leaderboard: What to Watch Beyond Rankings

    6.1 Each Group Ranking Has Its Own Reference Frame

    Taking the results in the source material as an example: a certain open-weight model ranks 1st among open-weight models and 13th among all participating models. The two numbers do not contradict each other—the former answers “who is stronger among the same class of models,” while the latter answers “where it stands among all participating models.” Likewise, this model also ranks first among open-weight models in the three categories of work, coding, and chat, indicating that its advantage is broad rather than limited to a single task type.

    6.2 Watch Sample Size and Task Realism

    The source material states that the ranking is based on more than 17,500 real long-horizon agent task sessions from users worldwide. Two aspects of this kind of ranking are worth emphasizing:

    • The samples come from real users’ long-horizon tasks, closer to production than a small, artificially constructed test set;
    • With a sample size in the tens of thousands, the sub-metric results for an individual model become statistically meaningful.

    6.3 Watch Version Timeliness

    Models iterate frequently, so leaderboard results are time-sensitive. For example, a certain closed-source model ranks 5th on the real-time leaderboard; such a rank only reflects the version’s performance at a specific point in time. A reasonable use is to treat the leaderboard as a continuously observed “dashboard” rather than a one-off conclusion.

    6.4 Generational Comparison Is More Informative than Absolute Rank

    Taking a model from the source material as an example, its composite net improvement rate is +5.1%, while the previous generation (the Max version) was +0.9%—an improvement of 4.2 percentage points. This comparison shows two things:

    1. Moving from slightly above average to clearly above average, generational progress is reflected in the position relative to the baseline;
    2. To judge whether an upgrade is worthwhile, one should look at whether the improvement exceeds the noise range of the interval estimate, not merely at the rank number.

    6.5 The Leaderboard Cannot Replace Business Validation

    The leaderboard measures average performance on public tasks, while what enterprises truly care about is their own business:

    • Domain terminology;
    • Private toolchains;
    • Compliance requirements;
    • Cost budgets.

    All of these need to be answered by a self-built evaluation set.

    7. Platform Implementation: Engineering Architecture and Key Design

    To continuously produce a credible leaderboard, a stable data and computation pipeline is required.

    1. [Agent product / execution environment]
    2. │ session logs, execution records, user feedback
    3. ▼
    4. [Collection & de-identification] ──► [Message queue] ──► [Stream processing: session segmentation & signal recognition]
    5. │
    6. ▼
    7. [Data lake / object storage: detail & aggregate results]
    8. │
    9. [Offline computation: weighted scoring & causal estimation]
    10. │
    11. ┌──────────────────┼──────────────────┐
    12. ▼ ▼ ▼
    13. [Leaderboard query service] [Monitoring & alerting] [Analytics dashboard]
    1. Collection and de-identification. Before data enters the analysis pipeline, de-identify sensitive content such as paths, credentials, and personal information. User feedback follows the principle of minimum necessity, collecting only the information needed for judgment.
    2. Message-queue peak shaving. Agent task traffic has clear peaks, and asynchronous buffering avoids overloading the computation pipeline while also supporting parallel processing by multiple consumers.
    3. Object storage tiering. Detail data is used for retrospection and review, aggregated results for leaderboard publishing, with hot and cold data managed in tiers.
    4. Unified stream and batch computation. Signal recognition runs as streaming processing, while metrics that require full-data comparison, such as net improvement rate estimation, are computed periodically by offline jobs.
    5. Idempotency and consistency. Use “session ID + version number” as the idempotency key to prevent retries from causing double-counting. Stream and offline results are reconciled to ensure a consistent leaderboard definition.
    6. Monitoring and alerting. Track data latency, signal-distribution drift (e.g., a sudden spike in the proportion of a certain signal), and abnormal traffic to detect evaluation distortion promptly.
    7. Privacy and compliance. The evaluation involves a large volume of sessions, so data usage and access permissions must be strictly limited, and users must be clearly informed that their data is used for service-quality evaluation.

    8. Building Your Own Agent Evaluation System: Implementation Checklist and Common Pitfalls

    Enterprises with evaluation needs can build their own evaluation system by following these steps.

    1. Define tasks and criteria. Select representative long-horizon tasks from real business and clarify “what counts as completion.”
    2. Design signal collection. Prioritize objective signals (command execution, task status), with subjective signals as a supplement.
    3. Design traffic splitting and randomization. When a new model goes live, ramp up traffic randomly by proportion while collecting signals in sync; where conditions are limited, use an offline replay set for paired comparisons.
    4. Establish a metric system. Use composite metrics for the overall picture and sub-metrics for diagnosis, always attaching confidence intervals.
    5. Review periodically. Cross-validate evaluation conclusions with online problems (e.g., complaint rate, rework rate) to continuously calibrate signals and weights.

    Common pitfalls:

    • Voting in place of outcomes: preferences cannot directly indicate task success;
    • Looking only at averages: ignoring long-tail failures and variance underestimates the impact of serious errors;
    • Ignoring environment differences: attributing environment or network problems to model defects;
    • Unrepresentative samples: using only simple examples cannot reflect performance on complex tasks.

    9. Conclusion

    The core shift in agent evaluation is moving from “judging answers” to “judging tasks.” Mining signals based on real sessions yields evidence closer to production; causal inference through randomization yields the net improvement rate relative to the average; and then using sub-metrics to locate capability gaps forms an interpretable, reproducible, and continuously iterable evaluation system.

    For developers and enterprises, model selection should reference both public leaderboards and testing in their own scenarios. Public leaderboards help quickly narrow down the candidate set, while the self-built test set is the ultimate basis for the decision.