Open-Weight Agent Evaluation Methodology: From Real Session Signals to Net Improvement Rate

As open-weight models continue to scale in parameter count, objectively measuring an agent’s capability in real-world task scenarios has become a central question for developers and enterprises making technology choices. Traditional model evaluation typically relies on humans voting on preferences between two responses—an approach that is costly, prone to sampling bias, and poorly suited to long-horizon, multi-turn agent tasks. This article walks through an evaluation approach designed for real agent workloads, examining its signal extraction mechanism, statistical inference method, and fine-grained metric design, and discusses how to turn evaluation conclusions into model selection and optimization decisions in engineering practice.

Why Traditional Evaluation Struggles to Measure Agent Capability

Agent tasks differ fundamentally from single-turn Q&A. A typical agent session involves multiple rounds of tool calls, state maintenance, error recovery, and environment interaction, so the quality of a single response cannot represent the performance of the whole task chain. Specifically, traditional evaluation has three prominent limitations:

1. Mismatched evaluation granularity. Preference voting targets a single response, while an agent’s value shows up in the completion of long-horizon tasks. Local optimality does not equal global optimality.
2. Unrealistic signal sources. Artificially constructed benchmarks struggle to cover how real users behave under ambiguous requirements, mid-task changes, or environment anomalies.
3. Cost versus scale. High-quality human annotation is expensive and cannot support session evaluation at the scale of tens of thousands, leaving statistical confidence insufficient.

A more sound approach, therefore, is to base evaluation on real interaction records between users and agents, letting actual user behavior serve as the natural source of evaluation signals.

Extracting Key Signals from Real Sessions

Real sessions contain rich implicit feedback. Users rarely provide explicit scores, but their behavior reveals their judgment of task outcomes. The signals can be grouped into several categories:

• Explicit confirmation: the user acknowledges the result and moves on to the next step, which usually indicates the current step met the bar.
• Praise and complaints: explicit positive or negative statements; these are high-intensity evaluation signals.
• Correction instructions: the user asks for a redo, a change of direction, or to fill in something missing, indicating the previous output deviated.
• Execution errors: failed tool calls, non-zero exit codes, invalid parameters, and similar—directly observable, objective failure signals.

Once structured, every session can be converted into a labeled sequence of samples. Compared with asking users to vote, this approach draws samples from the real task distribution, offers broad coverage, and keeps signals tightly coupled to task context.

On the engineering side, signal extraction typically requires a parsing pipeline, for example:

def extract_signals(session):
    signals = []
    for turn in session.turns:
        if turn.has_tool_call and turn.exit_code != 0:
            signals.append(("exec_error", turn.index, -1.0))
        if turn.user_intent == "correction":
            signals.append(("correction", turn.index, -1.0))
        if turn.user_intent == "confirm":
            signals.append(("confirm", turn.index, +1.0))
    return signals

It is worth emphasizing that the key to signal extraction is intent classification, not keyword matching. Users often phrase corrections tactfully, and simple rules tend to miss or misclassify them.

Causal Tracking: Estimating Net Improvement Rate with Randomized Inference

Signals alone are not enough; you also need a comparable statistic. Directly counting success rates is confounded by task difficulty distribution: different models face different task sets, so comparing raw success rates is not fair.

One viable approach is randomized inference, often described in the industry as “causal tracking.” The core idea is to introduce randomness at the session assignment stage so that, in expectation, the tasks faced by different models come from the same distribution. This attributes differences between models to the models themselves rather than to task difficulty. On this basis, you can estimate each model’s net improvement rate relative to the overall average (or a chosen baseline):

Net improvement rate = the model’s expected performance under randomized control − the average level

This metric has several advantages:

• Strong comparability: it removes systematic bias caused by task difficulty.
• Interpretability: the sign directly indicates whether a model is better or worse than the average.
• Aggregatable: it can be computed per task category and per metric dimension, enabling layered diagnostics.

Two caveats matter in practice. First, randomization must happen at session entry to avoid confounding from user self-selection. Second, the sample size must be sufficient: long-horizon tasks have high variance, and net improvement rates fluctuate noticeably on small samples, so report confidence intervals rather than point estimates.

Fine-Grained Metrics: From Aggregate Scores to Capability Profiles

The aggregate net improvement rate works for overall ranking, but engineering decisions usually need a finer capability profile. Common dimensions include:

Execution reliability. Measures the success rate of tool calls and command execution, with particular attention to recovery after failures. An agent may plan correctly, but if it frequently errors during execution, its practical usability suffers.

Steerability. Measures how well the model responds to user direction changes and corrective instructions—including whether it understands the change intent, whether it modifies only what is necessary, and whether it preserves already confirmed context. In multi-turn collaboration, this often matters more for user experience than single-turn accuracy.

Task-category breakdown. Splitting sessions into categories such as workflow, coding, and conversation prevents strong results in one category from masking weaknesses in another. For enterprise users, this layered view is more informative than a single ranking.

As a concrete example from an open-weight model: it achieved a net improvement rate of roughly +5.1%, up about 4.2 percentage points from the +0.9% of the previous generation in the same series; it ranked among the top open-weight models on fine-grained dimensions such as execution reliability and steerability, placed first among open-weight models across work, coding, and conversation task categories, and fell in the mid-to-upper tier among all evaluated models. These figures are based on more than 17,500 real long-horizon agent sessions from global users—a sample size that provides a reasonably solid basis for statistical inference.

Turning Evaluation Results into Engineering Practice

The value of evaluation results ultimately shows up in deployment. A three-stage approach is recommended:

1. Selection stage: Do not rely on overall rankings alone; prioritize the task categories and fine-grained metrics most relevant to your business. For scenarios dominated by automated script execution, focus on execution reliability; for human-in-the-loop editing scenarios, focus on steerability.
2. Launch stage: Using the signal types from the evaluation as a reference, build corresponding monitoring metrics on your own session logs to close the feedback loop. Execution error rate, correction rate, and confirmation rate all work as online health indicators.
3. Optimization stage: Target improvements at weak dimensions—for example, parameter validation and retry mechanisms for tool calls to improve execution reliability, or clearer instruction parsing and context anchoring to improve steerability.

Conclusion

The methodological value of evaluation built on real agent tasks lies in three things: replacing human voting with real sessions brings signal sources closer to production; replacing raw success-rate comparisons with randomized inference makes model differences attributable; and replacing a single ranking with layered, fine-grained metrics makes conclusions directly actionable for selection and optimization. For open-weight models, this framework both shows where a model stands relative to the average and exposes specific capability gaps, making it a worthwhile reference for technical decision-making. When adopting any leaderboard conclusion, developers should validate it against their own business distribution rather than equating general benchmark results with usability in a specific scenario.