Agent Evaluation’s New Paradigm: From Subjective Voting to Causal Attribution in Real Conversations

1. The Evolution of Evaluation Leaderboards: Why Voting Is Losing Its Usefulness

The way large language model capabilities are evaluated is undergoing a structural shift. Early leaderboards built around human preference voting still provide some reference value for dialogue quality and text generation. But as model capabilities expand from answering questions to completing long-horizon tasks, the limitations of voting become increasingly apparent.

The core problem comes down to three factors.

First, task cycles are getting longer. A real agent task often involves a dozen or even a hundred or more interaction turns, covering file reads and writes, command execution, error recovery, and direction changes. Asking an evaluator to vote once on an entire session loses an enormous amount of information.

Second, evaluation dimensions are multiplying. Whether a task succeeds no longer depends only on whether the final answer is correct. It also depends on whether the intermediate process is reliable, whether the model can self-correct after mistakes, and whether it strictly follows the user’s constraints. These signals are hard to express in a simple “A is better than B” judgment.

Third, voting samples are biased. Users who actively participate in voting are often a group with strong preferences in specific domains, and their distribution does not match real-world usage. When evaluation results are used for engineering decisions, this bias directly translates into decision risk.

As a result, a new approach to evaluation has emerged: instead of relying on human voting, it extracts objective signals directly from real user-agent conversations and uses statistical inference to estimate differences in model capability. This article discusses the technical principles behind this approach and the capability picture revealed by one open-weight model evaluation based on it.

2. How the Evaluation Works: Signal Extraction and Causal Tracing

2.1 Evaluation Targets and Task Types

This type of evaluation focuses on real agent tasks: long-horizon tasks that users assign to agents in actual workflows and that require multi-step execution. In terms of task domains, it typically covers three main categories:

• Work tasks: document processing, information organization, workflow automation, and similar activities.
• Coding tasks: code generation, debugging, refactoring, and script writing.
• Chat tasks: open-ended Q&A, multi-turn collaboration, and requirement clarification.

These three task types fail in very different ways. Work tasks depend heavily on instruction following and format stability. Coding tasks place greater weight on command execution correctness and error recovery. Chat tasks emphasize context retention and intent understanding. A single aggregate score is therefore rarely sufficient; granular metrics are needed.

2.2 Key Signals: The “Ground Truth” Hidden in User Behavior

This evaluation does not ask users to provide additional annotations. Instead, it automatically extracts the following types of signals from conversation history:

Signal type	Concrete manifestation	Meaning
Explicit confirmation	User says “that’s correct” or “exactly”	Task completed
Praise	User proactively gives positive feedback	Exceeded expectations
Complaint	User expresses dissatisfaction or repeats a complaint	Task failed or poor experience
Correction instruction	User points out an error and asks for a redo	First output was unsatisfactory
Command execution error	A shell command returns a non-zero exit code or an error	Execution reliability problem

What these signals share is that they are behaviors users produce naturally, without being asked to evaluate anything. They are therefore closer to real experience than after-the-fact votes. Correction instructions and command execution errors in particular are among the most diagnostically valuable negative signals in agent tasks.

2.3 Causal Tracing: Turning Signals into Comparable Scores

Signals alone are not enough. A statistical framework is needed to convert those signals into measurable differences in model capability. The method used here can be called causal tracing, and its core approach is as follows:

1. Randomization-based inference: Randomness is introduced at the session level so that the assignment of tasks of a given type across models is approximately random, thereby reducing confounding caused by task difficulty.
2. Baseline and net improvement rate: Using the average performance of all participating models as the baseline, the net improvement rate of each model relative to that baseline is estimated. A positive net improvement rate means that, under the same task distribution, the model is more likely than average to generate positive signals and less likely than average to generate negative signals.
3. Confidence intervals and sample size: Because the signals come from real sessions, sample size directly determines the robustness of the conclusions. The evaluation therefore explicitly discloses the number of sessions on which it is based.

The simplified logic of this process can be expressed in pseudocode:

def net_improvement_rate(model_sessions, baseline_sessions):
    # Extract positive and negative signals from each session
    pos = count_positive_signals(model_sessions)
    neg = count_negative_signals(model_sessions)
    # Compare against the baseline to estimate relative improvement
    delta_pos = pos / len(model_sessions) - count_positive_signals(baseline_sessions) / len(baseline_sessions)
    delta_neg = neg / len(model_sessions) - count_negative_signals(baseline_sessions) / len(baseline_sessions)
    # Net improvement rate: positive uplift minus negative uplift
    return delta_pos - delta_neg

It is important to emphasize that the net improvement rate is a relative metric. It answers the question “how much better is this model than the average?” rather than “what is this model’s absolute accuracy?” This distinction is critical when interpreting rankings.

3. Interpreting the Results: Where Open-Weight Models Stand

3.1 Overall Ranking and Sample Size

In a public evaluation, a particular open-weight model ranked first in the open-weight category, placed 13th among all participating models, and ranked first among open models across work, coding, and chat tasks. The results were based on more than 17,500 real long-horizon agent sessions from global users.

Seventeen thousand five hundred sessions is a notable scale. For long-horizon tasks, a single session may contain dozens of interaction turns, meaning the volume of underlying signals far exceeds the session count itself. A sufficiently large sample is what allows the confidence interval around the net improvement rate to narrow to a level where meaningful distinctions can be made.

3.2 Overall Net Improvement Rate: What +5.1% Means

The model’s overall net improvement rate was +5.1%, up 4.2 percentage points from the previous generation’s +0.9%.

Two measured interpretations are worth making here.

First, +5.1% is a net improvement relative to the average, not an absolute accuracy rate. In an evaluation where all participating models already have basic competence, the average itself is reasonably high. Each percentage point therefore corresponds to meaningful behavioral differences across a large number of real sessions.

Second, the jump from +0.9% to +5.1% suggests the improvement is not a single-point optimization. If the goal were simply to raise scores in one task category, the net improvement rate would typically be accompanied by declines elsewhere. A large increase in the overall net improvement rate usually indicates systemic strengthening of foundational capabilities, such as instruction-following stability, error recovery, and long-context consistency.

3.3 Granular Metrics: Bash Reliability and Steerability

Two granular metrics deserve particular attention.

Bash Reliability

Agents frequently need to execute shell commands during real tasks, such as file operations, environment checks, dependency installation, and script execution. A failed command not only interrupts the task but can also produce side effects that are hard to recover from. This metric measures how often a model produces a non-zero exit code or a clear error in sessions where command execution is required. Models with high reliability tend to confirm more context before generating commands and, after a command fails, actively read the error message before correcting it.

Steerability

This metric measures how quickly and accurately a model adjusts its subsequent behavior when a user gives a correction instruction or changes the direction of a requirement. Real-world scenarios include users modifying constraints mid-session, pointing out that a previous understanding was wrong, or asking for a different implementation approach. Models with poor steerability often appear to accept the feedback without actually changing behavior, or they change in the wrong direction. Users then have to issue repeated corrections, inflating the number of turns in the session.

This model ranked near the top among open models on both metrics. Its strength is not concentrated in a single task type but distributed across foundational capabilities that matter most for agents.

4. Practical Value for Developers and Enterprise Model Selection

4.1 How the Evaluation Signals Map to Real Engineering Pain Points

From an engineering perspective, these signals correspond closely to problems developers encounter on a daily basis:

• Frequent correction instructions → the model’s understanding of ambiguous requirements is unstable, requiring clearer prompt engineering.
• Frequent command execution errors → the model lacks sufficient awareness of the execution environment, requiring additional environment context or validation steps.
• Difficulty adjusting direction → the model has weak context management across multi-turn sessions, suggesting the need for session segmentation or externalized state.

The granular metrics in this type of evaluation can therefore serve directly as a checklist during model selection. If your business scenario involves dense shell operations, focus on execution reliability. If users frequently change requirements mid-session, focus on steerability.

4.2 Positioning Open-Weight Models

A core value of open-weight models is that they allow enterprises to deploy, fine-tune, and audit them in their own environments. When an open-weight model enters the upper tier of a real-task evaluation, it means that, under the condition of being self-hostable, the model already has the foundational capability to handle production-grade agent tasks.

That said, several points still deserve attention during model selection:

1. Whether the evaluation’s task distribution matches your business distribution. The evaluation covers work, coding, and chat tasks. If your business is concentrated in one of these, focus on the corresponding sub-metric.
2. The net improvement rate is a relative value. When building your own baseline, run a small-scale controlled experiment on your own task distribution rather than directly adopting public rankings.
3. Granular metrics matter more than the aggregate score. The aggregate score reflects overall capability, while granular metrics reveal risk points. For production systems with high stability requirements, the latter is more important.

5. Summary

Agent evaluation is moving from subjective voting toward real signals combined with causal inference. The significance of this shift is not just that rankings become more objective. More importantly, it returns evaluation to the questions developers actually care about: whether a task can be completed correctly on the first attempt, whether the model can fix its own mistakes, and whether it can keep up when users change requirements.

An evaluation based on more than 17,500 real sessions showed that a particular open-weight model ranked first in the open-weight category, achieved an overall net improvement rate of +5.1%, and performed particularly well in command execution reliability and steerability. For teams evaluating self-hosted agent solutions, these granular signals are more useful than a single ranking. They point to capability dimensions that are reproducible, verifiable, and directly actionable for specific engineering decisions.