A New Paradigm for Agent Evaluation: From Human Voting to Causal Inference on Real Conversations

Why Traditional Evaluation Methods Are Breaking Down

Evaluating agent products has long relied on two approaches: static benchmark suites and pairwise human voting. Benchmarks run fixed tasks in controlled environments; pairwise voting asks users to pick between two responses. Both were acceptable in the chatbot era, but in long-horizon, multi-step, tool-dependent agent scenarios, they expose three structural flaws.

First, the task distribution is distorted. Benchmark tasks are carefully designed with few steps, clean environments, and clear goals, while real user conversations are full of ambiguous instructions, mid-course redirections, environmental anomalies, and repeated corrections. A high benchmark score says little about reliability on real tasks.

Second, evaluation signals are sparse. Pairwise voting requires users to actively compare responses, which most users never do in real conversations. The volume of votes you can collect is far smaller than the volume of actual tasks, producing biased samples and low statistical power.

Third, confounders are hard to eliminate. Different models are assigned tasks of different difficulty, users of different skill levels, and conversations of different lengths. Directly comparing success rates mistakes differences in task difficulty for differences in model capability.

An evaluation system that reflects real-world usage and offers genuine causal identification has therefore become a hard requirement for bringing agents to production.

Redefining the Evaluation Target and Signal Sources

The new approach widens the observation window from individual “responses” to entire “sessions.” A complete agent session—user instructions, model planning, tool calls, execution results, user feedback—naturally contains a large volume of quality signals that can be extracted automatically. Typical categories include:

• Explicit confirmation: the user directly acknowledges the outcome, such as confirming a problem is solved or accepting a deliverable.
• Praise and complaints: explicit, sentiment-bearing feedback that reflects subjective satisfaction.
• Correction requests: the user asks the model to change direction, fix a mistake, or try again—a strong signal of capability gaps.
• Execution errors: tool call failures, non-zero exit codes, script errors—objective faults that reflect reliability and environment adaptation.

The common advantage of these signals is that they are non-intrusive: users pay no extra voting cost, and structured parsing of existing sessions is all that is required. Compared with human voting, signal density rises by an order of magnitude, and the signals cover tasks users actually care about.

In engineering terms, signal extraction typically runs as a pipeline: normalize each session (aligning messages with tool calls and their returns), classify the categories above using rules or lightweight models, then aggregate signals at task granularity. The pseudocode below illustrates the idea:

def extract_signals(session):
    signals = []
    for turn in session.turns:
        if turn.is_explicit_confirm():
            signals.append(("confirm", turn.task_id, +1))
        if turn.is_complaint():
            signals.append(("complaint", turn.task_id, -1))
        if turn.is_correction():
            signals.append(("correction", turn.task_id, -1))
        for call in turn.tool_calls:
            if call.exit_code != 0:
                signals.append(("exec_error", turn.task_id, -1))
    return aggregate_by_task(signals)

One point is critical: signals must be bound to tasks, not to individual messages. Otherwise, multiple corrections within a single task get counted repeatedly, distorting comparability across models.

Net Improvement Rate: A Comparable Capability Metric

Raw signals alone are not enough. Evaluation has to answer a specific question: how much better is a given model than the “average”? This is where the Net Improvement Rate comes in: positive and negative signals are aggregated with task-level weighting, and the baseline expectation is subtracted to yield the model’s net gain relative to the average.

Intuitively, it answers this question: after controlling for task difficulty and user distribution, by how many percentage points does using this model raise the probability of task success compared with an average session? Compared with absolute success rates, the Net Improvement Rate is more robust to shifts in task difficulty; compared with leaderboard rank, it delivers an interpretable, quantified effect size.

Three design points matter:

1. Task-level aggregation: score each task first, then average across tasks, so long sessions do not dominate the result.
2. Difficulty normalization: stratify or regression-adjust for task difficulty to strip out the artificial boost from “being assigned easier tasks.”
3. Confidence intervals: report interval estimates based on session sample size, and avoid strong conclusions when samples are insufficient.

Causal Tracing: Removing Confounders with Randomized Inference

The Net Improvement Rate is only valid if model assignment is independent of task difficulty. In real products, however, the model a user chooses often correlates with the type of task, creating selection bias. The core idea of Causal Tracing is to use randomized inference to isolate the effect of the “model” variable from confounding factors.

The technical path can be summarized in three steps:

• Randomization injection: randomly assign the model in a subset of sessions to construct approximate treatment and control groups, breaking user self-selection.
• Propensity score correction: for observational data that cannot be fully randomized, apply propensity score weighting to balance the distribution of task characteristics.
• Effect estimation: on the balanced sample, estimate each model’s net effect relative to the average, along with an uncertainty range.

The value of this method is that it upgrades evaluation from “describing correlations” to “estimating causal effects.” For agents—whose performance is heavily influenced by task difficulty—this step is practically mandatory; without it, rankings are easily skewed by changes in task composition.

Breaking Capability Down by Dimension in Long-Horizon Tasks

An aggregate Net Improvement Rate tells you whether a model is good overall, but engineering teams care more about where it falls short. Evaluation therefore needs to decompose results into finer-grained capability dimensions, typically:

• Execution reliability: the success rate of shell-type command execution and the ability to recover from exceptions, reflecting how well the model understands the runtime environment and how well it handles faults.
• Controllability: after a user redirects or corrects mid-task, whether the model understands the new intent and adjusts its plan instead of blindly retrying.
• Task completion: whether multi-step tasks actually reach the end state rather than being abandoned midway or falling into loops.

These dimensions complement the Net Improvement Rate: the aggregate metric supports horizontal comparison, while the fine-grained metrics pinpoint where to optimize. In practice, it is worth maintaining two dashboards in parallel so the team does not fixate on the total score and overlook weak spots.

Open-Weight Models: Results and Implications

One notable observation from evaluations built this way is that the gap between open-weight models and closed models is narrowing, particularly on long-horizon real-world tasks. In one case, a family of open-weight models achieved a Net Improvement Rate of roughly +5% across more than 17,500 real long-horizon sessions—an improvement of several percentage points over the previous generation—and ranked among the leading open models on dimensions such as execution reliability and controllability.

Two engineering implications follow.

First, the evaluation method itself determines the conclusion. With short-task benchmarks or human voting, the real strengths of open-weight models may be underestimated; evaluation grounded in real conversations does a better job of reflecting deployment value.

Second, reliability affects user experience more than “intelligence”. In long-horizon tasks, a single failed command execution or a single misread correction can sink the entire task. Model optimization should therefore treat tool-call robustness and error-recovery strategy as equal in priority to reasoning ability.

A Practical Roadmap

For teams building their own agent evaluation systems, the recommended sequence is:

1. Build the signal pipeline first: extract confirmation, complaint, correction, and execution-error signals from existing session logs, and validate extraction accuracy.
2. Aggregate at the task level: make sure signals bind to tasks, and define clear scoring rules and baselines.
3. Add causal correction: once sample sizes are sufficient, apply randomization or propensity score correction so rankings are not misled by task composition.
4. Decompose capability dimensions: cover at least reliability, controllability, and completion, producing an actionable optimization list.
5. Monitor drift continuously: task distributions and user behavior change over time, so evaluation baselines need regular re-estimation.

Evaluation is not a one-off exercise but the infrastructure for continuously calibrating the gap between models and real-world needs. Turning real conversations into quantifiable signals, then using causal methods to turn those signals into trustworthy conclusions, is a step agents cannot skip on the road to large-scale deployment.