From Voting to Causal Tracing: A New Approach to Benchmarking AI Agents on Real-World Tasks

Rethinking Agent Evaluation: Why Voting Is No Longer Sufficient

Evaluating AI agents has long faced a persistent dilemma: models often score high on static benchmarks yet deliver inconsistent results in real usage. Early evaluation methods relied heavily on “double-blind comparison voting,” where human evaluators were shown responses from two models and asked to judge which was better. This approach works reasonably well for single-turn question answering, but it breaks down quickly for agentic tasks for three main reasons:

1. Long task horizons. Agent tasks often involve dozens of tool calls, file read/write operations, and environment interactions. Evaluators cannot fully perceive the entire process in a single presentation, so voting is reduced to judging only the superficial quality of the final output.
2. Process signals are discarded. User confirmations, corrections, complaints, and repeated instructions during a real session are high-value indicators of whether an agent actually completed the task. Voting mechanisms capture none of this information.
3. Preference bias is hard to eliminate. Voting results are significantly influenced by display order, response length, and phrasing style, all of which correlate weakly with actual task completion quality.

As a result, evaluation systems need to shift from “asking users to score” to “inferring capability from real usage behavior.” This is precisely the design principle behind agent evaluation leaderboards that do not rely on subjective voting. Instead, they mine real user-agent conversations, automatically extract key behavioral signals, and apply statistical inference methods to estimate model capability.

Real Session Signals: The Data Foundation for Evaluation

Real sessions are the closest available proxy to production conditions for agent evaluation. A typical long-horizon session includes: the user issuing a task, the model planning steps, calling tools, returning intermediate results, the user confirming or correcting, and task closure. The signals extracted from this full chain fall into four broad categories:

• Explicit confirmation signals: The user accepts, approves, or proceeds to the next step, implying task completion.
• Praise and complaint signals: Explicit positive or negative feedback reflecting user satisfaction.
• Correction instruction signals: The user asks the model to change direction, redo a step, or fix an error, directly exposing planning flaws or execution deviations.
• Execution error signals: Tool call failures, non-zero exit codes, script exceptions, and other hard reliability indicators.

The key advantage of these signals is that they require no additional labeling cost — they already exist in the natural interaction between user and agent. Compared to manual annotation or crowdsourced voting, session-based evaluation offers inherent benefits in scale and authenticity. However, it also introduces a new challenge: different models face different session distributions. User demographics, task types, and session lengths all affect signal frequency. Simply comparing complaint rates or error rates across models introduces selection bias, which demands more rigorous statistical treatment.

Causal Tracing: Net Improvement Rate Under Randomized Inference

“Causal tracing” is the core methodology behind this evaluation approach. Its goal is not to produce absolute scores but to estimate each model’s net improvement rate relative to the average (or a baseline). The general process can be summarized as follows:

1. Build a comparable session pool. Stratify real sessions by task type, length, tool-call complexity, and other dimensions to ensure models are compared under similar distributions.
2. Randomize assignment and aggregate signals. Within qualifying sessions, randomly expose similar tasks to different models (or introduce randomization at the model routing layer). This makes model assignment approximately independent of user characteristics, cutting off confounding variables.
3. Weight signals and calibrate against baseline. Assign different weights to confirmation, complaint, correction, and execution error signals, aggregate them into a per-session quality score, and use the average performance of all models as the baseline to compute each model’s net improvement rate.
4. Provide interval estimates. Use resampling or Bayesian methods to produce confidence intervals, preventing small-sample fluctuations from being misinterpreted as genuine capability differences.

The core logic can be expressed in pseudocode as follows:

def net_improvement(sessions, model_id, baseline="global_mean"):
    # sessions: real session records already stratified by task type
    scores = []
    for s in sessions:
        if s.assigned_model != model_id:
            continue
        w = signal_weights(s)              # weighted signals: confirmation/complaint/correction/error
        q = aggregate_quality(s.signals, w)
        scores.append(q)
    model_mean = mean(scores)
    base_mean  = global_baseline(sessions, baseline)
    return (model_mean - base_mean) / base_mean

The engineering value of the net improvement rate lies in its interpretability: +5% means that, under the same task distribution, the model’s task completion quality is five percentage points above the average, with biases from task difficulty and user group differences already removed.

Sub-Metrics: Reliability Tells You More Than the Total Score

A composite net improvement rate is useful for ranking, but engineering teams care more about “where the model struggles.” Evaluation systems therefore break down multiple sub-dimensions, among which two are the most actionable:

Execution Reliability (Shell Command Execution Reliability). Long-horizon agent tasks rely heavily on command-line tools for file operations, builds, and tests. The command execution error rate directly determines whether a task can reach closure. This metric tracks tool call success rates, abnormal exit code ratios, and failure rates after retries. Models with low reliability scores tend to perform inconsistently on complex engineering tasks.

Controllability (Direction Adjustment and Correction Capability). Controllability measures whether a model, after receiving user correction instructions, accurately understands the intent and adjusts the execution path — rather than repeating mistakes or drifting off course. The key observable signal in real sessions is whether a user needs to issue a correction again after already correcting the model once. Models with poor controllability force repeated user intervention, significantly extending task duration.

These two metrics matter because they map directly to solvable engineering problems. Reliability can be improved through tool-call constraints, sandbox environment validation, and failure retry strategies. Controllability can be enhanced through instruction-following training and modeling of correction history in context. Composite scores are only useful for model selection; sub-metrics are what guide iteration.

Long-Horizon Task Performance of Very Large Open-Weight Models

Recent results from open-weight models in agent evaluation deserve attention. Taking a trillion-parameter open-weight model as an example, evaluation data from real long-horizon agent tasks reveals several trends:

• Open-weight models are now clearly stratified internally. The leading open model ranks first among open models across work, coding, and chat task categories, with a composite net improvement rate around +5.1% — roughly four percentage points higher than the previous generation of the same model family. This magnitude of improvement suggests that the combination of parameter scaling and agent-oriented post-training strategies delivers measurable gains on long-horizon tasks.
• A gap versus top closed-source models remains. Even the best open model ranks around the mid-teens among all evaluated models, indicating that closed-source models still hold an advantage in tool-call stability and long-context planning.
• Sub-metrics reveal both strengths and weaknesses for open models. Some open models have already entered the top tier among open models in execution reliability and controllability, but error accumulation control in extremely long sessions still leaves room for improvement.

Sample size is equally critical. Statistics drawn from more than 17,500 real long-horizon sessions from global users provide sufficiently narrow confidence intervals for net improvement rates, making differences in the range of +0.9% to +5.1% statistically significant rather than noise.

Practical Engineering Recommendations

For teams looking to use this type of evaluation system for model selection and optimization, the following practices are recommended:

1. Do not rely solely on the overall leaderboard. First clarify the task-type distribution of your own business (coding-heavy, office automation, or conversational), then examine the corresponding sub-leaderboards and sub-metrics.
2. Pay attention to confidence intervals. Models with similar rankings may have differences that fall within statistical error. When selecting between them, check whether their confidence intervals overlap.
3. Map sub-metrics to engineering actions. If execution reliability is low, prioritize tool-call validation and failure retry mechanisms. If controllability is low, focus on improving correction strategies in system prompts and context management.
4. Build your own small-scale session feedback loop. Platform-level evaluations provide horizontal reference points, but business-specific signals (such as success rates for your particular toolchain) can only be covered with self-collected data. Consider building a session log collection and signal extraction pipeline to establish an internal net improvement rate baseline.

Conclusion

Agent evaluation is moving from “subjective voting” toward an empirical phase of “behavioral signals plus causal inference.” Causal tracing methods convert confirmations, complaints, corrections, and execution errors in real sessions into comparable net improvement rates through randomized assignment and baseline calibration, giving capability differences statistical meaning. For developers, composite rankings serve as a first-pass filter, while sub-metrics such as reliability and controllability act as the signposts pointing to specific optimization actions. As open-weight models continue to close the gap with closed-source systems on long-horizon tasks, this evaluation methodology will evolve from a “model selection tool” into a comprehensive “iteration guide.”