From Pairwise Voting to Causal Tracing: The Evolution of AI Agent Evaluation

Over the past two years, the dominant paradigm for evaluating large language models has been the pairwise voting leaderboard: presenting two anonymous model responses to users, asking them to choose the better one, and using preference statistical models to convert massive vote counts into scores and rankings. While highly efficient for single-turn Q&A scenarios, the foundation of this paradigm begins to crack as models evolve from “answering questions” to “executing tasks”—autonomously calling tools, writing and running shell commands, and working continuously for tens of minutes. This article systematically breaks down the motivation, signal design, statistical methods, and practical implementation of a new generation of agent evaluation systems.

1. Why Pairwise Voting Fails for Agent Evaluation

• Inability to compare side-by-side: Agent tasks involve long-horizon, multi-turn real-world interactions. Users only let one model complete their task; they cannot afford to let two models each spend half an hour executing the same task just for comparison. Agent scenarios inherently lack the sample sources for “head-to-head” matchups.
• Decoupling of preference and success: Voting is easily swayed by superficial factors like tone, formatting, and length. In real tasks, however, users care about hard results: “Did the command run successfully? Was the file modified correctly? Was the result delivered?”
• Skewed query distribution: Queries on anonymous voting leaderboards tend to be short and Q&A-oriented. The true value of an agent lies in complex, real-world tasks with engineering context—two distributions that barely overlap.
• High variance in results: Agent actions depend on environmental states and randomness. Running the same prompt twice may yield completely different trajectories, introducing massive noise into a single vote.

Conclusion: Agent evaluation must change its data source. Instead of constructing controlled experiments for user voting, we must directly “mine” real user agent sessions and infer model capabilities from actual task outcomes.

2. Signal Design: Extracting Computable “Evidence” from Real Sessions

Real sessions are a goldmine for evaluation signals, but raw logs cannot be directly compared. They require a signal extraction pipeline. In engineering practice, a hybrid architecture of “rules first, model judgment as fallback” is typically adopted to transform unstructured sessions into structured metrics. Four key types of signals are typical:

1. Explicit Task Confirmation: The user provides explicit confirmation like “done” or “no problem” at the end of a task. This is the most direct evidence of task success and is usually assigned the highest weight.
2. Praise and Complaints: Positive and negative subjective feedback. Complaints generally carry more weight than praise because users tend to accept successes silently but express failures vocally.
3. Correction Commands: When a user says, “No, it should be…” or “Start over, I meant…”, it indicates the model deviated from the task direction. More corrections mean worse instruction-following ability.
4. Shell Command Execution Errors: A command’s exit code and error logs are objectively verifiable failure signals. The command execution success rate can form a specific metric for “command execution reliability,” completely independent of subjective annotation.

A conceptual signal extraction code snippet is shown below:

# Conceptual code: Extracting evaluation signals from a single agent session
def extract_signals(session):
    sig = {"task_confirmed": 0, "praise": 0, "complaint": 0,
           "correction": 0, "cmd_error": 0, "cmd_total": 0}
    for turn in session.turns:
        # 1) Rule layer: Prioritize structured fields for reproducibility
        if turn.tool_call and turn.tool_call.type == "shell":
            sig["cmd_total"] += 1
            if turn.tool_call.exit_code != 0:
                sig["cmd_error"] += 1
        # 2) Rule layer: Common expressions for explicit confirmation
        if re.search(r"(done|finished|no problem|exactly)", turn.user_text):
            sig["task_confirmed"] += 1
        # 3) Discrimination layer: Intent classifier for subjective feedback (requires periodic calibration with human-annotated datasets)
        intent = intent_clf(turn.user_text)
        if intent == "praise":      sig["praise"] += 1
        elif intent == "complaint": sig["complaint"] += 1
        elif intent == "correction": sig["correction"] += 1
    return sig

A few engineering best practices: Do not delegate issues solvable by regex and structured fields (like exit codes and tool receipts) to probabilistic models. The discrimination layer must regularly report precision and recall to prevent signal drift. Session logs must be desensitized (removing API keys and PII), retaining only aggregated metrics for statistics to meet privacy compliance requirements.

3. Causal Tracing: Estimating “Net Improvement Rate” via Randomized Inference

With session-level signals, one crucial step remains: how do we attribute “signal differences” to “model capabilities” rather than confounding factors like task difficulty or user proficiency?

The intuitive approach is to simply calculate the average score, but this introduces severe bias: if a model happens to receive more simple tasks, its average score inflates artificially. The solution borrows from Randomized Controlled Trials (RCTs): randomly assigning traffic at the gateway level so that each model faces a statistically comparable distribution of tasks and user groups, then estimating the “Average Treatment Effect” (ATE). This randomized causal inference approach can be termed “Causal Tracing.”

The net improvement rate can be formalized as:

Net Improvement Rate(m) = E[Y | Session executed by model m] − E[Y | Session executed by all evaluated models]

Where Y is the session-level outcome score (synthesized from the signals in Section 2). It answers the question: if a random task is assigned to model m compared to the “average level,” what is the expected percentage point gain in the outcome? For example (using hypothetical numbers): if the task confirmation rate across all sessions is 57%, and model A’s is 62%, its net improvement rate is +5 percentage points. If the previous generation’s rate was 58.5%, the inter-generational improvement is 3.5 percentage points.

Statistically, two more things are needed. First, confidence intervals: use Bootstrap resampling on session-level outcomes to provide interval estimates—the narrower the interval, the more credible the ranking. Second, significance testing: when the confidence intervals of two models overlap significantly, their ranking differences should not be over-interpreted.

# Conceptual code: Net improvement rate estimation and confidence intervals under random traffic splitting
def outcome_score(sig):
    return (2 * min(sig["task_confirmed"], 1) + sig["praise"]
            - 2 * sig["complaint"] - sig["correction"]
            - (sig["cmd_error"] / max(sig["cmd_total"], 1)))
def net_improvement(model_id, sessions, n_boot=10000):
    # Prerequisite: Model assignment is randomized by the gateway, ensuring comparable task distributions
    y_m   = [outcome_score(s.signals) for s in sessions if s.model == model_id]
    y_all = [outcome_score(s.signals) for s in sessions]
    point = mean(y_m) - mean(y_all)
    boots = [mean(sample(y_m, len(y_m))) - mean(sample(y_all, len(y_all)))
             for _ in range(n_boot)]
    return point, (percentile(boots, 2.5), percentile(boots, 97.5))

Compared to voting leaderboards: preference statistical models aggregate “relative preferences” in abstract score units. Randomized inference directly estimates the “outcome difference” in percentage points. The latter can answer, “If I switch to this model, how much will my task success rate change?” providing significantly stronger explanatory power for enterprise model selection.

4. Metrics System: Multi-Dimensional Slices Beyond an Aggregate Score

A single aggregate net improvement rate is insufficient for guiding procurement. In practice, the metrics system typically follows a “1+N” structure:

1. Aggregate Net Improvement Rate: The overall outcome difference across all sessions, serving as the primary sorting key for the leaderboard.
2. Scenario Dimensions: Sub-leaderboards categorized by task type, such as office collaboration, software development, and daily conversation. Model capabilities are often highly uneven; a model ranking in the middle overall might top the chart for open-weight models in coding scenarios.
3. Capability Dimensions: Derived granular metrics, such as “Command Execution Reliability” (proportion of successful command executions, defining the engineering baseline), “Direction Adjustment and Correction Capability” (measured by task confirmation rate after corrections), and “Task Completion Rate” (proportion of sessions with explicit confirmation).
4. Open-Weight Sub-Leaderboards: Open-weight models allow for private deployment and fine-tuning, placing them in a completely different procurement decision pipeline than closed-source API-only models. Comparing them separately is fairer and more practical.

Three principles for interpreting leaderboards: Look at intervals, not just point estimates; match scenarios rather than relying solely on aggregate rankings; and consider sample size—conclusions based on tens of thousands of real, long-horizon sessions are vastly more stable than small samples of hundreds of votes.

5. How Enterprises Can Build an Agent Evaluation System

Public leaderboards are a public good, but enterprises need “private leaderboards” tailored to their own business. The implementation path can be broken down into five steps:

1. Instrumentation and Session Collection: Record the model version, task description, per-turn user input, tool call receipts (including exit codes), and final state within the agent framework. This is the foundation of all evaluations.
2. Define Business-Specific Signals: Layer business-specific signals on top of the four general signals, such as “code passes unit tests” or “ticket successfully transitioned.” Signal definitions should be documented with positive and negative examples.
3. Establish Random Traffic Splitting: Randomly assign real traffic to candidate models at the gateway or feature flag layer to ensure comparable task distributions. This is a prerequisite for causal attribution and cannot be skipped.
4. Compute and Visualize: Calculate net improvement rates and confidence intervals, slice them by scenario, and build dashboards.
5. Regression Monitoring and Alerting: Incorporate the net improvement rate into version release gates. If a new version shows a statistically significant drop in key scenarios, automatically block the release and notify the responsible team.

If the cost of building proprietary compute and operations is too high, this pipeline can be assembled using general cloud components. Taking Baidu AI Cloud as an example, you can use its Log Service for session instrumentation collection and retrieval, Object Storage to沉淀 raw session samples and annotation data, Container Platform to run batch signal extraction and statistical computing, and Cloud Monitoring to handle metric anomaly notifications. The overall architecture can be abstracted into four layers: “Collection → Extraction → Estimation → Monitoring.”

6. Common Pitfalls and Best Practices

Common Pitfalls: Obsessing over aggregate rankings while ignoring scenario slices; ignoring task distribution drift post-launch and treating one-time evaluations as long-term conclusions; failing to recalibrate after upgrading the discrimination model, leading to incomparable metrics due to signal definition changes; over-interpreting small samples where confidence intervals are too wide to be meaningful; overfitting optimization towards public leaderboard signals, which ultimately damages real-world task performance.

Best Practices Checklist:

• Hybrid signal extraction architecture prioritizing rules with model fallback.
• Gateway-level random traffic splitting to ensure comparability.
• All point estimates must be accompanied by confidence intervals.
• A multi-dimensional metrics system combining aggregate, scenario, and capability scores.
• Integrate evaluation into release gates to form an “evaluate → discover → fix” closed loop, and rotate evaluation datasets periodically to maintain freshness.

Conclusion

Agent evaluation is undergoing a paradigm shift from “human preference” to “factual causality.” The data source is shifting from constructed voting samples to real sessions; the evaluation target is shifting from response quality to task outcomes; and the statistical method is shifting from preference models to randomized causal inference. For developers, understanding the value of this methodology goes beyond merely “reading leaderboards”—it’s about implementing the same principles into your own systems. After all, no external leaderboard is as honest as the confidence interval hiding within your own business data.