Evaluation Paradigms and Causal Inference Methods for AI Agent Leaderboards

1. Why Agent Evaluation Cannot Rely Solely on Voting

Early LLM evaluations widely adopted the “pairwise voting” approach: users choose the better output between two models, and rankings are calculated based on win rates. While this method holds some reference value for conversational models, it exhibits significant biases when applied to agent tasks.

Agent tasks are typically characterized by multi-turn interactions, long execution horizons, and heavy reliance on tools. A single task might involve environment comprehension, command execution, file operations, error recovery, and result verification. Whether the task is ultimately completed often matters more than the fluency of a single-turn response. Traditional voting struggles to capture these process-oriented signals and tends to amplify users’ subjective preferences from short conversations.

More critically, the sample distribution collected through voting mechanisms is highly skewed. Popular tasks and top-tier models receive more exposure, making it difficult for new models or long-tail tasks to accumulate sufficient data. Consequently, rankings might reflect “traffic bias” rather than actual “capability differences.” This necessitates an evaluation system that incorporates observation methods closer to real-world usage scenarios alongside more robust statistical inference techniques.

2. Extracting Valid Signals from Real Conversations

To mitigate manual voting bias, a more rational approach is to directly mine conversation logs between real users and agents. Real conversations contain a wealth of automatically recognizable process signals, offering much finer granularity than a simple thumbs up or down.

1. Task Completion Confirmation

When users provide responses like “that works,” “it ran successfully,” or “this is exactly what I wanted,” it indicates the agent has essentially completed the task. Conversely, responses like “that’s wrong,” “still getting an error,” or “I needed a different result” indicate failure. The evaluation system can transform these text patterns into structured labels to measure task success rates.

2. Course Correction and Retry Behaviors

During long-horizon tasks, users frequently need to adjust the agent’s direction midway. For instance, a user might initially say “process this with Python,” and subsequently add, “don’t use pandas, write native code instead.” These correction paths reflect the model’s adaptability to changing instructions, commonly known as the “steerability” metric.

3. Command Execution Errors

Many agents accomplish tasks through Shells, code interpreters, or other execution environments. Command syntax errors, permission issues, and missing dependencies that occur during execution are captured in the session logs. Compared to pure text scoring, command execution errors serve as a more objective capability signal.

4. Result Confirmation and Remediation

Whether subsequent errors occur after a user confirms a result, and whether the agent can automatically fix itself upon discovering an error, are crucial observation dimensions. A one-time success that cannot be replicated is entirely different from an execution that is stable, reproducible, and self-verifying in an engineering context.

The system can extract the above signals using a combination of rules and lightweight classifiers. Rules handle high-precision recognition of explicit expressions, such as confirmation phrases containing specific keywords. Classifiers process more implicit expressions, like “almost there, but this part needs a slight tweak.” These two methods complement each other, maintaining high accuracy when processing real conversations at scale.

3. Replacing Simple Win Rates with Causal Inference

Once signal extraction is complete, a more complex challenge arises: different models face varying task difficulties. If Model A only serves simple tasks while Model B frequently receives complex ones, directly comparing their success rates is unfair.

1. Basic Concept of Randomized Inference

A viable approach is to employ a randomized assignment strategy within real traffic. When a user initiates a session, the system assigns the task to different models based on certain probabilities, ensuring that each model faces a similar task distribution over a given time window. This is analogous to A/B testing in recommendation systems.

Subsequently, the evaluation system does not simply compare “who has a higher success rate.” Instead, it estimates the net improvement rate each model brings relative to the average level of all models. The advantage here is that the ranking expresses “how much this model improved task success rate relative to the baseline,” rather than an absolute vote count influenced by sample size.

2. Meaning of the Net Improvement Rate

The net improvement rate measures the incremental effect brought by a model’s intervention. If the average success rate of all agents on a certain task type is 60%, and a specific model achieves 63%, its relative improvement rate is approximately:

(63% - 60%) / 60% ≈ +5.0%

This metric filters out fluctuations caused by macroscopic changes in task difficulty. Even if users submit generally harder tasks over a period, as long as all models are randomly assigned, the average baseline will shift synchronously. The relative differences between models remain interpretable.

3. Fine-Grained Capability Breakdown

Beyond the comprehensive net improvement rate, the evaluation system can extract multiple process metrics. For example, Shell command execution reliability focuses on the proportion of runnable commands a model generates; steerability focuses on whether the model can quickly adjust and continue the task after receiving a corrective instruction; task completion efficiency focuses on the number of turns and time elapsed from start to user confirmation.

This breakdown transforms the ranking from a mere leaderboard into a diagnostic tool. Development teams can use it to determine whether the model lacks planning capabilities, suffers from unstable tool calls, or responds too slowly to user feedback.

4. Performance Differences of Open-Weight Models in Agent Scenarios

Open-weight models allow developers to deploy locally or further fine-tune, making them naturally attractive for enterprise data security, offline inference, and customization scenarios. Historically, open-weight models performed well on general conversational leaderboards but lagged significantly behind closed-source models in complex agent tasks.

As model scale and training strategies evolve, this gap is narrowing. Some open-weight models can now stably plan over long horizons, invoke tools, and retry upon failure. Evaluation results derived from real user conversations, rather than lab scripts, better reflect their engineering viability in complex environments.

Take tool invocation as an example. Open-weight models must simultaneously possess capabilities in code generation, command execution, error interpretation, and context maintenance. Whether a model can read an error output after a failed Shell command, locate the cause, and provide a corrected command is critical to measuring its reliability. Evaluations can stratify and track statuses such as “first command success,” “second correction success,” and “abandoned after multiple corrections” to determine if the model is engineering-maintainable.

5. How Enterprises Can Leverage These Evaluation Results

1. Focus on Task Domains Rather Than Overall Rankings During Selection

Different business scenarios have vastly different requirements for agents. Data analysis tasks rely heavily on code execution reliability, while knowledge QA tasks focus on information retrieval and citation accuracy. Overall rankings should only serve as an initial filter. Enterprises must further examine specific task domains and process metrics.

2. Establish an Internal Shadow Evaluation Set

Public leaderboards can serve as external references, but enterprises should ultimately build their own evaluation sets. Standardize historical real tickets, internal scripts, and common operations commands into tasks, run candidate models in identical environments, and record completion rates, time consumed, and manual intervention counts. This internal data aligns much closer to business value than general leaderboards.

3. Monitor Post-Deployment Process Metrics

After an agent goes live, teams should not only check if the final answer is correct. Enterprises can log user correction frequency, command failure rates, and tool invocation timeout rates. These metrics, derived from real conversations, can be automatically aggregated via logging services and analytics platforms. Continuously monitoring this data helps proactively detect model degradation or issues caused by environment changes.

6. Development Direction of the Evaluation System

Agent evaluation is shifting from “how good is the answer” to “was the task reliably completed.” This requires the evaluation mechanism to possess three characteristics: first, collecting process signals from real interactions rather than relying on static test questions; second, possessing randomization and causal inference capabilities to reduce traffic bias and sample skew; third, being able to break down engineering-oriented process metrics, giving the evaluation results diagnostic value.

For developers and enterprises, understanding the evaluation methodology is more important than memorizing a specific ranking. Only by clarifying the statistical definitions and data sources behind the metrics can one determine which conclusions are transferable to their own business environments. In the future, as agents take on increasing responsibilities in software engineering, data analysis, and IT operations automation, evaluation methods based on real conversations and causal inference will gradually become the foundational infrastructure for model selection and quality assurance.