How Agent Evaluation Leaderboards Are Designed: Causal Inference in Practice

1. Why Agent Evaluation Cannot Rely on Voting Alone

In the early evaluation of language model capabilities, a common approach was to have users vote on their preference between two responses. This method offers some value for subjective dimensions such as conversational fluency, readability, and style alignment. However, it often falls short when applied to agent tasks.

The core of an agent task is not whether the response “sounds human” but whether the task actually gets completed. A response may read very naturally yet frequently fail when executing shell commands, modifying configurations, calling APIs, or handling files. Once users enter real task scenarios, they quickly sense whether a model is reliable. Therefore, preference voting at the conversational level alone can hardly reflect an agent’s usability in actual workflows.

More critically, agent tasks often have long-horizon characteristics. A complete task may involve multiple rounds of interaction, repeated command execution, result validation, and path adjustments. If traditional evaluation only captures a single round of question and answer, it loses a great deal of process information, causing evaluation results to deviate significantly from real experience.

As a result, agent-oriented evaluation methods need to shift from “static response quality” to “dynamic task performance,” and from “subjective user preference” to “process signals and outcome signals combined.”

2. How Real Sessions Become Evaluation Data

For an agent evaluation leaderboard to be credible, the first problem to solve is data sourcing. One viable path is mining real user agent sessions rather than relying solely on closed test sets or manually constructed tasks.

The advantage of real sessions lies in task diversity. Users pose different types of questions: some focus on code generation and debugging, some on document processing, some require the model to operate in a command-line environment, and others involve multi-step information retrieval. This diversity is difficult to fully simulate with a fixed test set.

At the same time, real sessions preserve the dynamic process of a task. For example, whether a command returns an error after execution, whether the user issues a correction instruction, and whether the user ultimately confirms completion—all of these process signals can be recorded. Compared with looking only at the final output, this kind of process data can characterize model behavior more precisely.

Of course, using real sessions also requires handling privacy, noise, and imbalance issues. Typically, sessions need to be anonymized and filtered by task, removing sessions that are too short, have no clear goal, or were interrupted midway, so that samples entering the evaluation pipeline have a basic task structure.

3. What Key Signals to Extract from Sessions

In real agent sessions, some signals carry more behavioral value than a “user like.” Below are several common and important categories of extractable signals.

3.1 Task Completion Confirmation

A user explicitly stating that a task is done—for example, “that works,” “it’s running now,” or “problem solved”—is a positive signal. Note that such confirmations need to be distinguished from polite closing remarks. Some users may end a session with “let’s leave it here for now” or “I’ll take another look” even when the task is not complete. Therefore, extracting confirmation signals requires contextual judgment.

3.2 Praise and Complaints

When users express praise or complaints during a session, it directly reflects model performance. However, these signals tend to be sparse, and expression habits vary greatly across users. In practice, they are usually treated as auxiliary signals rather than the sole basis for judgment.

3.3 Correction Instructions

When a user issues a correction to the model—for example, “don’t use that parameter,” “try a different approach,” or “the last command was wrong”—it is an important negative signal. Correction instructions typically mean the model deviated in direction, parameters, or execution path. A model that is frequently corrected may have poor efficiency and controllability even if it eventually completes the task.

3.4 Shell Command Execution Errors

In command-line agent scenarios, shell command execution errors are very direct objective signals. Errors such as syntax mistakes, insufficient permissions, nonexistent paths, and missing dependencies can all be captured by the system. Compared with text-based evaluations, command execution errors are harder to mask with surface-level language fluency.

Combining the above signals allows for building an evaluation dimension that is closer to the task execution process, rather than relying on a single score.

4. Using Causal Inference to Estimate Net Improvement Rate

A difficult problem in agent evaluation is that task difficulty is naturally imbalanced across models. If some models only handle easier tasks, a lower failure rate alone cannot directly prove stronger capability.

To mitigate this issue, randomized inference methods can be used to estimate each model’s net improvement rate relative to the average level. The basic approach is as follows:

1. Randomly assign user sessions to different models;
2. Record outcome signals in each session, such as success, failure, number of corrections, and number of command errors;
3. Use a statistical model to compute each model’s expected performance under a similar task distribution;
4. Compare model performance against the overall average across all models to obtain the net improvement rate.

The “net improvement rate” here can be understood as: after removing the effects of task difficulty and random fluctuation, the number of additional successes or reduced failures per hundred tasks for a given model relative to the average level. It is not a simple difference in success rate but a marginal effect obtained through inference.

Below is a simplified illustration of how to estimate relative performance by session outcome using Python:

from sklearn.linear_model import LogisticRegression
import numpy as np
# Example data
# X: [task difficulty feature, model index]
# y: 1 indicates task success, 0 indicates failure
X = np.array([
    [2, 0], [1, 0], [3, 0],
    [2, 1], [3, 1], [4, 1],
    [2, 2], [3, 2], [1, 2],
])
y = np.array([1, 0, 1, 0, 1, 0, 1, 1, 0])
model = LogisticRegression()
model.fit(X, y)
# The portion of the model coefficients corresponding to "model index"
# can serve as a reference for relative performance
print(model.coef_)

A production system would be far more complex than the example above, but the core logic is the same: through regression or hierarchical inference, confounding factors such as task difficulty are stripped away to estimate the model’s own net contribution.

5. How Fine-Grained Metrics Reflect Agent Capabilities

Beyond overall rankings, fine-grained metrics help developers locate problems more clearly. For example:

• Command execution reliability: Measures the proportion of shell commands generated by the model that are successfully executed. A low value for this metric usually indicates that the model makes too many assumptions about the environment, handles paths inconsistently, or generates syntax that is not rigorous enough.
• Steerability: Measures whether the model can quickly adjust direction after receiving a user correction instruction. This metric looks not only at the final result but also at the cost of mid-course corrections.
• Task completion efficiency: Combines turns, command count, and successful outcomes to assess whether the model completes tasks at a relatively low cost.

These metrics carry different significance for different enterprises. For example, an agent used for internal operations cares more about command execution reliability, while a coding assistant may focus more on steerability and long-horizon task stability. Through fine-grained metrics, teams can choose the model that better fits their own business scenarios.

6. Open-Weight Models and Long-Horizon Task Performance

In recent years, open-weight models have gradually become an important option for agent applications. Their advantages include deployment flexibility, support for private deployment, and ease of further fine-tuning and integration. However, the overall performance of open-weight models in agent tasks often needs to be validated against real long-horizon data.

In long-horizon tasks, a model must not only complete single-step operations but also maintain consistent goal memory, reasonable step planning, and stable command generation throughout the entire session. Such tasks place higher demands on context management, error recovery, and tracking of user intent.

If evaluation data comes mainly from short conversations, model capabilities may be overestimated. Real user sessions contain a large number of mid-course corrections, environment changes, and retry logic—scenarios that better test whether a model possesses reliable agent capabilities.

7. How to Interpret the “Net Improvement Rate”

A positive net improvement rate indicates that the model outperforms the average under comparable tasks; a negative value indicates it falls below average. For example, a model with a net improvement rate of +5.1% can be understood as having a probability of successfully completing tasks that is about 5.1 percentage points higher than the average after controlling for task difficulty.

It is important to note that this metric is a relative value, not an absolute success rate. It is better suited for horizontal comparison and trend observation than for direct conversion into an “actual task success rate.” When selecting models, enterprises should also consider their own task distribution, runtime environment, and failure costs.

In addition, the net improvement rate is sensitive to sample size. Only after accumulating enough real sessions—especially long-horizon agent task sessions—do estimation results have statistical stability. Task types that are too concentrated, too few samples, or high annotation noise can all cause ranking fluctuations.

8. What Evaluation Capabilities Enterprises Should Focus On When Deploying Agents

When building production-oriented agent applications, enterprise users should not focus on a single composite score alone. It is advisable to evaluate model capabilities from the following perspectives:

1. Task type alignment: Whether the evaluation tasks are similar to the enterprise’s own business, such as workflow automation, code generation, or operations assistance;
2. Availability of process metrics: Whether process data such as command execution error rates and correction counts can be obtained;
3. Long-horizon stability: Whether the model can maintain consistent task goals across multiple rounds of interaction;
4. Environment adaptability: Whether the model performs stably in containers, cloud hosts, script execution, and similar environments;
5. Observability: Whether failure causes can be traced—distinguishing planning errors, parameter errors, and environment understanding errors.

These capabilities often determine whether an agent application is usable far more than language expression quality alone. For enterprises deploying on the cloud, general cloud capabilities such as container platforms, logging services, and monitoring and alerting can also be combined to build a complete agent operation and evaluation loop.

9. Conclusion

Agent evaluation is moving from “conversational preference” to a new stage that combines “task outcomes and process signals.” A trustworthy evaluation system requires real task data, reasonable process signals, rigorous statistical inference, and interpretable fine-grained metrics working together.

For developers and enterprises, understanding the evaluation methodology matters more than memorizing a particular ranking. Only by clarifying the data sources, estimation logic, and applicable boundaries behind the metrics can they be turned into effective inputs for model selection, tuning, and deployment.