Agent Evaluation Leaderboards: Evaluation Frameworks and Causal Inference Methods

Why Agent Evaluation Needs a New Paradigm

Traditional large model evaluation typically relies on static question-answer pairs or human voting to measure output quality in single-turn conversations. This approach had some reference value in early language model evaluation, but it has clear limitations when applied to agent tasks. Agent tasks usually involve multi-turn interaction, tool invocation, environment feedback, and long-horizon execution. The quality of a single response is difficult to map directly to task completion quality.

More critically, human voting mechanisms are themselves susceptible to expression style, stance preferences, and subjective judgment. When the evaluation goal is to measure whether a model “completes real tasks more reliably,” preference votes between two responses alone are insufficient to produce stable and interpretable conclusions. Evaluation methods therefore need to shift from “evaluating output text” to “analyzing task processes.”

Against this backdrop, agent evaluation systems based on real user sessions have gradually gained attention. The core idea is not to design additional closed test sets, but to dig into complete session records between real users and agents, extract key signals that reflect task success or failure and interaction quality, and then use statistical inference methods to estimate the net improvement in model capability.

Extracting Key Signals from Real Sessions

The foundational data for agent evaluation systems comes from real user sessions. These sessions are not generated specifically for evaluation; they arise naturally during actual usage. Compared with manually constructed benchmarks, real sessions cover a wider variety of task types, more complex context environments, and more authentic usage intents.

Extracting signals from these sessions requires attention to several dimensions.

The first is task confirmation signals. When a user explicitly indicates that a task has been completed or gives positive feedback on the result, this can be treated as a positive signal. Such signals directly reflect whether the task objective has been achieved and are among the most core outcome indicators in evaluation.

The second is correction and complaint signals. When a user requests revisions to agent output or expresses dissatisfaction, it usually means the agent has deviated in understanding intent, executing steps, or presenting results. The frequency and distribution of these signals can reflect a model’s performance in direction adjustment and fault tolerance.

The third is command execution reliability. In agent tasks involving code execution, file operations, or system command invocation, Shell command execution errors are a relatively objective failure signal. This signal does not depend on subjective user evaluation and can be automatically identified from execution logs, giving it high credibility in evaluation.

The fourth is interaction efficiency and direction adjustment capability. When a user needs to repeatedly correct the agent to get it back on track, even if the task is ultimately completed, it indicates insufficient controllability of the model. The evaluation system needs to account for friction in these processes, not just final outcomes.

To achieve evaluation at scale, signal extraction needs to be automated and standardized. A typical processing pipeline looks like this:

Raw session logs
    ↓
Session segmentation and task boundary identification
    ↓
Automatic annotation of multiple signal types
    ├── Task confirmation
    ├── Praise and complaints
    ├── Correction instructions
    └── Command execution errors
    ↓
Signal aggregation and feature construction
    ↓
Statistical modeling and improvement rate estimation

The key to this pipeline is that signal annotation rules must remain consistent across models. When evaluating different models, the same signal definitions and extraction logic should be used; otherwise the comparison baseline becomes inaccurate.

Applying Causal Inference Methods in Evaluation

Signals extracted from real sessions cannot be directly equated with “net improvement in model capability.” The reason is that real user sessions are not randomly assigned—task difficulty, user habits, usage scenarios, and other factors all influence the signals. Simply comparing signal means across models may mistake differences in task distribution for differences in model capability.

To control for these confounding factors, evaluation systems introduce randomized inference methods. The basic idea is to use approximate randomization statistical processing on existing session data to estimate the net effect of a given model relative to the average level of all participating models.

This method typically involves the following steps.

Step one: construct session features. For each session, extract background features such as task type, number of interaction rounds, initial instruction length, and number of tool invocations, for subsequent matching or weighting.

Step two: estimate treatment effects. Treat “using a particular model” as a treatment condition and task-completion-related signals as outcome variables, then use statistical models to estimate the effect of that treatment condition on outcomes. The “effect” here is not a simple outcome mean but a net effect after removing confounders such as task difficulty.

Step three: calculate relative improvement rate. Compare a single model’s treatment effect with the average treatment effect across all models to obtain the net improvement rate. A positive value indicates the model performs above average; a negative value indicates below average.

The calculation logic for net improvement rate can be simplified as follows:

# Conceptual example: net improvement rate estimation logic
# Actual implementation requires consideration of session features, matching strategy, and statistical significance
def estimate_net_improvement(model_sessions, all_sessions):
    # 1. Extract task features and outcome signals for each session
    # 2. Stratify or weight based on task features
    # 3. Calculate the model's average outcome on comparable task sets
    # 4. Calculate the average outcome of all models on the same task sets
    # 5. Return the improvement relative to the average level
    model_avg = adjusted_mean(model_sessions)
    baseline_avg = adjusted_mean(all_sessions)
    return (model_avg - baseline_avg) / baseline_avg

The code above is only for illustrating the concept, not an actual production implementation. Randomized inference in real evaluation systems needs to handle issues such as sample imbalance, signal sparsity, and multiple comparisons.

The core value of causal inference methods is that they advance evaluation from “whose score is higher” to “who brings measurable net improvement in real tasks.” This has more direct reference value for enterprises selecting models and optimizing agent products.

How Granular Metrics Characterize Agent Capability Differences

The composite net improvement rate is a summary metric, but looking only at summary metrics obscures capability differences across dimensions. Therefore, evaluation systems need to provide granular metrics as well.

Command execution reliability is a foundational metric in agent tasks. It measures whether a model can reduce syntax errors, path errors, and permission issues when invoking system commands or executing code. This metric is particularly critical for automated script generation, data processing, and operations tasks.

Direction adjustment and correction capability is a higher-order metric. It measures whether a model can quickly adjust its strategy after receiving user corrections and avoid repeating mistakes. This capability directly affects the completion efficiency of long-horizon tasks and also affects users’ trust in agents.

Task type coverage also deserves attention. The same model may perform exceptionally well on programming tasks but only moderately on work collaboration or open-domain chat. Evaluation systems need to break down results by task type to help users understand the applicable boundaries of models.

When analyzing granular metrics, interpretation should be combined with task features. For example, command execution errors carry higher weight in programming tasks, while user sentiment feedback plays a greater role in chat tasks. If evaluation systems can reflect these differences in metric weights, conclusions will be more instructive.

Interpreting Open-Weight Model Performance in Agent Evaluation

Open-weight models have developed rapidly in recent years. Their advantages include research reproducibility, deployment flexibility, and the ability for enterprises to perform secondary training and customization in private environments. However, open weights do not automatically mean strong agent capabilities—unified evaluation is still needed to verify them.

In real-session-driven agent evaluation, some open-weight models have already demonstrated levels approaching or even exceeding certain closed-source models. Such results can be interpreted from three angles.

From the perspective of model scale and training methods, continued increases in parameter count still have a positive effect on agent tasks, but only if training data and training objectives are more tightly integrated with actual task execution processes. A decline in language modeling loss alone does not necessarily translate directly into improved command execution reliability.

From the perspective of evaluation metrics, the magnitude of improvement in net improvement rate compared with the previous generation of models can intuitively reflect the effectiveness of training strategies. If a new model moves from near zero to a clearly positive net improvement rate, it indicates substantive progress in stability on real tasks.

From the perspective of task coverage, if a model maintains a leading position among open models across categories such as work, programming, and chat, it means its capabilities are not limited to a single scenario and have broader practical value.

It should be emphasized that ranking is only one reference for model selection. When deploying agent applications, enterprises also need to comprehensively evaluate based on their own task characteristics, performance requirements, deployment costs, and data security constraints.

Key Principles for Building High-Quality Agent Evaluation Systems

Summarizing current practical experience in agent evaluation, several key principles can be identified.

Evaluation targets should be task processes rather than isolated outputs. The core of agent capability is embodied in multi-turn interaction, tool use, and error recovery. Evaluation systems must cover these process signals.

Signal sources should prioritize objective behavioral data. Where possible, use objective data such as command execution results, task state changes, and tool invocation logs as primary signals to reduce dependence on subjective user evaluation.

Statistical methods should be able to control for confounding factors. Real data naturally contains selection bias. Evaluation systems need to use matching, stratification, or randomized inference to make comparisons between different models under comparable conditions as much as possible.

Evaluation results should be presented in layers. Composite metrics, granular metrics, and task category metrics should be provided simultaneously to avoid misleading conclusions from a single ranking.

The evaluation process should be reproducible and auditable. Data collection rules, signal definitions, statistical models, and computation pipelines should be transparent and public to facilitate verification and improvement by the research community.

As agents move from experimental prototypes to production environments, evaluation systems will continue to evolve. Future evaluation will likely further integrate environment simulation, task-level automated verification, and long-term runtime monitoring to form a more complete framework for measuring agent capability. For developers and enterprises, understanding the evaluation methods themselves is more valuable than focusing solely on a particular ranking.