Rethinking AI Agent Evaluation: Moving from Voting Rankings to Causal Attribution

1. Why AI Agents Require a Different Evaluation Approach

With the rapid adoption of generative models, comparative voting based on human preferences has become a standard evaluation method. Users simultaneously view responses from two models and select the preferred one, allowing platforms to calculate win rates or rankings based on numerous votes. While intuitive for short-form tasks like Q&A, summarization, and translation, this approach has clear limitations in agent scenarios.

Agent tasks are typically not single-turn interactions; they are long-horizon processes involving multiple steps. An agent might need to read files, execute commands, modify code, adjust its strategy based on intermediate results, and self-correct after failures. A simple “A is better than B” judgment cannot fully capture these behaviors. More importantly, a successful final result does not guarantee a stable process. A coincidentally correct output might mask command execution errors, path miscalculations, or multiple invalid operations.

Therefore, agent-focused evaluation must shift from “outcome preference” to “process signals.” Behavioral data collected during real task execution provides a much more accurate reflection of an agent’s actual working performance.

2. Key Signals in Real Agent Sessions

An evaluation approach that closely mirrors actual usage involves automatically extracting observable signals from real user-agent sessions, rather than requiring users to cast explicit votes. These signals, naturally embedded in user actions and feedback, are better suited for large-scale statistical analysis than manual scoring.

Common key signals include:

1. Task Completion Confirmation
   Users explicitly indicate the task is finished, using phrases like “That works,” “This is correct,” or “It’s running now.” This signal shows the agent has achieved its current goal.

2. Praise and Complaints
   Users express satisfaction or dissatisfaction, such as “This solution is great” or “Stop using this library.” While emotional expressions may not be precise, they reflect the user’s overall acceptance of the agent’s behavior.

3. Correction Commands
   Users redirect the agent during the session, saying things like “Don’t modify this file” or “Check the dependency version before installing.” A higher frequency of correction commands typically indicates larger deviations in the model’s initial judgment or execution strategy.

4. Command Execution Failures
   Errors occurring when the agent executes Shell commands, scripts, or build tasks. This signal is highly objective and directly reflects the model’s execution reliability in a real environment.

Together, these signals form a multi-dimensional behavioral profile. Compared to a single vote, they record the “process quality” of an agent in a real task, rather than a user’s subjective preference at a specific moment.

3. From Correlation to Causation: The Role of Randomized Inference

After collecting these signals, evaluation must address a critical question: Are the performance differences between models caused by model capability, or by factors like task difficulty, user habits, and session length?

Simply calculating the success rate or average score for each model makes it highly susceptible to confounding factors. For instance, a model might be used more frequently by advanced users for complex tasks, resulting in a seemingly low absolute success rate despite strong actual capability. Conversely, another model might primarily handle simple tasks, yielding higher scores that do not necessarily indicate superior performance.

To address this, randomized inference can be employed. When real user requests enter the system, tasks are randomly assigned to different models. Consequently, the distribution of requests faced by each model tends to become consistent across a large sample size. Subsequently, statistical models are used to estimate each model’s net improvement rate relative to the overall average.

The net improvement rate can be understood as the improvement a model achieves compared to the average level of all models, after stripping away common factors like task difficulty, session length, and user differences. Rather than a simple “success rate ranking,” it serves as a form of causal estimation. For example, if a model shows a 5% improvement over the average, it means that under equivalent task conditions, its behavioral outcomes are 5 percentage points better than the baseline.

This method is particularly suitable for long-horizon agent tasks. Because feedback signals in agent sessions are often sparse and unevenly distributed, directly comparing absolute scores can easily lead to distortions. Causal inference allows these differences to be accurately attributed to the models themselves.

4. The Performance of Open-Weight Models in Real Tasks

In the current model ecosystem, open-weight models are attracting increasing attention from developers. Unlike closed-source models accessible only via APIs, open-weight models allow developers to self-host, fine-tune, and integrate them independently. For enterprises, this controllability helps meet requirements for data compliance, private deployment, and cost optimization.

However, the capability of open-weight models in agent tasks cannot be solely judged by general conversational scores. A model might perform well in general Q&A but lag significantly in command execution, multi-step planning, and error recovery. Therefore, evaluation based on real agent sessions is particularly crucial for open-weight models.

According to existing public evaluation results, some open-weight models have already demonstrated strong competitiveness in long-horizon agent tasks. Taking a recent large-scale parameter model as an example, its comprehensive net improvement rate has increased significantly compared to its previous generation, particularly excelling in the following dimensions:

• Command Execution Reliability: Maintains a low error rate across multiple Shell command invocations, reducing interruptions caused by syntax, path, or dependency issues.
• Direction Adjustment Capability: Quickly understands and adjusts subsequent strategies when users issue correction commands, rather than repeating errors.
• Long-Task Stability: Minimizes goal loss and context drift in sessions involving multiple subtasks and file operations.

These dimensions align more closely with the actual concerns of developers than single-turn response quality. For scenarios requiring the construction of automated programming assistants, data processing pipelines, or operations script generation tools, these metrics are often more valuable than general leaderboards.

5. Applying This Evaluation to Model Selection

When developers or enterprises select a foundational model for agents, they should not focus solely on a single total score. Instead, they should integrate process-oriented metrics into their evaluation system based on their specific application scenarios. This can be approached from the following angles:

1. Define Task Types

First, identify the primary tasks the agent will handle. Examples include:

• Code generation and debugging
• Shell command generation and execution
• File reading, searching, and modification
• Multi-turn data analysis
• Automated operations script writing

Different tasks require different model capabilities. Programming scenarios prioritize command execution reliability and error recovery, while chat scenarios may focus more on semantic understanding and multi-turn memory.

2. Build an Internal Evaluation Set

Public leaderboards can serve as an initial screening tool, but the final choice should be based on proprietary data. You can collect real user sessions or construct simulated task sets covering common workflows and failure branches. During evaluation, prioritize recording the following:

task_id           Task identifier
model_id          Model used
session_length    Number of session turns
user_confirmed    Whether the user confirmed completion
correction_count  Number of corrections
command_error     Number of command execution errors
success           Final success status

These fields support the subsequent calculation of net improvement rates.

3. Adopt Randomized Routing

When conducting online comparisons, randomly assign user requests to different models whenever possible. If conditions are limited, you can shuffle task sequences in offline evaluations to prevent a model from consistently handling a batch of tasks with similar difficulty levels. Randomized routing is the foundation for the validity of subsequent causal inferences.

4. Focus on Trends Over Single-Point Scores

Model evaluation results will fluctuate with version updates. It is recommended to continuously record key metrics and observe trends in dimensions like command execution reliability and direction adjustment capability. A single ranking is easily affected by sample variance, whereas long-term trends better reflect the trajectory of model iteration.

6. Limitations and Recommendations

Although evaluation methods based on real sessions and causal inference are closer to actual usage, certain limitations must be noted:

Sample Coverage Bias
The distribution of real user sessions may be unbalanced. For example, an overrepresentation of programming tasks can weaken the evaluation stability for general chat scenarios. Therefore, statistics should be categorized by task type rather than relying solely on a comprehensive score.

Signal Extraction Noise
User expressions can be ambiguous. “It’s okay” does not necessarily mean success, and “This is wrong” may not pinpoint the specific erroneous step. The signal extraction system requires continuous optimization of its rules and models to improve accuracy.

Limited Offline Capability Coverage
This type of evaluation primarily reflects model performance in interactive tasks. It has limited coverage for scenarios like batch inference, offline data processing, and API integration. It still needs to be combined with internal stress testing, security evaluations, and cost assessments.

Unsuitable for Short-Term, Small-Sample Judgments
Causal inference requires a sufficient sample size to produce stable estimates. If a developer conducts only a few dozen tests, the results may be heavily influenced by noise. In such cases, focus more on failure case analysis and behavioral visibility rather than over-interpreting the net improvement rate.

7. Conclusion

AI agent evaluation is shifting from simple comparative voting to methods focused on real behavior and causal attribution. By collecting process signals—such as task confirmations, correction commands, and command execution errors—and introducing randomized inference, we can more objectively estimate a model’s net improvement effect in long-horizon tasks.

For developers, this evaluation provides a new perspective for model selection: Instead of merely asking “Which model gives better answers,” it attempts to answer “Which model is more likely to consistently achieve goals in real tasks.” When building enterprise-grade agent applications, the value of this question is often far greater than the outcome of a single conversation.