Agent Evaluation’s New Paradigm: From Human Voting to Causal Inference on Real-World Tasks

From Subjective Voting to Objective Signals: A Paradigm Shift in Agent Evaluation

As AI applications move rapidly into production, agents are taking on complex tasks such as code generation, data analysis, and system operations. Yet objectively and accurately evaluating their real capabilities remains a core challenge for developers. Traditional evaluation methods rely on humans voting on or scoring model responses in pairs. This approach is not only expensive and inefficient, but also prone to subjective bias: human reviewers may be influenced by surface-level fluency while overlooking whether a task was actually completed successfully.

More importantly, traditional evaluation struggles to cover long-horizon, multi-step tasks in real work environments. An agent may need to perform dozens of tool calls, handle exceptions, and adjust its strategy based on user feedback. These complex behaviors cannot be quantified by simply asking “which response is better.” In response to this pain point, agent evaluation is undergoing a profound methodological shift: moving from subjective human voting toward objective signal mining and causal inference based on real user sessions.

Multi-Dimensional Signal Extraction: Capturing Key Evidence of Task Execution

The core idea behind the new generation of evaluation frameworks is simple: instead of constructing artificial test questions, directly mine real user-agent interaction sessions. These session datasets naturally contain users’ genuine needs, evolving intentions, and feedback behaviors in real workflows, giving them higher ecological validity than artificially constructed evaluation sets.

Within these large-scale session datasets, the evaluation system needs to automatically identify and extract key signals that reflect an agent’s “net improvement rate.” Based on deep analysis of real interaction patterns, the following four categories of signals have proven to be the most discriminative and reliable.

Task confirmation signals. When a user clearly approves an agent’s proposed solution or execution plan—for example, “go with this plan” or “yes, continue”—it indicates that the agent’s understanding aligns closely with user expectations. These signals effectively measure an agent’s performance during requirement parsing and solution planning, serving as a foundational guarantee of task success.

Praise and complaint signals. Users’ emotional expressions often directly reflect task completion quality. Praise signals, such as “well done” or “that was fast,” indicate that the agent exceeded expectations in a particular step. Complaint signals, such as “this is not what I wanted” or “why did it fail again,” mark failure modes. The evaluation system needs to use sentiment analysis models to accurately distinguish these two types of signals and track their contextual positions, thereby locating specific weak points in long task chains.

Correction instruction signals. When users notice that an agent is deviating from the expected path, they issue corrective instructions, such as “no, use plan B instead” or “check the logs before retrying.” These signals contain two critical pieces of information: first, the agent made a cognitive or execution error in a previous step; second, whether the agent can quickly adjust its strategy after receiving feedback and avoid repeating the mistake. Therefore, both the frequency of correction instructions and the degree of behavioral improvement after correction are important quantitative dimensions for measuring an agent’s steerability and adaptability.

Shell command execution error signals. For agents handling engineering tasks, command execution success directly determines task outcomes. The evaluation system needs to extract all shell commands and script invocation records from sessions and identify typical failure patterns such as syntax errors, insufficient permissions, nonexistent paths, and missing dependencies. Error rates reflect not only an agent’s understanding of the system environment, but also its robustness in real-world operations scenarios.

Causal Tracking: Using Randomized Inference to Remove Confounding Factors

Simply counting the frequency of these signals is far from sufficient. Because users differ greatly in task complexity, clarity of expression, and domain expertise, directly comparing signal frequencies across agents introduces serious selection bias. For example, a model that is good at handling simple queries may receive more praise signals, but this does not mean it excels at complex tasks.

To address this challenge, the evaluation framework introduces a causal tracking methodology that uses randomized inference to estimate each model’s net improvement rate relative to the average. The core idea is analogous to a randomized controlled trial: a large number of real sessions are randomly assigned across the counterfactual space of different models. By statistically modeling the probability of signal improvement under the same task background, the framework strips away confounding factors such as task difficulty and user habits.

In practice, causal tracking first benchmarks all participating models on the same batch of sessions and computes the overall mean for each signal dimension. It then uses non-parametric statistical methods, such as permutation tests and bootstrap resampling, to construct null distributions and determine whether a specific model’s signal performance is significantly better than what would be expected under random assignment. The resulting net improvement rate is a debiased relative metric. It answers not “how much praise did this model receive,” but “if all tasks were executed by this model, by what percentage would success signals increase compared with the average.”

Case Study: Evaluation Rankings for an Open-Weight Model

Taking an open-weight model as an example, the latest evaluation results place it first in the open-weight category and 13th among all participating models. It also achieved the highest open-weight rankings in the “work,” “coding,” and “chat” task categories. These results are based on more than 17,500 real long-horizon agent sessions covering code development, data processing, conversational interaction, and other diverse scenarios.

Notably, the model achieved a combined net improvement rate of +5.1%, an increase of 4.2 percentage points over its previous version’s +0.9%. This jump did not come from a single breakthrough metric, but from the co-evolution of multiple capabilities. In particular, the model stood out in two sub-metrics: Bash command execution reliability and steerability.

The improvement in Bash reliability can be attributed to enhancements in system calls, error diagnosis, and path resolution. In observed sessions, the model was better able to identify the root cause of command failures, such as missing environment variables or file permission conflicts, and proactively propose fixes rather than repeatedly attempting the same failing command. The improvement in steerability was reflected in the model’s ability to quickly update its execution plan after receiving a correction instruction without overcorrecting parts that were already correct. This fine-grained correction capability is especially valuable in long-horizon tasks.

Implications for Agent Development and Selection

The signal-based and causal inference-based evaluation framework offers developers practical guidance at three levels.

Build reliability-centered optimization goals. Traditional evaluation pushes developers to optimize for single-turn answer accuracy. The new framework shifts attention to multi-step task success rates, error recovery capability, and responsiveness to user corrections. When designing prompt templates, tool-calling strategies, and state management mechanisms, developers should prioritize reducing failure rates on critical paths rather than merely improving the textual quality of individual outputs.

Data-driven weakness identification. By analyzing the contextual distribution of complaint signals and correction instructions, developers can precisely locate the task stages where a model is most likely to deviate from expectations, whether that is requirement understanding, code generation, command execution, or result verification. This fine-grained diagnostic capability goes far beyond manual error annotation in traditional evaluation and provides clear direction for subsequent fine-tuning and toolchain optimization.

Establish continuous evaluation mechanisms. Launching an agent does not mean evaluation ends. Development teams are advised to deploy session log collection pipelines in production environments and periodically process new data using a unified signal extraction and causal inference workflow to monitor trends in net improvement rate across model updates. This continuous evaluation mechanism can detect regressions introduced by model iterations and trigger alerts when data distribution shifts occur.

Conclusion: Toward Explainable and Reproducible Agent Evaluation

The evolution of agent evaluation reflects a deeper requirement from AI engineering practices: evaluation must answer not only “who is stronger,” but also “where is it stronger” and “why.” By automatically mining real session signals and applying rigorous causal inference, the new evaluation methodology is moving agent capability assessment from vague qualitative comparison toward transparent quantitative attribution. For developers, understanding the underlying logic of this evaluation paradigm helps not only in rationally selecting agent products for specific business scenarios, but also in establishing a scientific measurement baseline for iterating and optimizing in-house agent systems. Looking ahead, as new paradigms such as multimodal interaction and multi-agent collaboration emerge, evaluation systems will continue to evolve in signal dimension expansion, causal inference algorithm improvement, and real-time capability enhancement.