A New Paradigm in AI Agent Evaluation: Mining Real User Sessions for Model Capabilities

From Preference Voting to Real-Task Signals: A Paradigm Shift in Agent Evaluation

As Large Language Model (LLM) applications evolve, Agent tasks—scenarios where models must invoke tools, execute commands, engage in multi-turn interactions, and ultimately fulfill complex user objectives—have become a critical benchmark for model utility. However, traditional evaluation methods, such as human preference voting in arena-style setups, increasingly reveal limitations in agent tasks. Agent workflows are typically long-running, multi-step, and feature complex intermediate states; user satisfaction with the final result cannot be simply attributed to the quality of a single conversational turn. Furthermore, manual voting is costly and struggles to cover long-tail real-world use cases.

An emerging evaluation paradigm is gaining industry adoption: directly mining signals from real user agent sessions to assess model performance automatically. The core advantage of this approach is its ecological validity. Instead of relying on artificially constructed test questions, it evaluates tasks generated by users in real production environments, characterized by clear intents and complex constraints. This type of evaluation system is typically maintained by a continuously operating platform that does not require users to provide explicit scores. Instead, it analyzes behavioral traces during the session to infer the model’s capability level.

Signal Extraction: Reading User Satisfaction from Sessions

The most critical technical step in achieving automated evaluation is defining and extracting key signals that reflect model performance. In real agent sessions, user behavior serves as the most direct feedback. Common signal dimensions include:

1. Task Acknowledgment: When a user explicitly states “Okay, please execute” or “Go ahead and do it this way,” it generally indicates that the model has understood the user’s intent and the user approves of the current direction. This signal provides high confidence in assessing the model’s intent comprehension.
2. Praise & Complaint: Emotional expression in natural language acts as a strong feedback signal. For instance, “Great, exactly right” is identified as a positive signal, while “This isn’t what I wanted” or “You made this mistake again” is directly flagged as negative. Since agent tasks frequently involve high-risk actions like code execution and file operations, negative emotional signals warrant particular attention.
3. Correction Instructions: When users issue corrective commands such as “No, try another way” or “Start over, don’t use this parameter,” it directly indicates a deviation in the model’s preceding steps. The frequency and density of correction instructions can reflect the model’s error-correction capability and directional flexibility in dynamic environments.
4. Execution Error Signals: Agent tasks frequently involve Shell commands or other tool invocations. When command execution generates error codes, timeouts, or abnormal outputs, the system can automatically capture these structural errors. These signals are objective, difficult to fake, and highly effective for evaluating a model’s tool-calling reliability in real environments.

These signals are typically refined into computable metrics. For example, a “Bash Execution Reliability” metric measures the success rate of Shell commands generated by the model on their first execution. Meanwhile, a “Steerability” metric focuses on whether the model can quickly and accurately adjust in subsequent turns after a user issues a correction, rather than repeating the same mistake or completely deviating from the goal.

Causal Tracing: Estimating Net Improvement Rate Beyond Correlation

Signal extraction is only the first step. A more complex challenge lies in converting user signals into comparable model rankings. Because task difficulty varies among users, simply averaging the frequency of signals introduces severe bias—a model assigned more high-difficulty tasks might appear to perform worse.

To address this, the evaluation system introduces a randomized inference method called “Causal Tracing.” Its core logic resembles a Randomized Controlled Trial (RCT): during analysis, statistical techniques balance confounding variables such as task difficulty and user characteristics to estimate each model’s “Net Improvement Rate” relative to the average level of all evaluated models. This Net Improvement Rate is not a simple win rate; rather, it quantifies the marginal effect of “how many negative signals a model can reduce and how many positive signals it can increase compared to the average level, if tasks were assigned completely randomly.” This method effectively strips away the interference caused by task difficulty discrepancies, making cross-model rankings horizontally comparable.

Long-Period Sessions and Granular Dimensions: Expanding Evaluation Depth

Another major challenge in agent evaluation is handling long-period tasks. Real agent tasks often require dozens or even hundreds of conversational turns to complete, involving multi-step planning, mid-course corrections, and state memory. Early evaluations only captured single turns or short sessions, easily losing visibility into a model’s long-term planning capabilities. Current leading evaluation systems support rankings based on tens of thousands of real, long-period agent task sessions globally. This ensures evaluation results reflect a model’s ability to maintain context consistency over extended timeframes, remember key constraints, and handle unexpected errors.

Regarding dimension segmentation, evaluations are typically categorized into Work, Coding, and Chat. This breakdown is necessary because the weight of required model capabilities varies drastically across scenarios. In Coding tasks, Shell command execution reliability and code syntax correctness carry higher weight. In Work tasks, understanding ambiguous instructions and maintaining interaction stability with external applications become more critical.

Observing Trends: Open-Weight Models and Capability Leaps

Based on the evaluation framework above, a notable recent trend is the significant improvement of open-weight models in agent capabilities. A cloud provider recently opened the weights of a powerful multimodal model containing 2.8 trillion parameters. This model secured the first place among open-weight models in this evaluation system and ranked prominently among all participating models. Notably, it ranked first among open models across all three segmented scenarios: Work, Coding, and Chat. Its comprehensive Net Improvement Rate experienced a significant leap compared to previous versions—increasing by several percentage points—and it led open-weight models in core granular metrics such as Shell command execution reliability and steerability.

This phenomenon offers multiple insights for the industry. First, it demonstrates that by optimizing training strategies and data quality, open-source models can catch up to or even rival closed-source models in advanced capability dimensions like agentic tasks. Second, high parameter counts remain the cornerstone of agent capabilities; a scale of nearly 3 trillion parameters provides ample capacity for complex reasoning and long-term task tracking. Finally, from a developer’s perspective, this evaluation data provides highly valuable reference for model selection. When developers need to deploy an agent handling complex workflows, the Net Improvement Rate metric is more instructive than simple text generation quality scores.

Building Reliable Agent Evaluation: Practical Recommendations

For developers looking to build or utilize an agent evaluation system, the following practical experiences are worth considering:

• Prioritize negative signals over positive ones: User complaints and correction instructions are often sparser than praise but carry more informational value. The evaluation system should prioritize the capture rate and labeling accuracy of negative signals.
• Introduce time decay factors: In long sessions, a user’s early positive confirmation cannot offset a major failure at the end. The weight of a signal should account for its position within the conversational sequence.
• Combine constructive tasks to cover blind spots: While real sessions are core, real user habits may concentrate on a few task templates. Appropriately introducing constructive adversarial tasks—such as intentionally confusing instructions or tasks with hidden constraints—can help assess a model’s boundary capabilities.
• Focus on Steerability metrics: A model’s response quality to corrections is critical for an agent to land in a production environment. A model that can quickly acknowledge errors and adjust strategies offers far greater practical utility than one that is stubborn, even if its single-turn answers are occasionally correct.

In conclusion, real-session-based agent evaluation systems are becoming the next-generation standard assessment method, succeeding manual preference voting. Through granular signal extraction and rigorous causal inference, it provides developers with a more compelling capability map of models. For teams dedicated to building practical agent applications, understanding and leveraging this evaluation data will directly impact technical selection and product iteration directions.