A New Paradigm for AI Agent Evaluation: From Real-World Task Signals to Quantitative Analysis

Introduction: Challenges and Paradigm Shifts in Agent Evaluation

As Large Language Models (LLMs) evolve from simple text generation tools into autonomous Agents capable of executing complex tasks, traditional static evaluation systems face unprecedented challenges. Previously, the industry commonly relied on “blind voting on dual answers” or “standardized test bank scoring” to assess model capabilities. While acceptable in conversational scenarios, these methods fail to accurately reflect an agent’s comprehensive performance in dynamic tasks involving multi-turn interactions, tool invocation, and environmental feedback processing. The core value of an agent lies in its “actionability”—the ability to understand user intent, plan execution steps, invoke external tools, and dynamically adjust strategies based on results. This capability is almost impossible to capture effectively through static Q&A evaluations.

Consequently, the field of agent evaluation urgently needs an assessment method that closely aligns with real-world application scenarios and objectively quantifies model improvements. Starting from the foundational logic of the evaluation system, this article systematically explores a new evaluation paradigm based on extracting real user session signals and applying causal inference. It provides an in-depth analysis of result interpretation methods, offering a practical reference framework for technical decision-makers.

1. Core Logic of Evaluation Design: Real-World Task-Driven Signal Mining

1.1 From Manual Voting to Real-World Session Analysis

Traditional model comparison evaluations often rely on human preference rankings of model responses. This approach has several inherent flaws: First, manual voting is costly and difficult to scale continuously. Second, evaluators tend to prefer answers that “look more fluent,” ignoring the actual success rate of task completion. Furthermore, static Q&A cannot cover complex agent scenarios such as “multi-turn error correction,” “tool invocation exception handling,” or “mid-task user requirement changes.”

The core shift in the new evaluation paradigm is to avoid deliberately constructing test tasks and instead deeply mine everyday session data between real users and agents. The advantage of this design lies in the natural authenticity and diversity of the data—user needs generated in actual usage reflect the model’s real-world application performance far better than any lab-designed test set. Specifically, the system analyzes massive amounts of real session logs to identify interaction sequences with clear task objectives and extracts key behavioral signals that reflect task completion quality.

1.2 Constructing the Key Signal System

In real sessions, user behavior exhibits observable differences depending on the smoothness of the task. The evaluation system needs to abstract quantifiable signal dimensions from these behavioral differences, primarily including the following categories:

• Success Confirmation Signals: Users explicitly express approval of the task result in the session, such as “Great,” “This is exactly what I need,” or “Done.” These signals directly reflect task delivery quality and serve as the strongest evidence for evaluating whether a model “can do things right.”
• Frustration and Complaint Signals: When a model fails to meet user expectations, users may exhibit complaints, disappointment, or directly point out errors, such as “This is not what I wanted,” “You misunderstood,” or “How is this wrong again?” Such signals accurately identify a model’s weak points in specific task types.
• Correction Command Signals: Users modify or request adjustments to the model’s output, such as “No, try another way” or “Please redo this following XX.” The frequency and timing of these signals reflect the model’s first-time execution accuracy and its correction efficiency after receiving feedback.
• Execution Exception Signals: In tasks involving command-line operations, code execution, or multi-step operations, objective execution exceptions like Shell command errors or program crashes provide critical data for assessing a model’s reliability in real operating environments.

By fusing multi-dimensional signals, the evaluation system avoids the bias that a single metric might introduce. For example, a model might frequently receive positive user feedback while simultaneously maintaining a high error rate when executing complex commands. This will be comprehensively reflected in its final score.

2. Causal Tracing: A Quantitative Method of Randomized Inference

2.1 What is the “Net Improvement Rate”

Once various signals are extracted from real sessions, the next challenge is converting them into comparable quantitative metrics. Given the natural differences in user demographics and task difficulty distributions across session samples for different models, a simple comparison of signal frequencies is unscientific. For instance, a model used in professional programming scenarios may face far greater task complexity than one used for daily consultations; directly comparing their “user complaint rates” would be inherently unfair.

To address this, the evaluation system introduces “Causal Tracing,” a randomized inference method. Its core idea is to estimate each model’s “Net Improvement Rate” relative to the average level of all participating models. Specifically, this method performs counterfactual inference on historical session data—estimating how the task success rate would change if the same user used a different model in the identical task scenario. Through large-sample randomized controlled analysis, the net increase or decrease in each model’s task-handling capability is determined. The unique value of this method lies in eliminating confounding factors such as scenario differences and user behavioral habits, establishing a unified baseline for cross-model capability comparison.

2.2 Calculation Logic Illustration

For easier understanding, consider this simplified logical illustration: Assume the average success rate of all participating models for a specific task type is 60%. Through causal inference analysis, Model A’s success rate for tasks of equivalent difficulty is found to be 5 percentage points higher than the average, giving it a Net Improvement Rate of +5%. Model B is 3 percentage points lower, resulting in a Net Improvement Rate of -3%. This quantification of relative performance not only intuitively reflects the capability gap between models but also precisely characterizes the iterative improvement effects of different versions of the same model.

During the calculation process, the evaluation system also applies weighting to the signals. Different signal categories have varying confidence levels for predicting task success. For example, an explicit task completion confirmation is typically more informative than a session lacking subsequent interaction. Through a scientific weighting strategy, the evaluation results more closely align with actual user experience perceptions.

3. Interpreting Evaluation Results: From Comprehensive Scores to Granular Capability Dimensions

Beyond producing a single comprehensive ranking, the greater value of agent evaluation lies in providing granular metrics that guide optimization directions. Taking a recent open-weight model’s evaluation performance as an example: it ranked first in its open-weight category and thirteenth among all participating models. This ranking was derived from the analysis of over 17,500 real, long-cycle agent task sessions, covering three core scenarios: work assistance, software development, and daily chat. The model achieved a comprehensive Net Improvement Rate of +5.1%, a significant 4.2 percentage point increase from its previous version’s +0.9%, fully demonstrating the substantial capability growth brought by model iteration.

A deeper analysis of the granular metrics reveals that this model excels in Bash command execution reliability. This indicates a low command error rate in real programming and system operation scenarios; even if the user provides incomplete or slightly ambiguous instructions, the model can accurately map them to the correct Shell operations. Another prominent advantage is its direction adjustment and error correction capability—when users propose modifications to the model’s initial feedback, this model demonstrates higher accuracy in guessing the user’s new intent in subsequent turns. It can quickly adapt to changing user requirements, significantly shortening the convergence path of multi-turn interactions.

This granular capability assessment provides developers with actionable guidance. If a model scores low in dimensions related to “complaint signals,” it indicates a tendency to dissatisfy users in those specific scenarios. The R&D team can then conduct targeted data collection and fine-tuning optimizations for those scenarios, turning the evaluation system into a critical feedback loop that drives model iteration.

4. Future Evolution and Application Value of the Evaluation System

4.1 From Periodic Evaluation to Continuous Monitoring

As the penetration rate of agents increases across various business scenarios, the traditional “periodic evaluation report” model may no longer meet the demands of rapid iteration. One future direction for this evaluation system is to automate the signal mining and causal inference processes, building a continuously running model quality monitoring platform. Under this system, as soon as a model is updated and deployed, the system can capture real-time performance fluctuations under live traffic. If a significant drop in the Net Improvement Rate is detected, or a spike in specific exception signals occurs, an alert is triggered immediately, helping technical teams locate and roll back issues before they spread.

4.2 Decision Support for Model Selection and Procurement

For enterprise users, this type of evaluation system provides an objective reference dimension beyond conventional benchmarks. Typically, when enterprise technical teams select a foundational model, they comprehensively consider the model’s scores on public datasets, inference costs, and compliance requirements. However, high scores on public datasets do not always translate to real-world effectiveness in specific vertical business applications. An evaluation system based on real user sessions bridges the gap between traditional benchmarks and actual performance. By leveraging real-world scenario rankings provided by third parties alongside their private data evaluations, enterprises can more comprehensively assess different models’ adaptability to their business scenarios, thereby reducing trial-and-error costs in model selection.

Conclusion

Agent evaluation is transitioning from “understanding text” to “evaluating action.” By deeply mining success and frustration signals within real user sessions and applying rigorous causal inference quantification methods to derive the Net Improvement Rate, a measurement standard much closer to practical application value has been established for agent capability assessment. A prominent advantage of this methodology is its ability to continuously self-evolve as real usage data accumulates, freeing model evaluation from its reliance on manual scoring and moving towards a more dynamic, objective, and sustainable path. For developers and technical managers, understanding the structural logic of this evaluation system is crucial for scientifically driving the evaluation, selection, and continuous optimization of agent products.