A New Paradigm for AI Agent Evaluation: Measuring Model Capabilities via Real-World Conversations and Causal Inference

I. Limitations of Traditional Evaluation Methods

As large language models (LLMs) and AI agent applications rapidly evolve, accurately assessing model performance in real-world tasks has become a shared challenge for developers and researchers. Historically, the industry has relied on two primary evaluation methods: fixed benchmark datasets based on manual annotation, and arena-style voting where users compare responses from two different models.

The shortcoming of fixed benchmarks lies in their static nature. If a model encounters similar questions during training, its benchmark scores may be artificially inflated, failing to reflect its generalization capabilities in real-world scenarios. While voting arenas introduce real user feedback, they exhibit distinct biases. Users often prefer longer, more enthusiastically styled responses over factually correct ones. Furthermore, the demographic composition of voters, sample sizes, and task distributions directly impact ranking stability. Additionally, single-turn comparisons struggle to capture critical multi-turn interaction capabilities, such as continuous error correction, tool calling, and autonomous recovery from failed command executions.

As AI agents transition from simple “chat models” to “task-executing systems,” the gap between traditional metrics and actual capabilities widens. We need an evaluation system capable of mining objective signals from real-world, long-horizon, multi-turn agent conversations and quantifying the model’s contribution using rigorous statistical methods.

II. Evaluation System Design: Signal Mining from Real Conversations

2.1 Data Sources and Sample Composition

Unlike artificially constructed Q&A pairs, this evaluation system directly collects complete interaction sessions between real users and AI agents. These conversations cover various scenarios, including work assistance, programming and debugging, daily chat, information retrieval, and task planning. Session lengths often span dozens to hundreds of turns. In current public evaluation scales, the data foundation encompasses tens of thousands of real, long-horizon agent task sessions, providing a sufficient sample size for statistical analysis.

The core advantage of using real conversations is that task objectives are spontaneously generated by users, eliminating artificial cues. Successes and failures naturally emerge during the process, requiring no manual annotation of “correct answers.” The contextual dependencies across multi-turn interactions are fully preserved, laying the data foundation for subsequent causal inference.

2.2 Key Signal Definitions

The core of this evaluation system lies in extracting quantifiable behavioral signals from raw conversation logs. Through extensive observation and analysis, the following four types of signals have proven highly discriminative and stable:

Signal 1: Explicit Task Confirmation. When a user explicitly states affirmations like “Okay, let’s do it this way” or “This plan works,” it indicates that the agent’s proposed solution or execution result has been accepted. This signal directly reflects the model’s ability to accurately understand and respond to user intent.

Signal 2: Praise vs. Complaint. Natural emotional expressions from users serve as strong feedback signals. Praise typically occurs after an agent solves a complex problem or provides unexpected value. Complaints usually point to incorrect results, invalid suggestions, or endless loops that fail to converge. Using both signals creates a bidirectional positive and negative constraint on model behavior.

Signal 3: Correction Instruction. When users notice an agent deviating from the expected direction, they often issue corrective commands, such as “That’s not what I meant; you need to read the config file before executing” or “Try a different approach; don’t use recursion.” The frequency of correction instructions and the subsequent execution success rate reflect the model’s ability to accept real-time feedback—specifically, its direction adjustment and error correction capabilities.

Signal 4: Shell Execution Error. In tasks involving code generation or system operations, whether the Shell commands generated by the agent are correct and execute successfully on the first try is a hard metric for tool calling reliability. The ability to autonomously repair after a command execution failure (whether it retries and adjusts the command based on error messages) is equally noteworthy.

2.2 Technical Implementation of Signal Extraction

Extracting these signals from unstructured conversation text requires natural language processing (NLP) technologies. The implementation path includes:

1. Regex and Pattern Matching: Used to extract error codes from Shell commands and their execution results with high precision.
2. Few-Shot Classification with Pre-trained Language Models: Classifies the sentiment and intent of user utterances to determine if they constitute praise, complaints, or correction instructions. By carefully constructing prompt templates with a small number of labeled samples, high accuracy can be achieved.
3. Temporal Annotation: Pairs each user message with the subsequent model response to determine behavioral changes before and after signal triggering, providing temporal dimension information for subsequent causal calculations.

In practical engineering, the signal extraction module must possess real-time processing capabilities, continuously ingesting new conversation data in a pipeline fashion to ensure the timeliness of evaluation results.

III. Core Algorithm: Causal Tracing and Net Improvement Rate

3.1 Why Causal Inference is Necessary

Directly comparing the frequency of signal occurrences across different models on the same task is unreliable due to selection bias. Different user demographics, task difficulties, and conversation turns can introduce systematic variances. For instance, a model excelling in programming might be assigned more complex debugging tasks, naturally resulting in more Shell command errors. This does not imply the model is less capable.

To address this, the evaluation system introduces the concept of “Causal Tracing.” The core idea is: given the conversational context and task type, estimate how the probability of a signal occurring would change if the current model were replaced by an average-level model. This probability increment represents the model’s “Net Improvement Rate” for that specific signal.

3.2 Implementation Logic of Randomized Inference

The specific implementation steps are as follows:

Step 1: Conversation Splitting and Feature Encoding. Each complete conversation is split into segments at decision points. Each segment contains state features (historical dialogue content, current task type, executed action sequence) and outcome signals (whether praise was received, whether a Shell error occurred, etc.).

Step 2: Constructing a Control Distribution. For each segment, “counterfactual outcomes” are sampled from the historical data of all model outputs. In other words, we ask: under the same state features, if an “average-level” model responded, what feedback signal would the user provide?

Step 3: Calculating the Net Improvement Rate. Let P_actual be the actual proportion of a positive signal (e.g., praise) triggered by a target model M across all segments. Let P_expected be the expected proportion based on the counterfactual distribution. The Net Improvement Rate R for model M regarding this signal is defined as:

R = (P_actual - P_expected) / (1 - P_expected)

When R is positive, the model is more capable of stably triggering positive signals compared to the average level. When R is negative, its performance is below average. This metric is normalized to the [-1, 1] interval, facilitating cross-model and cross-signal comparisons.

3.3 Multi-dimensional Signal Weighting

After obtaining the Net Improvement Rate for individual signals, the evaluation system computes a comprehensive ranking through multi-dimensional weighting. The weight design must balance the signal-to-noise ratio and business importance. For example, the weight of Shell execution reliability should be higher in “Coding” tasks than in “Chat” tasks, while direction adjustment and correction capabilities are more critical in “Work Assistance” tasks.

Taking an open-weight model as an example: its comprehensive Net Improvement Rate reached +5.1%, a significant increase from its previous generation (+0.9%). It ranked at the top among open-weight models in sub-metrics like Bash execution reliability and user correction responsiveness. This result reflects the genuine capability growth brought by model iteration and validates the evaluation system’s sensitivity to capability changes.

IV. Multi-dimensional Interpretation of Evaluation Results

4.1 Differentiated Analysis by Task Type

The evaluation system categorizes tasks into three main classes: Work, Coding, and Chat. Different models often show significant ranking variances across these categories. This reminds us that a broad “comprehensive ranking” cannot fully reflect a model’s practical usability. When selecting a model, developers should examine rankings and sub-metrics within specific categories relevant to their business scenarios.

Taking the aforementioned open-weight model as an example, it achieved first place among open-weight models in all three task categories, indicating a strong balance between execution tasks (Work, Coding) and interactive tasks (Chat). This level of balance is uncommon in model architecture design, as models typically show a tendency toward a specific type of task.

4.2 Applied Value of Sub-metrics

Beyond comprehensive rankings, the engineering value of sub-metrics is more direct. For example:

• Bash Reliability: Reflects the model’s ability to understand and generate system commands, which is crucial for automated operations and data pipeline construction.
• Direction Adjustment and Correction Capability: Demonstrates the model’s context-tracking ability in long conversations and its flexibility in rapidly adjusting generation strategies when user requirements change.
• Complaint Signal Frequency: Can be used for online monitoring. If the Net Improvement Rate for complaint signals suddenly drops for a specific model version, it typically indicates deteriorating user satisfaction and should trigger alerts and rollback checks.

V. Practical Paths and Optimization Directions

5.1 Application in Model Iteration

Integrating this evaluation system into the development pipeline is key to closed-loop iteration. Once a new model version completes training, a small fraction of real user requests can be routed to it internally. By quickly calculating the Net Improvement Rate through signal extraction and causal tracing, it serves as a decision basis for a gradual rollout or full release. Compared to traditional offline evaluation datasets, this method captures performance degradation caused by data distribution shifts much faster.

5.2 Current Limitations and Evolution Directions

No evaluation system can completely eliminate bias. The current causal tracing method still faces the following limitations:

1. Accuracy of Counterfactual Estimation: The definition of an “average-level model” relies on historical data distributions. If the user demographic undergoes structural changes (e.g., shifting from C-end consumers to B-end enterprise users), the control distribution may require recalibration.
2. Limited Signal Coverage: Currently extracted signals are primarily explicit user feedback. Reliable signal models for “implicit failures” (e.g., a user abandoning a session out of dissatisfaction) have not yet been established.
3. Multi-model Confounding: When multiple models are invoked sequentially in a session (e.g., routing to different specialized models), causal tracing struggles to accurately attribute independent contributions to each model.

Future exploration directions include utilizing finer-grained conversation state vectors to replace manual feature extraction, optimizing signal weights through reinforcement learning, and building a unified cross-model evaluation benchmark to foster community standardization.

VI. Conclusion

Evaluation methods for AI agents based on real-world conversations and causal inference are becoming an essential complement to traditional benchmarks and manual voting. By mining behavioral signals from massive natural interactions and quantifying the model’s true contribution to user outcomes with statistical rigor, it provides a more compelling decision-making basis for model developers, platform operators, and enterprise adopters. As the application scenarios for AI agents expand in depth, this “from the real world, to the real world” evaluation philosophy will increasingly demonstrate its methodological value.