A New Paradigm in AI Agent Evaluation: Real-World Signal Mining and Causal Tracking

As Large Language Models (LLMs) transition into autonomous agents executing complex tasks, traditional static benchmarks are no longer sufficient to reflect actual performance in real-world business scenarios. Particularly in long-horizon tasks requiring multi-turn interactions and tool usage—such as coding and automated IT operations—an agent’s capability depends less on single-shot generation quality and more on context maintenance, error correction, and instruction following. Consequently, building a dynamic evaluation system based on real user sessions and causal inference methods is becoming the critical path to measuring agent engineering reliability.

1. Limitations of Traditional Evaluation and Real-World Signal Mining

Traditional model evaluations typically rely on predefined benchmark datasets or blind A/B testing based on user preferences. The former struggles to cover the long-tail, highly variable demands of real-world environments, while the latter is easily skewed by subjective user emotions and inconsistent task difficulties. To accurately assess an agent’s real-world performance, next-generation evaluation systems directly mine real user-agent session logs to automatically extract key signals that indicate task success or failure.

During the signal extraction process, the evaluation system focuses on several core features:

1. Explicit User Confirmation: When an agent completes a task, affirmative feedback from the user (e.g., “Perfect,” “Exactly what I needed”) typically marks a successful task completion.
2. Corrective Instructions and Direction Adjustments: If a user responds with “No, I meant…” or provides new constraints after an agent’s output, it indicates the agent deviated from the intended direction and requires realignment.
3. Complaints and Negative Sentiment: Direct expressions of dissatisfaction or error reports from users serve as direct indicators of an agent’s failure rate.
4. Shell Command Execution Errors: In programming and automation tasks, agents frequently generate and execute shell commands. Terminal error messages such as syntax errors, permission denied, or file not found objectively reflect the agent’s deficiencies in code generation and environmental comprehension.

By automatically extracting these multi-dimensional signals, the evaluation system transforms subjective “good or bad” judgments into objective, quantifiable feature vectors, laying the data foundation for subsequent causal inference.

2. Causal Tracking and Net Improvement Rate Estimation

After collecting massive amounts of real session signals, the core challenge is eliminating confounding variables introduced by varying task difficulties and diverse user phrasing. To address this, the system introduces a randomized inference method called “Causal Tracking” to estimate each model’s Net Improvement Rate relative to the industry average.

The core logic of causal tracking is to strip away correlation to find causality. In practice, the system constructs a counterfactual inference framework:

1. Baseline Construction: It calculates the average performance of all evaluated models across specific signal dimensions to serve as the evaluation baseline.
2. Randomized Control Simulation: For successful or failed sessions of a specific model, the system uses randomized algorithms to simulate the potential outcome distributions of other models under identical contextual constraints.
3. Net Effect Calculation: By comparing the model’s actual signal feedback with the expected value of the simulated distribution, the system calculates the model’s net causal effect on the task success rate, effectively controlling for external variables like task difficulty.

Ultimately, these causal effects are normalized into a “Comprehensive Net Improvement Rate.” For instance, a model with a +5.1% net improvement rate indicates that, given the same task difficulty, choosing this model improves task success rates by 5.1% compared to the industry average. Compared to traditional win-rate calculations, this metric more scientifically reflects the intrinsic capability growth of the model itself.

3. Core Evaluation Dimensions: Reliability, Controllability, and Interaction

Based on the signal extraction and causal inference framework, agent capability assessment is broken down into multiple dimensions with practical engineering significance. Two specific metrics are particularly critical:

1. Bash Reliability

This metric specifically evaluates the accuracy and robustness of an agent when generating and executing system-level commands. In real-world development scenarios, agents must not only write logically correct code but also correctly configure environments, install dependencies, and execute compilation and deployment. Bash Reliability examines the model’s understanding of underlying OS mechanisms, file path rules, and environment variables. A highly reliable model can generate syntax-error-free scripts that align with the execution context in a single pass, significantly reducing the cost of manual debugging.

2. Steerability

Long-horizon tasks are often accompanied by changing requirements or early misunderstandings. Steerability measures how quickly an agent can understand new constraints after receiving a corrective instruction, abandon its erroneous path, and make corrections based on the existing context. This requires not only strong context window management but also high sensitivity to instruction following. In engineering practice, a model with high Steerability can significantly reduce context forgetting and logical deadlocks during multi-turn interactions.

4. Engineering Insights and Evolution from Real-World Long-Horizon Tasks

The validity of evaluation results relies heavily on the diversity and complexity of test samples. Take a representative industry evaluation as an example: its rankings are based on over 17,500 real long-horizon agent task sessions from global users. These tasks span three major categories—work assistance, coding, and general chat—comprehensively covering scenarios from simple information retrieval to complex multi-file collaborative refactoring.

From a technological evolution perspective, large-parameter open-weight models are experiencing a significant leap in real-world scenarios. Taking an open-source model system with a 2.8 trillion parameter scale as an example, through architectural optimization and alignment strategy upgrades, its comprehensive net improvement rate in agent evaluations increased from +0.9% in the previous generation to +5.1%, achieving a 4.2 percentage point leap. It also ranks at the top among open-weight models in specific metrics like Bash Reliability and Steerability. This data demonstrates that a large parameter count not only broadens the model’s knowledge base but also brings qualitative changes to complex instruction parsing and tool invocation stability.

For enterprise application deployment, these real-world evaluation results provide crucial references for model selection. When building enterprise-grade agents, development teams should focus on the model’s corrective capabilities and engineering reliability in long-horizon tasks, rather than blindly trusting benchmark scores. Furthermore, by integrating cloud-native observability tools (such as logging services and monitoring platforms), development teams can perform closed-loop analysis on error logs and user feedback during actual runtime. This enables continuous optimization of the agent’s system prompts and tool invocation chains, ultimately maximizing efficiency in real-world business operations.