A New Paradigm in AI Agent Evaluation: Measuring Net Improvement Rate via Causal Inference

As Large Language Model (LLM) technology evolves from conversational assistants to autonomous agents, traditional static evaluation methods—such as pairwise human preference voting on isolated responses—are no longer sufficient to assess agent performance in complex, real-world tasks. Real-world agents must execute multi-step planning, invoke external tools, and continuously correct errors. To address this, the industry is exploring a new evaluation paradigm: an agent leaderboard based on real-world session mining and “causal tracing.” This article delves into the technical core of this evaluation system, exploring how it isolates environmental noise through statistical methods to precisely measure model performance in real tasks.

1. Limitations of Traditional Evaluation and the Rise of Real-Task Assessment

In traditional LLM evaluation, a common practice is to feed preset prompts to the model, collect responses, and have human annotators or stronger models perform pairwise voting. While effective for evaluating text generation quality, this approach has significant flaws in agentic scenarios:

1. Delayed Feedback and Multi-step Interaction: Agent tasks typically involve multiple tool calls and state judgments. The quality of a single response cannot reflect the success rate of the entire task chain.
2. High Environmental Dependency: Agent performance heavily relies on the execution environment (e.g., sandbox environments, API availability). Simple voting cannot isolate failures caused by the environment itself.
3. Lack of Process Correction Evaluation: When using agents in real scenarios, users frequently issue correction commands or adjust directions. Traditional evaluation ignores this dynamic interaction process.

To address this, next-generation agent evaluation platforms choose to directly mine long-term agent sessions from real users. They use automated methods to extract key behavioral signals, using “task chain completeness” and “user intervention frequency” as evaluation baselines.

2. Automated Extraction of Multi-dimensional Key Signals

In real session data, the evaluation system no longer relies on subjective user ratings. Instead, it automatically extracts objective signals through conversational behavioral analysis. These signals cover the entire lifecycle of task execution:

1. Task Terminal State Confirmation Signals

The system uses semantic recognition to extract explicit user statements at the end of a session. This includes:

• Praise and Confirmation: Phrases like “That’s right” or “Perfectly solved,” which are mapped as positive signals for task success.
• Complaints and Negation: Phrases like “Still not working” or “Completely failed to solve my problem,” which serve as indicators of task failure.

2. Process Intervention and Correction Signals

When an agent deviates from the goal during execution, user intervention is a key metric for measuring the model’s “controllability.” The system focuses on extracting:

• Correction Commands: The user interrupts the agent’s current execution flow, e.g., “Don’t use this method, try another approach.”
• Direction Adjustment (Steerability): The user forcefully resets the context or modifies intermediate parameters. A lower frequency of correction commands indicates stronger planning and comprehension capabilities of the agent.

3. Underlying Execution Reliability Signals

For agents capable of code execution or system operations, the system directly parses the logs of the execution environment to capture:

• Shell Command Execution Errors: Tracking the proportion of exceptions such as syntax errors, permission denied, or non-existent paths.
• Tool Invocation Failure Rate: Calling non-existent APIs or using incorrect parameter formats.
  These hard metrics directly reflect the agent’s reliability in generating executable code.

3. Core Methodology: “Causal Tracing” and Net Improvement Rate

After extracting the above signals, how do we transform these multi-dimensional signals into a comprehensive capability score for the model? This evaluation system introduces a randomized inference method called “causal tracing” to calculate each model’s “Net Improvement Rate” relative to the average level.

1. Why Causal Inference?

In real traffic, the complexity of user groups and the distribution of task difficulty faced by different models are uneven. If we simply calculate the success rate, biases may arise because a particular model was assigned more simple tasks. The goal of causal tracing is to strip away confounding factors like task difficulty and user habits, isolating the pure causal contribution of “model capability itself” to “task success.”

2. Implementation Logic of Randomized Inference

This method draws on the idea of Randomized Controlled Trials (RCTs) in A/B testing but performs retrospective inference on observational data. Its engineering implementation steps are as follows:

# Pseudocode: Causal tracing net improvement rate calculation logic
def calculate_net_improvement(sessions, model_name):
    # 1. Feature alignment: Extract task complexity features (lines of code, number of tools, turns, etc.)
    aligned_groups = align_sessions_by_complexity(sessions)
    # 2. Randomization simulation: Simulate random model assignment within sessions of similar complexity
    for group in aligned_groups:
        # Calculate the current model's performance on similar tasks
        actual_score = extract_signals(group[model_name])
        # Calculate the average performance of other models in the same group
        baseline_score = extract_signals(group['other_models'])
        # 3. Calculate causal effect (Individual Treatment Effect - ITE)
        ite = actual_score - baseline_score
    # 4. Aggregate to Average Treatment Effect (ATE), i.e., Net Improvement Rate
    net_improvement_rate = average(ites) * 100
    return net_improvement_rate

3. Business Implications of the Net Improvement Rate

The Net Improvement Rate is a relative value representing the probability proportion by which a model “netly outperforms” the industry average after excluding environmental and task difficulty interferences. For example, if an open-weight LLM improves its comprehensive net improvement rate from +0.9% in the previous generation to +5.1% after iteration, it means that in real tasks of the same difficulty, its success probability increased by 4.2 percentage points compared to the average baseline. This metric provides harder data support for enterprises during model selection.

4. Subdivision of Evaluation Dimensions and Application Practices

While the comprehensive Net Improvement Rate indicates overall strength, enterprises are often more concerned with vertical capabilities in specific scenarios during actual deployment. This evaluation system subdivides capabilities into the following three major categories, providing independent rankings for open-weight models:

1. Workspace Scenarios

Evaluates daily office automation tasks, such as document processing, information retrieval, and summarization. In this dimension, the agent’s ability to comprehend long texts and the efficiency of information flow across multiple documents are the core assessment points.

2. Programming Scenarios

Focuses on Bash Reliability (Shell command execution reliability) and code error correction capabilities. The agent must not only generate logically correct code but also possess the closed-loop ability to autonomously capture runtime errors, analyze tracebacks, and correct the code. Reducing human intervention is key in long-term programming sessions.

3. Chat and Interaction Scenarios

Evaluates foundational multi-turn dialogue coherence and instruction-following capabilities. Although foundational, preventing context forgetting and maintaining persona consistency in long-term sessions remain technical challenges.

5. Implications for Building Enterprise Agent Evaluation Systems

For development teams building enterprise-grade AI applications, adopting this real-traffic and causal-inference-based evaluation system helps establish a more scientific model iteration loop:

1. Establish Fine-grained Logging Services: Do not merely record user inputs and model outputs. Integrate tool execution stdout/stderr, user interruptions, and correction commands into a unified logging service and monitoring alerting platform to form a traceable session evidence chain.
2. Introduce Causal Attribution Models: During internal evaluations, avoid simply comparing the conversion or success rates of new and old versions. Introduce feature alignment algorithms to ensure that both versions face the same traffic distribution, thereby deriving true iteration benefits.
3. Scenario-specific Benchmark Testing: Referencing the subdivided dimensions of industry leaderboards, enterprises should set exclusive reliability metrics for their own business (e.g., ticket processing, code generation) and conduct high-frequency stress testing in real container environments.

In summary, agent evaluation is evolving from “subjective perception testing” to “objective data tracking and causal computation.” By mining real long-term session signals and calculating the Net Improvement Rate, the industry can not only more accurately locate the true level of models but also point the way forward for the next generation of agents in terms of reliability, controllability, and complex tool invocation capabilities.