Evaluating AI Agents in Real-World Sessions: Signal Extraction and Causal Tracing

As Large Language Models (LLMs) evolve into autonomous agents, traditional static benchmarks based on multiple-choice questions or short Q&As can no longer effectively reflect a model’s execution capability in complex, real-world tasks. Agent evaluation must account for long-horizon sessions, multi-turn tool calls, and complex environmental interactions, making the construction of a scientific and objective real-task leaderboard a core challenge in the industry. This article explores an agent evaluation system based on real user session mining and a “causal tracing” inference method, detailing its technical architecture and signal extraction mechanisms.

I. Limitations of Traditional Evaluation and the Real-World Session Breakthrough

In traditional model arena modes, evaluation typically relies on users subjectively voting between the responses of two models. While this approach offers some distribution advantages in lightweight conversational scenarios, it has significant flaws in agentic scenarios:

1. Long Task Cycles and Delayed Feedback: Agent tasks often involve long-horizon operations such as code writing, automated script execution, and multi-file modifications. After a task is completed, it is difficult for users to objectively compare and score two models from memory.
2. Binary Execution Results: The success of agent tasks (such as executing a Shell command) usually has objective criteria. Subjective voting cannot accurately reflect a model’s ability in direction correction or error recovery.

Therefore, the industry needs an evaluation system that does not rely on direct voting but instead deeply mines real user agent sessions to automatically extract key execution signals. This system focuses on performance in real business scenarios and can objectively reflect a model’s practical value in actual development and work assistance.

II. Core Architecture: Automated Multi-Dimensional Signal Extraction

When executing tasks, agents generate extensive interaction records with the environment. By tapping into real session streams, the evaluation system utilizes automated parsing mechanisms to extract key signals that reflect task quality. The main dimensions include:

1. Explicit Confirmation Signals
   When an agent completes a complex instruction, user feedback such as “correct,” “no problem,” or “that’s it” is automatically captured as a positive signal of task success. This requires the evaluation system to have robust semantic recognition capabilities to pinpoint the user’s confirmation intent accurately within a long context.

2. Praise and Complaint Signals
   Unlike simple satisfaction or dissatisfaction, praise and complaints are often directed at specific model behaviors. For instance, a user might praise “this code is well optimized” or complain “modifying this file caused a compilation failure.” The system needs to bind these sentiment tendencies to specific agent actions causally.

3. Correction Command Signals
   This is a core metric for measuring an agent’s “steerability”—its ability to adjust direction and correct deviations. When a user notices the agent deviating from the goal, they will issue commands like “No, you should use another module” or “Stop the current operation and roll back to the previous step.” The evaluation system quantifies the model’s controllability by tracking the frequency of correction commands and the success rate of task recovery after a correction.

4. Shell Command Execution Error Signals
   In programming and automation tasks, Bash or Shell commands generated by the agent interact directly with the operating system. The evaluation system captures the system output after command execution to identify error messages (such as non-zero Exit Codes or Syntax Errors). The error rate directly reflects the model’s performance in “Shell execution reliability.”

Below is a simplified pseudocode for automated signal extraction, demonstrating how to parse structured features from session logs:

def extract_signals_from_session(session_log):
    signals = {
        "explicit_confirmation": False,
        "praise_count": 0,
        "complaint_count": 0,
        "correction_commands": 0,
        "shell_execution_errors": 0
    }
    for turn in session_log.turns:
        # Extract user feedback based on an intent recognition model
        if turn.role == "user":
            intent = classify_intent(turn.text)
            if intent == "CONFIRM":
                signals["explicit_confirmation"] = True
            elif intent == "PRAISE":
                signals["praise_count"] += 1
            elif intent == "COMPLAINT":
                signals["complaint_count"] += 1
            elif intent == "CORRECTION":
                signals["correction_commands"] += 1
        # Extract execution reliability based on environment feedback
        if turn.role == "tool" and turn.tool_name == "shell":
            if turn.output.exit_code != 0:
                signals["shell_execution_errors"] += 1
    return signals

III. Causal Tracing: Estimating Net Improvement Rate via Randomized Inference

Once multi-dimensional signals are extracted, how do we transform these discrete metrics into relative capability scores between models? Traditional weighted scoring is susceptible to confounding variables such as differences in user demographics and uneven task difficulty distributions. This evaluation system introduces a randomized inference method called “causal tracing.”

The core idea of causal tracing is to estimate the “Net Improvement Rate” of each model relative to the average level. Its technical implementation path is as follows:

1. Randomized Task Allocation: Within real user sessions globally, the system implicitly or explicitly assigns similar task contexts to different models for processing, constructing a controlled environment similar to A/B testing.
2. Confounding Variable Control: Through the randomization mechanism, it severs the spurious correlation between task difficulty, user habits, and model selection, ensuring that the observed performance differences are caused by the models’ own capabilities.
3. Net Improvement Rate Calculation: Statistical inference methods are used to calculate the expected probability of a model achieving a key signal (e.g., completing a task without correction commands), minus the average expected probability of all models on that task, resulting in the Net Improvement Rate.

The advantage of this method is that it not only provides a ranking but also explains the actual gain of a model in specific capability dimensions compared to the industry average baseline. For example, if a model’s comprehensive Net Improvement Rate is +5.1%, it means that under the same real-world task constraints, the model solved 5.1% more problems than the industry average.

IV. Refining Evaluation Dimensions and Long-Horizon Session Challenges

Real-world agent evaluation must cover diverse application scenarios. A comprehensive leaderboard typically categorizes tasks into three main groups: Work, Programming, and Chat.

1. Programming and Automation Scenarios
   This focuses heavily on Shell command execution reliability and code generation accuracy. The system needs to process tens of thousands of real long-horizon task sessions, involving multi-file collaborative modifications, dependency installation, and test case execution. A model’s error correction capability and context retention in these tasks are the key judging criteria.

2. Work and Task Planning Scenarios
   This emphasizes the agent’s task decomposition and tool-calling orchestration. Evaluation signals are heavily centered on the frequency of user correction commands and the completeness of sub-task fulfillment.

3. Chat and Generalized Interaction Scenarios
   Although leaning towards natural language interaction, within the agent framework, it still requires evaluating intent recognition accuracy and logical consistency across multi-turn dialogues.

When processing over 17,500 real long-horizon sessions, the evaluation system faces significant engineering challenges. Contexts often exceed tens of thousands of tokens and involve the concatenation of multi-turn tool return results. The system must rely on efficient logging services and distributed processing frameworks to clean, extract features, and trace states from massive unstructured session streams, ensuring the timeliness and accuracy of evaluation data.

V. Engineering Practices and Outlook for Enterprise-Level Agent Evaluation

For enterprises, building an internal agent evaluation system is critical to ensuring the effectiveness of large model deployment. When constructing such an evaluation architecture, enterprises can refer to the following best practices:

1. Streamline Internal Real Business Flows: Instead of relying on open-source static datasets, extract desensitized real agent session logs from internal R&D pipelines, customer service systems, or office automation workflows.
2. Establish Fine-Grained Signal Collection Standards: From the beginning of agent framework design, incorporate user feedback channels, tool execution return codes, and state machine rollback operations into a structured log collection system. Manage these using standard container platforms and logging services.
3. Introduce Causal Inference Thinking: When comparing models of different parameter scales or fine-tuning strategies, avoid looking solely at success rates. Use randomized traffic routing mechanisms to calculate the Net Improvement Rate to eliminate inherent task difficulty biases.

Currently, the evaluation of large model capabilities is shifting comprehensively from “static response” to “dynamic execution.” An evaluation system based on real sessions and analyzed via causal tracing provides the industry with an objective, quantitative yardstick. Through refined signal extraction (such as execution reliability and steerability), developers can accurately pinpoint model weaknesses, thereby driving the collaborative iteration of underlying models and upper-layer agent frameworks. As automated evaluation technologies mature, future agent leaderboards will align more closely with real business value, providing solid data support for the deployment of enterprise-grade AI applications.