Agent Evaluation Through Causal Tracking: Mining Real Sessions for Real-Task Leaderboards

    This article explains why traditional agent evaluation is insufficient and introduces a paradigm that mines real user sessions for multidimensional behavioral signals, then applies a causal tracking framework based on randomized inference to estimate each model’s contribution. It also examines how to interpret results and offers practical guidance for teams building internal agent evaluation systems.

    1. Why Agent Evaluation Needs a New Approach

    As large language model agents take on more long-cycle, multi-step tasks in real business settings, the limits of traditional evaluation are becoming increasingly clear. The early mainstream approach was head-to-head voting: two models answer the same question, and the user chooses the better response. This works reasonably well in chat scenarios, but it exposes several problems when applied to agents:

    1. Tasks are not controllable. Voting scenarios are usually short question-and-answer interactions, while agent tasks often span dozens of turns and involve code execution, file operations, environment configuration, and more. A single comparison cannot reflect complete capability.
    2. Feedback signals are coarse. User votes produce only a binary conclusion about “who is better,” losing structured information about what is better and what is worse.
    3. Sample bias. Users willing to participate in voting differ from users in real production environments, making it difficult to extrapolate evaluation results.

    As a result, the industry has begun exploring a new paradigm: instead of actively organizing head-to-head comparisons, mine real user agent sessions, automatically extract evaluation signals, and then use statistical methods to estimate each model’s real contribution. That is the subject this article expands on.

    2. Real Session Signal Mining: From Votes to Multidimensional Signals

    The core idea of the new paradigm is that user behavior while using an agent is itself the most authentic feedback. By parsing session logs, multiple types of key signals can be extracted automatically without requiring users to provide additional labels:

    • Explicit confirmation: When a user gives positive confirmation such as “done” or “meets expectations” at the end of a task, this can serve as direct evidence of task success.
    • Praise and complaints: Natural language feedback in sessions contains attitude information, and positive or negative sentiment can be automatically identified with classification models.
    • Correction instructions: Instructions such as asking the agent to “try another way” or saying “you did that wrong earlier” are high-value signals for measuring correctability.
    • Command execution errors: When an agent executes Shell commands, the error rate, retry rate, and whether the issue is eventually fixed directly reflect its engineering operation reliability.

    Compared with a single vote, these signals form a multidimensional evaluation system. For example, “command execution reliability” measures an agent’s success rate in a real terminal environment, while “direction adjustment and correction ability” measures a model’s ability to accept mid-task user corrections and adjust its behavior. These two capabilities are almost impossible to cover in traditional chat evaluation, yet they are exactly what matters for deploying agents in practice.

    3. Causal Tracking: The Estimation Principle Behind Randomized Inference

    Once signals are available, a statistical question remains: are models randomly assigned in user sessions? If not, different models face different task difficulty and user groups, and directly comparing raw success rates will produce serious bias.

    The causal tracking method borrows the inference framework of randomized trials to strip out these confounding factors:

    1. Construct a reference baseline. Aggregate sessions from all evaluated models and calculate the global average level for each signal metric as the reference for an “average model.”
    2. Estimate the net improvement rate. For a given model, count the relative proportion of positive and negative signals in its sessions and estimate its net improvement rate relative to the average. A positive value indicates better-than-average performance, while a negative value indicates below-average performance.
    3. Control confounding variables. Use stratification or weighting across dimensions such as task type, session length, and user activity to reduce bias caused by differences in task distribution, making the ranking closer to the causal contribution of the model itself.

    The advantage of this method is that it does not require building a manual evaluation set. Evaluation rolls continuously with real traffic and can reflect a model’s dynamic performance on long-cycle tasks. The disadvantage is that it depends heavily on the accuracy of signal recognition, and rankings are sensitive to sample size. Therefore, when interpreting results, attention should be paid to confidence intervals and sample size rather than only to rank.

    4. An Interpretation Example: How to Quantify Generational Improvement

    Consider a public evaluation result for an open-weight model (used here only as a methodological example and not pointing to a specific vendor): the model’s overall net improvement rate was +5.1%, while the previous generation was +0.9%, a generational improvement of 4.2 percentage points. It ranked mid-table among all evaluated models, but ranked first among open-weight models and among open models in the three subscenarios of work, coding, and chat.

    This example illustrates two interpretation points:

    • Read rankings by layer. Overall leaderboard rankings are affected by closed-source flagship models, while rankings within open-weight models are more useful for self-deployment selection.
    • Submetrics are more actionable than total scores. If the business is mainly automated operations, command execution reliability should be the focus. If it involves coding tasks with intensive human-agent collaboration, the weight of correction ability should be increased.

    5. How Developers Can Implement It: Suggestions for Building an Internal Evaluation System

    For teams that want to build internal agent evaluation capabilities, the real-session evaluation paradigm offers the following practices:

    1. Establish structured collection of session logs. Record task descriptions, each turn’s tool calls, command execution results, and original user feedback to provide a data foundation for signal extraction. Logs can be archived through standard logging services or object storage for later batch analysis.
    2. Define a signal set aligned with the business. In addition to general signals, add business-specific signals such as deployment success rate, test pass rate, and ticket resolution time.
    3. Report results with quantiles and confidence intervals. Avoid publishing only a single ranking. Provide interval estimates so decision-makers understand uncertainty.
    4. Run continuous rolling evaluation. Let evaluation run continuously with real traffic to capture changes in model behavior on long-tail tasks rather than relying on one-off benchmark tests.
    5. Pay attention to privacy and compliance. Session logs may contain sensitive data. Desensitization should be completed before collection and modeling, and the principle of data minimization should be followed.

    6. Conclusion

    Agent evaluation is moving from “organizing user votes” toward “automatically extracting causal evidence from real usage.” Multidimensional signal mining based on real sessions, combined with a randomized inference framework, can more objectively measure agent capabilities in key dimensions such as engineering execution and task correction. For developers, understanding this methodology not only helps with correctly interpreting public leaderboards but also provides a reusable framework for building internal evaluation systems: use real tasks as samples, behavioral signals as metrics, and statistical inference as conclusions, so that model selection is based on a reliable data foundation.