Agent Evaluation Beyond User Votes: Session Signals and Causal Inference for Net Improvement Rate

The Agent Evaluation Problem: Why Traditional Methods Are Falling Short

As large language model capabilities iterate rapidly, agents have evolved from simple conversational question answering into autonomous systems capable of executing multi-step, complex tasks. They need to call tools, write code, operate command-line environments, and maintain goal consistency across long-horizon interactions. Faced with this level of application complexity, traditional model evaluation systems have exposed significant shortcomings.

For a long time, the industry has widely adopted arena-style evaluation based on user votes as a benchmark. This approach has two models answer the same question, and then users or expert raters decide which is better. However, this scheme has several structural flaws when evaluating agents:

1. Task granularity mismatch: Agent tasks are often long-horizon sessions involving multiple rounds of interaction, tool calls, and state changes, rather than single-turn Q&A. It is difficult for users to make a fair judgment of two models’ overall collaborative capability based only on a final answer.
2. Subjective preference interference: User votes are easily influenced by answer style, wording, and even how talkative a model is, rather than the actual efficiency and correctness of task completion.
3. High annotation cost: Manually annotating complex long-horizon tasks requires annotators with strong technical backgrounds. This is not only expensive but also difficult to scale.

The industry therefore needs a new evaluation paradigm that can automatically extract feedback from real application scenarios and supports causal inference.

The Core of the New Paradigm: Mining Key Signals from Real Sessions

To address these problems, an evaluation mechanism called the “agent arena” has emerged. Its core idea is: do not rely on artificially designed test questions; instead, directly use the large volume of session data generated by real users and agents in daily use as the evaluation sample pool.

The first challenge this evaluation approach must solve is: how can we automatically and accurately judge whether an agent performed well or poorly from unstructured, multi-turn user sessions? The answer lies in extracting “key behavioral signals.” These signals are not users’ general praise or criticism, but behavioral manifestations that users unconsciously reveal during interaction and that carry clear directional meaning.

From a technical implementation perspective, this evaluation system mainly focuses on the following categories of signals:

• Explicit confirmation of task completion: Users explicitly stating during use that “the problem is solved,” “this is exactly what I wanted,” or “task complete,” or other conclusive affirmative statements. Such signals usually appear at the end of a complete task chain and directly reflect the success rate of task delivery.
• Intent recognition of praise and complaints: Unlike simple keyword matching, the evaluation system needs to use sentiment analysis models to accurately identify praise (such as “great” or “so efficient”) or dissatisfaction (such as “this is not what I wanted”) expressed by users in a session. Such signals need to be judged comprehensively in context.
• Frequency and density of correction instructions: During an agent’s task execution, if the user frequently issues corrective instructions (such as “no, you should check this first” or “stop, try another way”), it indicates that the agent’s understanding or planning capability has deviated. The density of correction instructions is an important negative indicator of model controllability.
• Command-line execution error rate: For agent tasks involving code generation or system operations, error logs generated when executing Shell commands or scripts are the most objective “hard metric,” free from emotional interference. Frequent errors mean the model has obvious shortcomings in code accuracy and environment adaptability.

Causal Tracing: Inferring Net Improvement Rate Beyond Correlation

After extracting these signals, how can they be converted into comparable scores? Simply summing and averaging the signals cannot eliminate bias caused by different user habits and differences in task difficulty. For example, one user may habitually say thanks after every interaction, while another is silent. Directly comparing the feedback signals of the two user groups is clearly distorted.

To solve this problem, the evaluation system introduces a randomized inference method called “causal tracing.” The essence of this method is to try to answer a counterfactual question: “If a certain model had not been used, how would the positive signals in that session have changed?”

Its technical logic is as follows:

1. Build a randomized control baseline: In user session data, not all responses come from the model being evaluated. The evaluation system anonymizes all historical data of users using models and builds a random Markov chain model as the “average baseline model.”
2. Estimate the average treatment effect: Through complex probabilistic models, calculate the probability increment of specific key signals (such as task confirmation signals) after the evaluated model intervenes in the conversation, relative to the “average baseline model.” This increment is not a simple difference but a net effect after multivariate control.
3. Output the Net Improvement Rate (NIR): Finally, the evaluation system outputs a standardized value, the “Net Improvement Rate (NIR).” This value represents the percentage by which the model, in real task environments, can increase the probability of user task success relative to the average level of all models on the market.

This calculation process strips away confounding factors such as user preferences and task difficulty. The resulting score is more rigorous in a statistical sense and can help developers more realistically perceive the actual gains brought by model iteration.

A Deeper Breakdown of Evaluation Dimensions: More Than an Overall Ranking

Based on the methodology above, the evaluation system not only outputs an overall leaderboard but also provides multi-dimensional, fine-grained capability radar charts. This is crucial for developers seeking to optimize models in a targeted way.

• Work scenarios: Focus on simulating long-chain tasks such as office automation, information retrieval, and document processing. This dimension emphasizes the model’s ability to decompose complex instructions and the stability of calling external tools (such as calendars and email APIs).
• Coding scenarios: Mainly examines code generation, debugging, and repository-level code understanding. Evaluation signals rely heavily on “command-line execution error rate” and objective behaviors such as whether users directly copy code to run it.
• Chat scenarios: Although this appears to return to traditional Q&A, in the agent context this dimension leans more toward context retention and intent clarification in multi-turn conversations.

Notably, looking only at the overall ranking may mask a model’s weaknesses. For example, a model ranked high overall may perform only averagely on “steerability.” This means that when a user tries to change strategy midway, the model may not respond flexibly. Therefore, the evaluation system gives specific rankings in individual metrics (such as Bash reliability and correction capability) for developers in different business scenarios to select models on demand.

Interpreting Empirical Data: The Accumulation of Long-Horizon Tasks

From real-world data, statistical results based on more than 17,500 real long-horizon agent task sessions can reflect significant model capability stratification. Among the open-weight model camp, the top-ranked model reached +5.1% in overall Net Improvement Rate, while the previous-generation baseline model reached only +0.9%. This 4.2 percentage point jump means, statistically, that the model has made a substantive leap in simply increasing the probability that users “get the task done.”

A deeper analysis of its sub-scores shows that the model’s leading advantage does not depend on a single specialized skill. Instead, it ranks near the top in multiple fine-grained metrics, including Bash reliability (rarely making errors during Shell command execution) and direction adjustment and correction capability (responding quickly to temporary user changes). This reflects a trend: the new generation of strong agents is evolving from “being able to talk” toward “being able to get things done with less rework.”

Implications for Developers and Future Outlook

For developers building agent applications, this evaluation system based on real sessions and causal inference provides valuable reference points:

1. Return to real business value: Traditional score-chasing evaluation is being phased out. Future model selection will focus more on a model’s stable performance under real traffic pressure and its potential impact on user “word of mouth.”
2. Emphasize intervention and control: The logic of the evaluation system suggests that in engineering practice, we should not only focus on what the model generates but also on how the model’s behavior changes the user’s subsequent operation path. This suggests that developers should incorporate clearer feedback loop mechanisms into their designs.

In summary, agent evaluation is undergoing a paradigm shift from “subjective impression” to “objective causality.” This method, based on real session mining and causal tracing technology, is expected to become the “gold standard” for measuring the capabilities of next-generation agents. It prompts developers to reexamine: rather than pursuing empty fame on leaderboards, it is better to study in depth the “net improvement” a model brings in the real world.