Agent Evaluation’s New Paradigm: Real Conversation Signals and Causal Tracking

Why Voting-Based Evaluation Falls Short in the Agent Era

Over the past few years, mainstream LLM leaderboards have relied heavily on a pairwise voting mechanism: users ask the same question to two anonymous models, compare the answers, and vote for the better response. The system accumulates win rates and converts them into scores. This approach works reasonably well for conversational Q&A, where responses are manageable in length and evaluation focuses on a narrow set of dimensions such as accuracy, fluency, and usefulness. Users can typically make a reliable judgment within seconds.

Agent tasks, however, fundamentally change the complexity of evaluation.

1. Long task duration. A typical agent session may involve dozens of turns, spanning code modifications, file operations, and environment debugging. It is difficult for users to judge overall performance at the end of a session as reliably as they would judge a single response.
2. Ambiguous success criteria. A math problem has a standard answer, but whether “refactor this module for me” succeeds depends on user expectations, coding conventions, hidden constraints, and many other unstated conditions. There is no universal scoring standard.
3. Mid-task interventions are common. During agent execution, users frequently interrupt, correct direction, or add constraints. These interventions are themselves strong quality signals, but they are discarded by a simple final-vote mechanism.
4. Environment interaction introduces objective failure points. Command execution failures, missing dependencies, and permission errors are objectively measurable, but voting mechanisms cannot make use of this hard evidence.

As a result, the industry has begun exploring a new path: instead of relying on users to actively vote, evaluation signals are mined directly from real user agent sessions. Evaluation happens inside the task, rather than after the task ends.

Automatically Extracting Key Signals from Real Sessions

The core idea of the new paradigm is that real usage is itself the best evaluation environment. Without disturbing users, the system continuously collects agent session data and automatically identifies several key types of signals.

Explicit Confirmation Signals

When users give clear positive confirmation of an agent’s output—for example, accepting all modifications, confirming task completion, or adopting a generated plan—this constitutes a positive signal. Compared with a vague “continue” instruction, explicit confirmation is more credible because it represents an intentional judgment made after the user has inspected the output.

Praise and Complaint Signals

Sentiment polarity in session text also carries evaluation value. Statements like “this is exactly what I wanted” and “completely wrong, start over” correspond to positive and negative signals respectively. Automatic extraction typically requires intent classification based on context to avoid misinterpreting sarcasm or polite phrasing as genuine sentiment.

Correction Instruction Signals

When users issue instructions such as “not this direction,” “undo the change you just made,” or “you should have checked the config file first,” it means the agent’s previous execution deviated from expectations. The frequency and magnitude of corrections are direct indicators of an agent’s ability to get things right the first time—a good model should require less user intervention.

Shell Command Execution Error Signals

For agents with command execution capabilities, non-zero exit codes, compilation failures, and test failures returned by the shell are objective environment-level failures. These signals do not depend on subjective judgment and contain almost no annotation noise, making them the hardest type of signal among all categories.

From an engineering perspective, these signals are usually recorded in a structured format. A simplified signal record looks like this:

{
  "session_id": "sess_20240601_001",
  "model_version": "agent-x",
  "task_type": "coding",
  "turns": 37,
  "signals": [
    { "type": "explicit_confirm", "ts": "...", "polarity": "positive" },
    { "type": "correction",       "ts": "...", "polarity": "negative" },
    { "type": "shell_error",      "ts": "...", "exit_code": 1 }
  ]
}

Compared with voting data, these signals have three notable advantages: stronger resistance to fabrication because users are not consciously participating in an evaluation, built-in temporal and contextual information that helps locate where problems occurred, and coverage of real task distributions because long-tail complex tasks are also included in statistics.

Causal Tracking: Estimating Net Improvement Rate with Randomization Inference

After collecting signals, a naive approach is to directly compare the ratio of positive to negative signals across models. But this introduces serious confounding bias: sessions from different models may come from tasks with different difficulty distributions, users with different skill levels, and different engineering environments. Directly comparing averages can easily mistake “easier tasks” for “stronger models.”

The causal tracking method addresses this by borrowing ideas from randomization inference. The core logic can be summarized in three steps:

1. Define the net effect. For each session, combine positive and negative signals into a net effect value using weights. For example, confirmation counts as +1, praise as +1, correction as -1, and command error as -1. In practice, the weights are estimated by a statistical model rather than manually assigned.
2. Compare against the average. Calculate the mean net effect for the target model’s sessions and subtract the overall average across all evaluated models. The result is the model’s net improvement rate, meaning its net gain relative to the average level.
3. Randomization test. Through a large number of random permutations of the session-to-model mapping, construct the distribution under the null hypothesis that model quality is unrelated to signals. Then determine whether the observed net improvement rate is significantly beyond random fluctuation.

The following pseudocode illustrates the process:

def net_improvement_rate(sessions, target_model, n_perm=10000):
    # 1. Net effect per session: positive signal count - negative signal count (illustrative)
    def net_effect(s):
        return sum(1 if x.polarity == "positive" else -1 for x in s.signals)
    observed = (mean(net_effect(s) for s in sessions
                     if s.model == target_model)
                - mean(net_effect(s) for s in sessions))  # relative to the average level
    # 2. Randomly permute model labels to construct the null distribution
    null_dist = []
    for _ in range(n_perm):
        shuffled = permute_model_labels(sessions)
        null_dist.append(observed_statistic(shuffled, target_model))
    # 3. Significance: position of the observed value in the null distribution
    p_value = mean(v >= observed for v in null_dist)
    return observed, p_value

The value of this method is that it turns “which model is better” from a subjective voting question into a testable statistical hypothesis. That said, several caveats apply: confidence intervals will be wide when the sample size is insufficient; when task distributions drift over time, rolling re-estimation is needed; and extreme outlier sessions should be handled with truncation or robust statistical methods.

Breaking Down Evaluation Dimensions: Reliability, Correctability, and Overall Performance

Based on the signal system described above, evaluation can be further decomposed into several sub-dimensions that correspond directly to engineering pain points.

Shell command execution reliability measures the success rate of agent-executed commands and the model’s ability to self-recover after failures. It directly reflects how well the model understands environment syntax, dependencies, and error handling. For coding agents, this is the most fundamental baseline metric—an agent that cannot even get its environment running will struggle to deliver results no matter how strong its planning ability is.

Correctability measures whether the model can align with user intent within a small number of turns after receiving a correction instruction, rather than repeatedly drifting off course. This dimension has a major impact on real collaboration experience. A highly capable but hard-to-steer model may be less useful in practice than a somewhat weaker model that responds well to guidance.

Overall net improvement rate is a composite metric that combines weighted signals for cross-model ranking. It is worth noting that the composite metric and sub-dimension metrics may lead to different conclusions: a model may rank average overall but top the leaderboard on command execution reliability. This means evaluation results should always be interpreted in the context of your own business scenarios, rather than judged solely by a total score.

Implications for Developers: Building Your Own Agent Evaluation Loop

This methodology is not limited to public leaderboards. Any team can adapt it into their own evaluation system. A practical five-step framework includes:

1. Instrument signals. Add signal collection to the agent framework’s turn callbacks: user confirmation or rejection, correction instruction detection, command exit codes, and task abandonment events. Whether built on general-purpose logging and monitoring capabilities or a fully self-managed event pipeline, signals should be stored in a unified format and session content should be de-identified.
2. Structure the signals. Define type, polarity, timestamp, and associated turn for each signal category, forming a unified record format as shown in the earlier example. This lays the data foundation for subsequent statistical analysis.
3. Baseline and traffic splitting. Keep a stable model version as the baseline. Route a small percentage of traffic to the new version randomly for comparison, avoiding the attribution problem caused by mixing old and new versions in the same analysis.
4. Statistical testing. Calculate net improvement rate and significance using the randomization inference approach described earlier. Set a minimum sample size threshold to avoid being misled by small-sample fluctuations.
5. Continuous regression monitoring. Connect evaluation to monitoring and alerting. When a key sub-metric such as command execution reliability shows a significant regression, trigger an alert automatically—so model iterations receive regression testing similar to code changes.

In practice, two details are easy to overlook. First, distinguish between “task failure” and “the user changed their mind.” The latter should not be counted as a negative signal. Second, anti-abuse and spam prevention should be part of the design—for example, deduplicating signals within the same session and down-weighting abnormally high-frequency users.

The Rise of Open-Weight Models and Evaluation Transparency

Looking at the overall trend in public evaluations, the gap between open-weight models and top closed-source models on real agent tasks is narrowing. Some open-weight models have already reached the top tier on specific sub-dimensions. This has two implications for the industry:

• For users: Privately deployed models can also handle complex agent workloads. Teams with data sensitivity requirements now have meaningfully more choices. Model selection should rely more on real task signals than parameter counts.
• For the evaluation ecosystem: When evaluation is based on real session signals rather than human voting, auditability becomes more important. How signals are defined, how weights are estimated, and how confidence intervals are reported should be made public. Otherwise, rankings may still be gamed and lose credibility.

Conclusion

Agent evaluation is shifting from “asking users which model they prefer” to “observing what actually happens during tasks.” Real session signals address the problem of evaluation data authenticity, while randomization-based causal tracking addresses the problem of fair attribution. Together, they make it possible to measure model capability against real workflows for the first time. For developers, understanding and adapting this approach means building an objective, sustainable, regression-ready capability measurement system for the agents in their own products—which has far more long-term value than chasing any single leaderboard ranking.