Agent Evaluation That Works: Mining Real Session Signals and Causal Inference

Why the Evaluation Paradigm Must Evolve

In the first half of the large model race, the dominant evaluation method was pairwise comparison plus human voting: two models answer the same question, users pick the better response, and aggregated votes produce arena-style scores. This approach works well for single-turn Q&A—the answer is the result, and comparison is judgment.

Agents change the premises:

1. Task shape has changed. Agents handle long-horizon, multi-step tasks: reading codebases, editing files, executing commands, looking up information, and continuously adjusting based on feedback. Quality is no longer determined by “how good a particular response is” but by “whether the whole job got done.”
2. The user’s role has changed. In voting scenarios the user is a judge making a single choice; in agent scenarios the user is a collaborator who issues instructions, interrupts, corrects course, and accepts results. Evaluation information is scattered across the entire interaction rather than concentrated in a single click.
3. Preference does not equal outcome. An agent whose response “looks nice” but leads the task astray may score well in voting while being a disaster in production. Evaluation must shift from “which one does the user prefer” to “was the user’s task actually advanced.”

As a result, the industry is converging on a new evaluation approach: instead of organizing dedicated comparison experiments, return to real users’ actual usage and mine evidence from massive volumes of naturally occurring agent sessions. This is the key backdrop for understanding today’s various “real-task evaluations.”

Automatically Extracting Evaluation Signals from Real Sessions

Real sessions are “dirty” data: users do not respond in evaluation formats, and feedback arrives as natural language, operational behavior, or even silence. To evaluate, these fuzzy evidence must first be converted into computable signals. Four categories have proven stable and usable:

1. Explicit confirmation. The user approves the output in unambiguous terms—confirming a plan, accepting a result, saying “that’s it.” This is the strongest positive signal because it is a completion judgment actively given by the user rather than a system inference.
2. Praise and complaints. Emotional expressions of satisfaction or dissatisfaction with the agent. Weaker than explicit confirmation but large in volume and broad in distribution, making them suitable for trend indicators. Extraction must distinguish “praise for the process” from “praise for the result” and filter out polite formalities.
3. Correction instructions. Statements where the user interrupts the current direction, points out an error, or resets the goal. These signals carry dual meaning: in the short term they are negative events (something went wrong); in the long term they are raw material for measuring steerability—a good agent returns to the right track quickly after correction, while a poor one repeatedly drifts off course.
4. Execution-level errors. Structural facts such as non-zero shell exit codes, retries after errors, and command timeouts. This is the “hardest” signal: it does not depend on natural language understanding, is barely contaminated by subjective expression, and is especially critical for coding agents.

In engineering terms, signal extraction typically uses a hybrid “rules + model” pipeline:

# Illustrative code: extracting evaluation signals from a session
def extract_signals(session):
    signals = []
    for turn in session.turns:
        if turn.role == "tool":
            # Use rules for structured fields: exit codes, retry counts, etc.
            if turn.exit_code != 0:
                signals.append({"kind": "exec_error", "weight": 1.0})
        elif turn.role == "user":
            # Use a classification model for natural language semantics
            label = classify(turn.text,
                             labels=["confirm", "praise", "complaint",
                                     "correction", "neutral"])
            if label != "neutral":
                signals.append({"kind": label, "weight": 1.0})
    return signals

Three engineering details that are easy to overlook:

• Noise governance. Users’ own operational mistakes, complaints unrelated to the agent, and misjudgments caused by missing context all pollute signals. Classifiers need continuous spot checks, and low-confidence samples should be down-weighted.
• Attribution boundaries. A failed command may result from the model writing the wrong command or from the environment lacking a dependency. Evaluation systems should record environment context wherever possible to avoid blaming the model for environment issues.
• Privacy compliance. Session content is sensitive; desensitization and minimal storage are prerequisites for all real-data evaluation.

Causal Tracking: Estimating Net Improvement Rate with Randomized Inference

Once signals are available, can you directly compare “Model A’s average session score is higher than Model B’s”? No. Real sessions are observational data and inherently confounded: different models serve different user populations (novices and experts have different task difficulties), are used in different scenarios (some write scripts, some do data analysis), and task difficulty itself is uncontrolled. Comparing raw averages may measure “whose users are harsher” rather than “which model is stronger.”

The idea of “causal tracking” is to place each model’s performance in the same frame of reference—the average level across all sessions:

1. Define net outcome. Aggregate all signals within a session into a net score: confirmations and praise count positive; complaints, corrections, and execution errors count negative. After offsetting, you get “did the user come out ahead or behind in this session.”
2. Compute the net improvement rate. A model’s net improvement rate is the proportional deviation of its net score across all sessions relative to the overall average. It answers: handing the task to this model, compared with “using some average-level model,” how much better is the net outcome for the user’s task.
3. Use randomized inference for significance testing. Randomly regroup the full session pool (permutation test) to construct a score distribution under the null hypothesis that “this model is not better than average,” then observe where the true net improvement rate falls in that distribution to obtain a p-value and confidence interval.

# Illustrative code: permutation test for the significance of net improvement rate
import random
def net_improvement_rate(model_sessions, pool, n_perm=10000):
    def net(ss):
        return sum(s["score"] for s in ss) / max(len(ss), 1)
    observed = net(model_sessions) - net(pool)   # deviation from overall mean
    null_dist = []
    for _ in range(n_perm):
        shuffled = random.sample(pool, len(model_sessions))
        null_dist.append(net(shuffled) - net(pool))
    p = sum(1 for x in null_dist if x >= observed) / n_perm
    return observed, p

The advantages of this method: it does not rely on strong assumptions about data distribution, naturally handles unbalanced sample sizes, and its conclusions can be stated directly as “in real tasks, this model delivers on average X net improvement”—which is more instructive for engineering decisions than an abstract arena score. It does require sample scale: narrowing confidence intervals for细分 dimensions like “command execution reliability” and “correction capability” to a usable level typically requires thousands to tens of thousands of real sessions, which is why such evaluation systems must be operated long-term with continuous data collection.

Fine-Grained Dimensions Worth Watching in Real-Task Evaluation

Beyond the total score, fine-grained dimensions often expose structural differences between models more clearly:

1. Shell command execution reliability. Non-zero exit rate, retry success rate after failure, and accidental execution of dangerous commands. For coding and operations agents, this is a hard, veto-level metric: if the underlying execution is unreliable, no amount of上层 planning can deliver.
2. Steerability. How many turns it takes to return to the correct track after a correction instruction, and whether the same errors recur. It measures the cost of “managing” the model and directly determines the human supervision burden in long tasks.
3. Long-horizon task completion rate. Whether tasks spanning multiple turns, files, and tools can be reliably closed out, and what proportion collapse midway or silently drift off course.
4. Scenario breakdown. Rankings within categories such as work, coding, and conversation. A model with a high total score may well be weaker than a lower-scoring specialist model in coding scenarios; breakdown data is key to avoiding overgeneralization during model selection.

Implications for Developers: Building Your Own Evaluation Loop

Public evaluations provide horizontal reference, but every team’s prompts, toolchains, and task distributions are unique, and leaderboard scores do not transfer directly. A more reliable approach is to follow the methodology above and build an evaluation loop on your own business data:

1. Instrument first. Session logs, tool-call results, and explicit user feedback should all be structured and stored. Such data pipelines can be built on general cloud platform capabilities—for example, using log services to aggregate session events, object storage to retain raw sessions, and monitoring and alerting to track anomalous fluctuations in signal metrics. Major cloud platforms (such as Baidu AI Cloud) offer corresponding product categories.
2. Define a “signal dictionary” before automating. Clarify what counts as “confirmation” and what counts as “correction” in your business, write annotation guidelines, manually label a few hundred examples to calibrate the classifier, and then scale to full automated extraction.
3. Use randomized traffic splitting to ensure comparability. When onboarding a new model in canary releases, split traffic randomly by request to ensure different models face the same task distribution—this is the precondition for causal interpretability of comparisons.
4. Weight and look at breakdowns. Assign weights to fine-grained metrics based on your own scenarios—teams doing operations automation should substantially increase the weight of command execution reliability, while teams doing customer-service automation should focus more on steerability and complaint rates.
5. Beware of overfitting to public benchmarks. Leaderboard scores reflect performance on the “average user”; your private workflows, private codebases, and domain terminology can all cause significant shifts in performance. Final judgment should be delegated to your own session data.

Summary

Agent evaluation is undergoing a paradigm shift: from “organizing experiments and collecting preferences” to “mining real sessions and inferring causal effects.” Its core chain can be summarized in three steps—using real sessions as the data source, automated signal extraction as the measurement method, and randomized inference as the statistical tool, ultimately producing a net improvement rate that reflects production reality. For developers, understanding this methodology matters not only for interpreting various evaluation results but also for reusing the same approach to build a continuous, objective, and attributable model evaluation system on their own business, so that every model upgrade decision is backed by real data.