Agent Evaluation Beyond Chat Quality: Mining Real Session Signals and Causal Tracking

1. From “Chats Well” to “Gets Things Done”: Why Agent Evaluation Is Getting Harder

Once large models enter the agent stage, evaluation difficulty changes qualitatively. Traditional dialogue model evaluation generally relies on “pairwise voting”: users compare two responses side by side and pick the better one. This approach is cheap and produces large samples in chat scenarios, but applying it to agents runs into four structural problems:

1. Long-horizon, multi-stage tasks. A real agent task often spans dozens of interaction turns, covering requirement understanding, planning, tool invocation, execution and verification, and correction based on feedback. A single response looking good does not represent the delivery quality of the whole task chain.
2. Outcomes have objective right and wrong. Chat response evaluation is largely a matter of preference, while agents execute commands, modify files, and run tests—command errors, compilation failures, and failing tests are factual failures that cannot be explained away as “style preference.”
3. Human voting cannot cover execution details. Voting only captures a user’s overall impression and cannot quantify process information such as how many commands were executed, how many retries occurred, and how many errors were made—yet these are exactly where agent capability differences concentrate.
4. Strong environment coupling. The same model performs very differently across operating systems, permission configurations, and tool sets. Dedicated benchmarks detached from real usage environments easily become “exam questions” that diverge from online behavior.

As a result, the industry is converging on a new evaluation consensus: stop relying on specially organized voting and instead use real users’ agent sessions themselves as the evaluation data source, automatically extracting evidence from massive everyday usage. This solves the sample authenticity problem and refines evaluation granularity from “overall impression” to “process behavior.”

2. Signal Engineering: Extracting Four Categories of Key Evidence from Raw Sessions

Real sessions are naturally noisy: there is small talk, abandoned tasks, and users’ own misoperations. The first core job of an evaluation system is signal engineering—defining a signal system that can be automatically recognized and cross-validated. In practice, four categories of signals are typically targeted:

Signal type	Typical form	Capability reflected	Noise handling focus
Explicit confirmation	“That works” / “Let’s go with this plan”	Task completion	Filter pleasantries and responses with no substantive content
Praise and complaints	“The reasoning is clear” / “Completely missed the requirement”	Overall satisfaction	Requires dual filtering by rules and a classification model
Correction instructions	“That’s not what I meant, rewrite it” / “Change it to async”	Understanding deviation, steerability	Distinguish directional corrections from minor detail tweaks
Execution errors	Shell command errors, failing tests	Tool invocation reliability	Exclude problems caused by the user’s own environment

Several important engineering lessons:

1. Signals must align with session stage. The same phrase “do it again” means something completely different at the requirement clarification stage versus the delivery acceptance stage. Signals should be bound to “which turn, targeting which artifact.”
2. Objective signals take precedence over subjective ones. A user may verbally say “done,” but if subsequent commands keep failing, the objective signal should prevail—subjective expression may stem from politeness or fatigue.
3. The “magnitude” of a correction is more informative than the “count.” A user demanding a full redo differs in severity from changing a single variable name; statistics can be graded by scope of change.
4. Privacy and compliance come first. Sessions often contain code, paths, and key fragments, so desensitization and permission isolation must be completed before storage.

3. Causal Tracking: The Statistical Foundation That Makes “Net Improvement Rate” Credible

Once signals are obtained, the most intuitive approach is to group by model and average the scores. But real traffic is not randomly assigned: different models naturally differ in user population, task difficulty, and usage duration. Directly comparing means conflates “model capability” with “task difficulty and user habits”—a model may simply happen to receive more easy tasks.

To approximate causal conclusions from observational data, an approach based on randomization inference can be used (often called “causal tracking” in the industry):

1. Introduce randomization on the traffic side. For experimentable sessions, route requests to different models with predefined probabilities to construct statistically comparable control groups, similar to the traffic splitting idea in online experimentation.
2. Use randomization tests for historical data. For existing sessions that cannot be randomly assigned, construct a “null hypothesis distribution” via permutation tests and other methods to determine whether observed differences significantly exceed random fluctuation.
3. Output a net improvement rate. Define a model’s net improvement relative to the average level of all participating models: a positive value means above average, a negative value means below average. It is a relative metric, and its advantage is comparability and aggregation across tasks.

Below is illustrative pseudocode showing the calculation of the net improvement rate and the significance test idea:

# Illustrative code: randomization inference for net improvement rate (pseudocode, not a production implementation)
def net_improvement(scores_by_model, n_perm=10000):
    baseline = mean(all(scores_by_model.values()))      # average across all models
    observed = mean(scores_by_model["model_a"]) - baseline
    null_dist = []
    for _ in range(n_perm):
        shuffled = randomly_reassign_labels(scores_by_model)  # shuffle model labels
        null_dist.append(mean(shuffled["model_a"]) - baseline)
    p_value = mean([d >= observed for d in null_dist])  # randomization test
    return {"net_rate": observed, "p_value": p_value}

Two caveats when using the net improvement rate: first, it is a relative metric that fluctuates with the set of participating models, so it should be interpreted alongside absolute metrics such as task completion rate and error rate; second, randomized traffic splitting must follow basic experimental design norms—splitting granularity, observation period, and sample size estimation must all be defined in advance, or the conclusions will be equally unreliable.

4. Segment Metrics Worth Close Attention

Beyond overall rankings, the following segment dimensions are more instructive for engineering decisions:

1. Command execution reliability. Measure the error rate, invalid retry rate, and fatal error rate when agents execute Shell commands. Low reliability not only slows tasks down but also multiplies session costs—this is currently one of the “hardest” metrics for distinguishing agent model tiers.
2. Steerability. After being corrected by a user, can the model quickly return to the right direction, and does it repeat the same type of error in later turns? It measures “collaborability,” which is especially critical for long-term collaboration scenarios such as pair programming.
3. Long-horizon task completion rate. The proportion of tasks spanning multiple turns or even multiple workdays that are ultimately delivered successfully. This metric is extremely sensitive to context management and planning capability.
4. Efficiency metrics. The number of interaction turns and token consumption required to complete equivalent tasks, which map directly to usage cost.

Looking at the overall industry trend, the gap between open-weight and closed-source models on agent tasks is narrowing, and on some dimensions open models have already reached the first tier; but closed-source models are usually still more stable in long-horizon planning and complex tool orchestration. For technical teams, weighting segment metrics according to their own task distribution is more useful than looking only at overall scores when selecting a model.

5. How Enterprises Can Build Their Own Agent Evaluation Systems

The methodology above is not exclusive to evaluation institutions—enterprises can absolutely build an internal evaluation loop based on their own traffic. A five-layer approach is recommended:

1. Data layer. Persist agent sessions in a structured way, aggregating them with general-purpose cloud capabilities: centralized collection via logging services, and raw session snapshots retained in object storage. Mainstream cloud platforms such as Baidu AI Cloud offer such general-purpose capabilities, while the evaluation logic itself remains platform-agnostic.
2. Signal layer. Extract the four categories of signals using a “rules engine + lightweight classification model,” build a signal dictionary, and version-manage it to ensure metric definitions are traceable.
3. Metric layer. Fix the calculation definitions of metrics such as net improvement rate, execution reliability, and steerability; connect them to dashboards and monitoring alerts so that abnormal fluctuations (such as a sudden error rate spike in a version) reach the owner immediately.
4. Experiment layer. Before launching a new model or version, run a small-traffic randomized control to observe the net improvement rate and segment metrics, and only ramp up after targets are met—avoiding “discovering a regression only after a full switch.”
5. Iteration layer. Feed online failure cases back into a regression test set, forming a closed loop of “online evaluation → offline regression → model and prompt iteration → re-launch verification.”

A minimal viable pipeline can be summarized as: session collection → signal extraction → metric calculation → dashboard alerts → case feedback. Most steps can be assembled from managed cloud components; the real differentiation lies in the quality of the signal dictionary and the rigor of metric definitions.

6. Summary

Agent evaluation is moving from “human voting” to “real session signal mining + causal inference”: the former solves sample authenticity, the latter solves attribution credibility. For developers, when reading various leaderboards, three things should be confirmed—whether signal sources are real, whether statistical methods exclude confounding, and whether segment metrics cover the scenarios you care about. For engineering teams, rather than trusting a single ranking, it is better to build your own evaluation loop early so that every model upgrade is backed by data.