Agent Evaluation on Real Tasks: From Preference Voting to Causal Tracking of Net Improvement Rate

1. Why Traditional Evaluation Methods Fail in Agent Scenarios

There is a fundamental difference between evaluating an agent and evaluating a single-turn QA model. Single-turn QA can be scored against reference answers or through pairwise preference voting. An agent, however, produces a long-horizon, multi-step behavioral trajectory that includes tool calls. On the same task, the trajectories of two models are often both “partially correct”: one completes the task but executes three redundant commands, while the other fails to finish but proactively asks the user for clarification midway through. If reviewers are asked to choose between these two trajectories, their judgments tend to be unstable, and inter-annotator agreement drops noticeably.

A more practical concern is cost. Manual pairwise voting requires evaluators to read through two conversation logs that may each span dozens of turns. Scaling this kind of annotation is almost impossible. Static benchmark suites, meanwhile, are prone to saturation and leakage and cannot reflect model robustness in real business environments. A new evaluation paradigm is therefore needed: instead of adding annotation burden, automatically extract evaluation signals from real sessions that have already occurred.

The core assumption behind this approach is that real users, after spending a long time working with an agent, reveal their judgment of model quality through their behavior. Users explicitly confirm task completion, express praise, complain, interrupt and correct the agent’s direction, or retry repeatedly after failed command executions. These are all high-value signals, and they naturally carry the distributional properties of “real tasks, real environments.”

2. Evaluation Methodology: Signal Mining and Causal Tracking

2.1 Automatic Extraction of Four Signal Categories

The first step in the evaluation pipeline is to extract structured signals from session logs. Signals can be grouped into four categories:

Signal Category	Specific Manifestation	Semantic Direction
Task confirmation	User explicitly says “that works,” “exactly,” or “the result is correct”	Positive
Praise and complaints	Explicit positive or negative evaluative statements	Positive / Negative
Correction instructions	User interrupts and asks for a change of direction, a redo, or additional constraints	Negative (drift)
Execution errors	Commands return non-zero exit codes, timeouts, permission denials, or parsing failures	Negative (reliability)

Signal extraction typically combines rules with lightweight models. Command execution errors can be parsed directly from the tool call return structure and are the most deterministic. Confirmation and praise/complaint signals require text classification. Correction instructions require identifying the “interruption—restatement” pattern in a dialogue, which can be detected by combining turn intervals, instruction similarity, and intent classification.

It is important to note that raw signals are evidence, not scores. Session lengths vary greatly across tasks, and longer tasks naturally have more opportunities for corrections and execution errors. Without normalization, evaluation results degrade into a “task difficulty leaderboard.”

2.2 Causal Tracking: Using Randomization-Based Inference to Remove Confounding Factors

The biggest problem with real session data is confounding. A model assigned to more complex tasks or serving a more demanding user base will naturally produce worse raw signals. Directly comparing the average signals across models leads to biased conclusions.

The idea behind “causal tracking” is to frame evaluation as an inference problem from a randomized experiment. Assuming sessions are assigned to models randomly, or approximately randomly, the net improvement of each session can be defined relative to an “average level.” Formally, the net improvement rate for model m can be written as a weighted average of the per-session treatment effects:

# Pseudocode: randomization-based estimation of net improvement rate
# sessions: each session contains model_id, task_type, and signals
# signals: normalized counts of positive/negative evidence
def net_improvement(sessions, model_id, baseline="global_mean"):
    effects = []
    for s in sessions:
        # Expected signal strength for this session across candidate models
        # (comparability guaranteed by random assignment)
        expected = baseline_signal(s.task_type, baseline)
        observed = normalized_signal(s.signals) if s.model_id == model_id else None
        if observed is not None:
            effects.append(observed - expected)
    # Stratified weighting by task type and session length
    # to prevent long sessions from dominating
    return stratified_weighted_mean(effects)

The key is the “relative to the average level” framing. Rather than pursuing absolute scores, the goal is to estimate how much better or worse each model is than the average across all participating models. This reduces the need for absolute annotation standards and makes results comparable across evaluation batches—adding a new model does not substantially raise or lower the scores of existing models.

2.3 From a Single Composite Score to Diagnostic Sub-Metrics

A single composite score is insufficient for guiding optimization. An evaluation system usually needs to break out sub-dimensions, such as:

• Command execution reliability: the proportion of successful tool calls and the ability to self-recover after failures;
• Steerability: after a user issues a correction, whether the model converges correctly in the next turn instead of repeating the original error;
• Task completion confirmation rate: the proportion of sessions that end with explicit user confirmation;
• Redundant action rate: the share of unnecessary tool calls beyond the steps required to complete the task.

The value of sub-metrics lies in diagnosability. Two models with similar composite scores may differ fundamentally: one may be “steady but slow,” while the other is “fast but prone to drift.” Their optimization paths are completely different.

3. Engineering Implementation Considerations

3.1 Data Pipeline Design

A practical evaluation pipeline typically consists of four layers:

1. Collection layer: session logs, tool call records, exit codes, and latency;
2. Extraction layer: signal extraction and normalization, producing a structured evidence table;
3. Inference layer: stratified randomization-based estimation, outputting each model’s net improvement rate and confidence intervals;
4. Presentation layer: overall leaderboard, sub-metric leaderboards, and task-type-specific leaderboards.

Privacy and compliance require particular attention at the collection layer. Session content should be used only after de-identification, retaining only behavioral features relevant to evaluation. The retention period for raw text should be explicitly defined.

3.2 Sample Size and Confidence Intervals

Net improvement rate is an estimate and must be reported with uncertainty. Recommendations:

• Use stratified sampling by task type and enforce a minimum sample size per stratum;
• Use bootstrap resampling to estimate confidence intervals;
• Mark models with insufficient sample sizes as “insufficient data” rather than assigning them a rank directly.

A common engineering mistake is ignoring intra-session correlation. Multiple sessions from the same user are not independent samples; merging them directly underestimates variance. Robust variance estimation can be performed by clustering at the user level.

3.3 Anti-Gaming and Contamination Resistance

Open evaluation faces two types of risk: models may be optimized specifically against the evaluation signals, for example by over-producing “task completed” language to trigger confirmation signals, and session data may be contaminated. Mitigations include:

• Signal detection should not rely on fixed phrasing but on subsequent user behavior, such as whether corrections still occur after a confirmation;
• Periodically sample sessions for manual review to calibrate consistency between automatic signals and human judgment;
• Run distribution checks on abnormally high frequencies of positive signals and flag anomalies.

4. Interpreting a Real Evaluation Example

Take the results of an open-weight model participating in an evaluation as an illustrative example. On more than 17,500 real long-horizon sessions, the model achieved a composite net improvement rate of +5.1%, up about 4.2 percentage points from its previous generation’s +0.9%. It ranked first among open-weight models and 13th on the full leaderboard. It also ranked first among open-weight models on general work, coding, and conversational tasks, and led open models on both command execution reliability and steerability.

The interpretation of these numbers deserves emphasis:

• The absolute value of net improvement rate should not be compared across different leaderboards. It should only be interpreted within the same evaluation system and the same batch;
• What +5.1% means: relative to the average of all participating models, the model is about 5 percentage points better on normalized signals, which falls into the “consistently positive but not overwhelming” range;
• Sub-metric rankings carry more information than the overall ranking: leadership in command execution reliability and steerability suggests the model’s advantage lies in “making fewer mistakes and responding well to corrections” rather than raw reasoning ability alone;
• The 4.2 percentage point generational improvement is more informative than the absolute ranking because it controls for drift in the evaluation system itself.

5. Implementation Checklist

1. Define signals before discussing rankings. If signal definitions are unstable, rankings are not comparable. Write signal definitions as executable rules and version them.
2. Stick to relative framing. Net improvement rate is preferable to absolute scores, especially for continuously iterating model lineages.
3. Stratify before aggregating. Task type, session length, and user segment should all be treated as stratification variables to avoid Simpson’s paradox.
4. Prioritize sub-metrics over the composite score. Use the composite score for external communication and sub-metrics for internal optimization.
5. Report uncertainty. A leaderboard without confidence intervals has limited value for engineering decisions.
6. Maintain a manual review channel. Automatic signals are efficiency tools, not a source of truth. Periodic calibration significantly improves the credibility of the system.

Conclusion

The challenge of agent evaluation is not “scoring” but obtaining trustworthy causal comparisons from real data full of confounding factors. The evaluation paradigm described here—using real sessions as the data source, multiple behavioral signal categories as evidence, and randomization-based inference as the estimation framework—converts annotation cost into engineering cost while preserving the most valuable property: the distribution of real tasks. For teams building or selecting agent capabilities, it is worth prioritizing the construction of such an internal evaluation baseline. It may not provide absolute conclusions, but it is sufficient to answer the most critical engineering question: “Is this version better than the previous one?”