From Real Conversations to Causal Inference: A Methodological Guide to AI Agent Evaluation

Why Evaluating AI Agents Is More Difficult

As large language models evolve from “conversation assistants” to “task executors,” evaluation complexity rises accordingly. Traditional LLM benchmarks revolve around static datasets—multiple-choice questions, code generation, and math reasoning—where scores are straightforward to compare. However, applying this approach to AI agents exposes three fundamental issues:

1. Data contamination and benchmark gaming. Questions from public benchmarks are likely already present in training corpora. Models may memorize answers rather than solve problems, causing leaderboard scores to diverge from real-world capability.
2. Task format mismatch. Agent work is typically long-horizon and multi-turn: reading code, modifying files, executing commands, interpreting errors, and retrying. A single task can span dozens of minutes. Single-turn question-answering metrics cannot capture core capabilities such as goal persistence, error recovery, and tool invocation.
3. Limitations of human voting. Arena-style pairwise voting relies on users actively providing feedback, but preferences are easily swayed by presentation style—“sounding good” is not the same as “getting the job done.” Moreover, agent tasks are long, and users rarely return to vote after finishing; samples are sparse and subject to selection bias.

These contradictions have driven a new evaluation philosophy: rather than asking users to vote, extract rating signals automatically from behavior traces embedded in real user–agent conversations.

From Active Voting to Passive Signal Mining

The core premise of this paradigm is that authentic sessions are the most trustworthy evaluation data. During a complete task session, users naturally produce ample feedback without any additional effort. Four types of key signals stand out:

• Task confirmation signals: The user explicitly states that the task is complete or the result meets expectations—the strongest positive evidence.
• Praise and complaints: Explicit emotional expressions in conversation with clear positive or negative polarity.
• Correction instructions: When users are forced to issue corrective commands like “No, it should be…”, the model’s previous output deviated from expectations. This is a high-value negative signal.
• Execution errors: Shell command failures, tool-call exceptions, and repeated retries—objective signals accurately captured from logs.

These signals can be categorized along two dimensions—explicit/implicit and positive/negative—and assigned different confidence levels. The following JSON illustrates the structure of extracted signals (provided for conceptual clarity, not as an interface definition of any real system):

{
  "session_id": "ag-session-0001",
  "turns": 23,
  "duration_minutes": 41,
  "signals": [
    { "type": "correction",       "turn": 7,  "polarity": "negative", "confidence": 0.93 },
    { "type": "shell_error",      "turn": 9,  "exit_code": 1, "polarity": "negative", "confidence": 0.99 },
    { "type": "explicit_confirm", "turn": 22, "polarity": "positive", "confidence": 0.96 }
  ]
}

Compared to human voting, this approach offers three advantages: samples come from continuously flowing real traffic at scale over long periods; users behave naturally without the psychological burden of being evaluated; and signals are generated from the same task context as the production environment, aligning evaluation criteria with operational reality.

Causal Tracking: Making the Statistics Defensible

Once signals are available, can we simply count which model has more errors per session? No. Raw session data contains many variables unrelated to model capability:

1. Task difficulty varies: Models assigned to complex tasks naturally produce higher error rates.
2. User groups differ: Experienced developers correct errors and tolerate failures differently from novices.
3. Temporal distribution varies: The mix of task types differs between peak and off-peak hours.

Without controlling these confounders, a leaderboard merely reflects “who drew the easier tasks.” The solution borrows from randomized controlled trials: route user requests to different models randomly, so that the task distribution and user distribution across models are statistically similar. Under this design, outcome differences can be attributed more convincingly to model capability rather than task differences.

Building on this foundation, the evaluation platform computes a core metric—Net Improvement Rate—which measures, relative to the average performance of all participating models, the extent to which a given model produces more positive signals and fewer negative signals on comparable tasks. A conceptual formulation is:

Net Improvement Rate ≈ (weighted signal score for sessions of model − baseline score for same-distribution random sessions) / |baseline score|

It must be emphasized that the actual implementation is far more complex than this formula suggests: stratified randomization, confidence interval estimation, and multiple-comparison corrections are typically involved. Net Improvement Rate should be reported as an interval rather than a single point. For long-horizon agent tasks, collecting a single session is costly, so sample size and statistical power require careful trade-offs. Additionally, because evaluation material originates from continuously flowing real traffic and test tasks are not publicly disclosed, such leaderboards are inherently harder to optimize against than static benchmarks.

A Multi-Ruler Indicator System: One Composite Score, Multiple Perspectives

The composite Net Improvement Rate answers “who is better overall,” but developers often find granular metrics more decision-relevant. Common breakdown dimensions include:

• Command execution reliability: Success rate of shell/tool invocations and autonomous recovery after errors—the primary metric for automation, DevOps, and infrastructure scenarios.
• Correctability: Whether the model quickly understands intent and returns to the right path after receiving user correction instructions—this defines the lower bound of interactive collaboration quality.
• Long-horizon consistency: Whether the model maintains goal focus across multi-turn, multi-hour tasks without drifting.
• Engineering task performance: End-to-end quality of code comprehension, modification, and verification.

For example, if you are building an unattended automated pipeline, command reliability should outweigh correctability; for a pair-programming-style interactive assistant, correctability matters more. The composite score suits initial screening, while granular metrics guide model selection.

Practical Recommendations for Developers

This methodology is not confined to evaluation platforms—it can be adopted directly in enterprise agent development:

1. Evaluate against real workloads. When selecting internal models, collect genuine task traffic from your own business in addition to public benchmarks, avoiding the “high leaderboard score, production failure” trap.
2. Record signals, not scores. Structure session logs while preserving objective traces such as task outcomes, subsequent user actions, and command exit codes, leaving sufficient raw material for post-hoc analysis.
3. Ensure fairness through randomization. When comparing multiple models, use random traffic splitting to control the distribution of task difficulty and user groups; only then can conclusions carry causal weight.
4. Focus on failure recovery. First-attempt success matters, but retry strategies, error comprehension, and self-correction after failures often distinguish stronger models more effectively.
5. Monitor the open-ecosystem landscape. In recent years, open-weight models have steadily narrowed the gap with closed-source flagships on complex agent tasks. For teams with private deployment or data compliance requirements, the selection space is expanding.

A minimal viable internal evaluation pipeline can be assembled as follows:

Real requests → Random routing (model A/B/...) → Session log collection
        → Signal extraction (confirmation/complaint/correction/execution error)
        → Weighted aggregation → Net Improvement Rate + confidence interval → Evaluation report & regression dashboard

Conclusion

The evaluation framework determines the direction of optimization. Moving from “asking users to vote” to “tracing causality in real conversations,” agent evaluation is becoming more aligned with production reality, harder to game, and more interpretable. For developers, understanding this methodology offers more than the ability to read leaderboards—applying signal mining and randomized comparison thinking to your own iteration cycles is the key to continuous improvement of agent capabilities.