Agent Evaluation’s New Paradigm: Real Conversation Signals and Causal Tracking

1. From exams to real work: Why agent evaluation has to change

Traditional LLM evaluation usually follows one of two paths. The first is static benchmarks, which use fixed question sets to measure knowledge and reasoning. The second is side-by-side voting, where users choose between answers from two models. These approaches are generally sufficient for chat scenarios, but their weaknesses become obvious once the subject is an agent.

Agent tasks are multi-step and long-running. A typical task may involve dozens of conversation turns, multiple tool calls, file reads and writes, and environment state changes. Static question sets cannot easily cover such a complex task distribution, and once questions are public, models can overfit to them, producing inflated scores. Side-by-side voting requires reviewers to compare two complete execution processes, which is expensive. Ordinary users lack both the patience to replay dozens of tool-call steps and the expertise to judge which step went wrong. As a result, preference signals collected through voting are sparse and noisy, which makes fine-grained model ranking difficult.

A more sensible approach is to stop creating a separate exam environment and instead return to production. Use real user-agent conversations as the evaluation data source, then automatically extract quality signals from those sessions. This is a major paradigm shift in agent evaluation: moving from artificial tasks plus human voting toward real tasks plus automatic signals plus causal inference.

2. Real conversations: An evaluation data source closer to production

Using real conversations as the data source has three main advantages.

1. The task distribution is realistic. Users raise long-tail tasks such as debugging environments, writing scripts, troubleshooting failures, and cleaning up data. These tasks rarely appear in manually constructed test sets, yet they are exactly the high-frequency scenarios where agents are used.
2. Samples are generated continuously. Evaluation no longer depends on one-time collection. New samples accumulate as the product operates, so results can be updated weekly or monthly, reducing the problem of stale question sets.
3. Feedback exists naturally. Confirmation, complaints, and corrections from users during a conversation are themselves quality signals. There is no need to design a separate scoring step.

Of course, this route has strict prerequisites. Conversation content must be desensitized and used with user authorization, following the principle of minimal collection. Tasks must also be sampled in a stratified way to ensure that different models face a broadly similar difficulty distribution. Otherwise, later statistical comparisons are meaningless.

3. Four key signals: Turning subjective experience into measurable indicators

Real conversations are raw material. The core of evaluation is extracting stable, objective signals from them. The industry generally focuses on four types.

• Explicit confirmation: The user clearly says something like “that works” or “that is exactly what I wanted.” This is a strong signal that the task has been accepted as complete, and it carries the highest weight.
• Praise and complaints: Positive and negative feedback with emotional intent is more nuanced than binary success and failure. It can reflect partial completion, experience quality, and other intermediate states.
• Correction instructions: The user interrupts and asks for a different direction, such as “no, try another approach.” This measures steerability, meaning how quickly the model can converge after being told it is wrong.
• Tool execution errors: Shell commands returning non-zero exit codes, script runtime errors, and similar failures. These signals are fully objective and can be collected automatically, with almost no dependence on human judgment.

Signal extraction usually uses a two-channel approach combining rule templates and a discriminative model. High-frequency signals with clear patterns are captured by rules, while semantically ambiguous feedback is handled by the model. A small proportion of model judgments should be spot-checked by humans to calibrate accuracy. The logic looks like this:

def extract_signals(session):
    signals = []
    for step, event in enumerate(session.events):
        if event.type == "user_message":
            if match_confirmation(event.text):      # "Great, that works"
                signals.append(("confirm_positive", step))
            elif match_complaint(event.text):       # "This is not what I asked for"
                signals.append(("feedback_negative", step))
            elif match_correction(event.text):      # "Try another approach, check logs first"
                signals.append(("correction", step))
    for call in session.tool_calls:
        if call.kind == "shell" and call.exit_code != 0:
            signals.append(("tool_error", call.step))
    return signals

4. Causal tracking: Making net improvement rate defensible

Once signals are available, the next step is comparing models. The biggest pitfall here is confounding factors. It is not fair to simply say that model A has a higher confirmation rate than model B, because model A may have received easier tasks. To make conclusions credible, causal inference thinking is required.

The core idea of causal tracking is to treat online model assignment as approximately random and use randomization-based inference for testing.

1. Define net improvement rate. For similar tasks, subtract the average signal rate across all models from a given model’s signal rate, such as confirmation rate. This difference removes common fluctuations caused by task difficulty and time period. It measures the model’s net contribution relative to the average.
2. Randomization test. Under the null hypothesis that model assignment is unrelated to task difficulty, use permutation testing to repeatedly shuffle assignment labels. Observe where the real difference falls within the random distribution to obtain significance level and confidence interval.
3. Stratified control. Stratify by task category, conversation length, user activity, and similar dimensions. Compare within strata first, then aggregate. This further reduces the influence of confounding factors.

A simplified permutation test looks like this:

import random
def net_improvement(sessions, target_model, n_perm=10000):
    def confirm_rate(ss):
        return sum("confirm_positive" in s.signals for s in ss) / max(len(ss), 1)
    obs = confirm_rate([s for s in sessions if s.model == target_model]) \
        - confirm_rate([s for s in sessions if s.model != target_model])
    labels = [s.model == target_model for s in sessions]
    diffs = []
    for _ in range(n_perm):
        random.shuffle(labels)
        a = confirm_rate([s for s, l in zip(sessions, labels) if l])
        b = confirm_rate([s for s, l in zip(sessions, labels) if not l])
        diffs.append(a - b)
    p_value = sum(d >= obs for d in diffs) / n_perm
    return obs, p_value

In a real system, stratification, weighting, and bootstrap confidence intervals should also be added. When reporting results, net improvement rate must be presented together with confidence intervals. “Improved by 3 percentage points” and “improved by 3 ± 0.5 percentage points” have completely different decision-making value.

5. Granular metrics and scenario-specific leaderboards: Avoiding a single misleading score

The overall net improvement rate answers the question “is the model stronger overall,” but engineering decisions often require finer-grained views. Common breakdown dimensions include:

• Command execution reliability: Tool-call failure rate and autonomous recovery rate after failure. These directly determine whether an agent can work reliably in unattended scenarios.
• Steerability: After receiving a correction instruction, how many steps the model needs on average to return to the correct track. This reflects how coachable the model is.
• Task efficiency: Average number of steps and token consumption to complete a task, which affects real usage cost.
• Scenario-specific leaderboards: Programming, office tasks, everyday conversation, and other scenarios should be measured separately. A model may perform well in programming but average in long-running office tasks. Separate leaderboards prevent a single overall score from hiding weaknesses.

6. Industry observation: Open-weight models on real tasks

Recent evaluation practice shows a noteworthy trend. As open-weight models continue to improve in parameter scale and post-training methods, their net improvement rates in real agent task evaluation have risen significantly. Some open models now rank first within the open-weight group and enter the leading tier in scenarios such as programming and office tasks, with a shrinking gap compared with closed-source models.

This has practical implications for enterprise users. In scenarios where data cannot leave the organization and private deployment is required, open-weight models can also support fairly complex agent workflows. Accordingly, model selection should rely more on evaluation results based on real-task signals rather than static benchmark scores alone. The correlation between static scores and production performance is becoming weaker.

7. Implementation guide: Building an internal agent evaluation pipeline

Enterprises that need to implement similar capabilities in-house can follow this five-step framework.

Step 1: Standardize conversation logging. Store each interaction as structured event streams rather than plain-text logs:

{
  "session_id": "s-000123",
  "model_variant": "candidate-A",
  "task_category": "coding",
  "events": [
    {"type": "user_message", "ts": 1717200000},
    {"type": "tool_call", "kind": "shell", "exit_code": 1},
    {"type": "correction", "ref_step": 3},
    {"type": "task_confirmation", "verdict": "success"}
  ]
}

Step 2: Build the signal extraction pipeline. The rule channel handles high-confidence signals, while the model discrimination channel handles semantic feedback. The merged results enter a signal store, and raw events are kept for auditing and traceability.

Step 3: Design traffic allocation. Use canary releases and random assignment between new and old models to ensure that assignment is decoupled from user groups and task difficulty. Keep a dual-run period for comparison and validation.

Step 4: Statistics and dashboards. Output net improvement rates and confidence intervals across stratification dimensions. Configure automatic alerts so that when a model shows abnormal fluctuations in objective indicators such as command execution reliability, the release can be frozen in time.

Step 5: Continuous operation. User task distributions drift over time. The evaluation pipeline should periodically re-stratify, update sampling strategies, and organize manual annotation to supplement calibration sets when necessary.

On the infrastructure side, there is no need to build everything from scratch. Mainstream cloud platforms generally provide logging services, object storage, containers, function compute, and monitoring alerts. For example, organizations can use logging services to collect conversation events, object storage to archive raw data, and monitoring alerts to watch for metric anomalies, assembling a complete pipeline quickly while meeting compliance requirements.

8. Five pitfalls to watch

1. Survivorship bias: Conversations abandoned midway often indicate failure. If you only count sessions that finished the full process, success rate will be systematically overestimated.
2. Signal misinterpretation: A correction instruction is not always the model’s fault. It may come from unclear user instructions. Complaints are not always equivalent to task failure and need to be judged in context.
3. Data recycling contamination: If evaluation conversations flow back into training data, this creates a self-evaluation and self-training loop that can amplify errors. Strict isolation is required.
4. Metric gaming: Once the optimization target becomes “improve confirmation rate,” models may start inducing users to confirm. Metric design should retain human spot-checking as a safeguard.
5. Privacy and compliance: Conversation data contains sensitive user information. Desensitization, authorization, minimal collection, and retention limits must be designed into the system from the start.

9. Conclusion

Agent evaluation is moving from human-written tests and human voting toward real conversations, automatic signals, and causal inference. The essence of this shift is returning evaluation authority to real usage: tasks come from real needs, feedback comes from real users, and conclusions come from testable statistical inference. For enterprises, the value of understanding this methodology is not only in reading leaderboards. It also lies in turning the same thinking into an internal evaluation system, so that every model upgrade decision is based on real-task evidence.