Evaluating AI Agents: A Causal Tracing Paradigm Based on Real-World Conversations

From Voting-Based to Behavior Signal-Based Evaluation

Evaluating the capabilities of AI agents has long faced a critical challenge: a significant gap exists between benchmark scores and actual user experience. Traditional evaluation methods typically rely on two approaches—scoring single-turn Q&A on static datasets, or using blind A/B testing where users choose between two anonymous responses. The former fails to cover real-world, long-horizon, multi-step tasks involving tool calls. The latter is constrained by high annotation costs, subjective preference noise, and the discrepancy between “which answer looks better” and “which answer actually solves the problem.”

A new evaluation paradigm suggests: instead of asking users to do extra work, treat the real usage process itself as evaluation data. The core assumption is that user behavior during real agent sessions implicitly reflects their judgment of the model’s performance. High-value, automatically recognizable signals include explicit task completion confirmations, praise, complaints, correction instructions, and shell command execution errors. The evaluation system mines these signals and uses statistical inference to estimate each model’s net improvement rate relative to the overall average.

This paradigm offers three main advantages: high ecological validity since samples come from real long-horizon tasks; zero additional annotation burden as signals are naturally generated by user behavior; and statistically significant cross-model comparisons achieved through randomized inference that controls for confounding factors.

Key Signal Design and Extraction

The reliability of this evaluation depends primarily on clear and automatable signal definitions. Common key signal categories include:

• Explicit Confirmation: The user confirms the task meets expectations using phrases like “that’s it,” “looks good,” or “no problem.” This is the strongest positive signal.
• Praise: The user gives positive feedback on the result, such as “great job” or “exactly what I needed.” Slightly weaker than explicit confirmation but indicates a positive direction.
• Complaint: The user expresses dissatisfaction, repeats requests, or points out errors. This is a negative signal.
• Correction Instruction: The user adjusts the model’s direction, format, or scope, saying things like “not what I meant” or “only change the second paragraph.” This reflects the “steerability and correction” dimension. It is not entirely negative but indicates the task wasn’t completed in one go.
• Command Execution Error: In agent scenarios involving tool calls, shell commands returning non-zero exit codes, timeouts, or permission failures are objective, observable failure signals used to measure “execution reliability.”

In practice, signal extraction combines rule-based matching with classification models: using keywords and regular expressions for high-recall initial filtering, followed by a lightweight classifier to reduce false positives. For command execution signals, the system directly parses exit codes and error types from tool call logs. To ensure comparability across sessions, normalization is required for session length, task type, and user proficiency.

# Example: Extracting key signals from a conversation event stream (pseudo-code)
SIGNAL_PATTERNS = {
    "confirm": ["that's it", "looks good", "no problem", "meets expectations"],
    "praise":  ["great job", "well done", "exactly what I needed"],
    "complain":["incorrect", "still wrong", "start over"],
    "correct": ["not what I meant", "only change", "try another way"],
}
def extract_signals(events):
    signals = []
    for e in events:
        if e.type == "user_message":
            for label, kws in SIGNAL_PATTERNS.items():
                if any(k in e.text.lower() for k in kws):
                    signals.append((label, e.turn_id))
        elif e.type == "tool_result" and e.exit_code != 0:
            signals.append(("exec_error", e.turn_id))
    return signals

Causal Tracing: Estimating Net Improvement via Randomized Inference

Signals alone are insufficient—different models face varying task difficulties, user demographics, and session lengths. Directly comparing signal frequencies leads to severe confounding. The Causal Tracing approach introduces randomization: applying random perturbations to session routing or control group assignments so that task difficulty and user characteristics are approximately balanced across models. This isolates “model differences” from “task differences.”

The specific methodology can be summarized as follows:

1. Randomized Assignment: Within the same stream of real tasks, requests are randomly assigned to different models (or alternated within time windows), ensuring no systematic differences in task distribution across groups.
2. Signal Weighting: Confirmations and praise are scored as positive; complaints and execution errors as negative. Correction instructions are weighted separately. This forms a net signal score for each session.
3. Baseline Calibration: The average net signal score of all participating models serves as the baseline. The system calculates each model’s net improvement rate relative to this baseline—essentially, how much the model increases the proportion of successfully completed tasks compared to the average.
4. Interval Estimation: Bootstrapping or stratified sampling is used to estimate confidence intervals, preventing small-sample fluctuations from being misinterpreted as true differences.

The advantage of this design is that it doesn’t rely on manual preference judgments from A/B testing, but directly measures “whether the task was genuinely resolved.” Simultaneously, randomization ensures causal validity rather than mere correlation.

Granular Dimensions: Reliability, Steerability, and Scenarios

A single aggregate score is insufficient to guide optimization. Evaluations typically break out multiple granular dimensions, two of which are particularly critical for agents:

• Command Execution Reliability (Bash Reliability): Measures whether an agent’s command-line tool calls execute successfully on the first try, and whether errors are correctly identified and retried. This dimension directly determines if automated tasks can achieve a closed loop. It can be characterized by metrics like “non-zero exit code ratio” and “success rate after retry.”
• Steerability and Correction Capability: Measures whether the model accurately understands the scope of modifications upon receiving a correction instruction, changing only what needs to be changed without rewriting the whole output or deviating from the goal. Metrics include “first-pass success rate after correction” and “average correction turns.”

Furthermore, ranking by task scenario is crucial. Work tasks (document processing, workflow orchestration), programming tasks (code generation, debugging, script execution), and chat tasks (information synthesis, multi-turn clarification) demand significantly different model capabilities. Scenario-specific rankings prevent aggregate scores from masking specific weaknesses.

Open-Weight Model Performance and Interpretation

In an evaluation based on over 17,500 real-world, long-horizon agent sessions from global users, an open-weight model ranked first among open-weight models and 13th overall. It also topped the open-weight category across work, programming, and chat scenarios. Its overall net improvement rate was +5.1%, a 4.2 percentage point increase from the +0.9% of its previous generation. It also ranked at the top of open models in granular metrics like command execution reliability and steerability.

When interpreting this data, several points should be noted:

• Positive and significant net improvement: The new version shows a notable increase in the proportion of tasks confirmed as complete by users in real-world scenarios relative to the average. The generational leap of 4.2 percentage points is significant in long-horizon task evaluation.
• Divergence in rankings: The gap between the overall ranking and the open-weight ranking indicates a persisting overall gap between closed-source and open-weight models, though open models are becoming competitive in specific capabilities.
• First place across all scenarios: This demonstrates the model’s capabilities are well-balanced, rather than being inflated by a single scenario.

Implementation Best Practices

Teams looking to replicate or adapt this evaluation paradigm should consider the following:

1. Define a signal dictionary first: Align trigger words and rules for confirmations, praise, complaints, and corrections with your product’s language habits. Maintain a manual sampling step to control the false positive rate.
2. Ensure randomization quality: Assignment strategies must be truly random and auditable. Log random seeds and task characteristics for each assignment to facilitate post-hoc stratified analysis.
3. Distinguish correlation from causation: Randomization only addresses observable confounders. Remain vigilant regarding biases introduced by mediators like session length and user retention, and apply stratified regression when necessary.
4. Prioritize granular dimensions over aggregate scores: Treat command execution reliability and steerability as independent dashboards to guide model iteration.
5. Continuously calibrate the baseline: As the pool of evaluated models changes, the average will drift. The net improvement rate must be recalculated using the same baseline window to ensure longitudinal comparability.

Conclusion

Agent evaluation based on real-world conversation signals and causal tracing focuses on a core metric: “whether the user actually delegated the task to the model and got the desired result.” This circumvents the subjective noise of voting-based evaluations and the low ecological validity of static benchmarks. Combined with granular dimensions like command execution reliability, steerability, and scenario-specific rankings, this paradigm provides both a comparable aggregate score and actionable insights into weaknesses, creating a feedback loop closely aligned with real-world usage for continuous agent capability iteration.