Evaluating AI Agents in Real-World Conversations: Mechanisms and Engineering Practices

1. Shift in Evaluation Paradigm: From Subjective Voting to Behavioral Signals

Evaluating AI agent capabilities has long faced a core challenge: how to measure actual performance in real-world tasks without introducing subjective bias. Traditional pairwise comparison relies on users voting between two responses. While simple to implement, this method has significant flaws—voters often focus on surface-level fluency rather than whether the task was genuinely completed. Additionally, collecting voting data is costly and yields low coverage, making it inadequate for large-scale, long-term evaluation needs.

A more practical engineering approach is to extract behavioral signals directly from real user-agent conversations. During actual usage, users generate a large amount of “unconscious annotations”—confirming task completion, expressing satisfaction or dissatisfaction, or issuing correction commands. Furthermore, agents executing Shell commands naturally expose success or failure results. These signals inherently possess authenticity and high coverage without requiring additional manual annotation. The core value of an evaluation leaderboard built on this approach lies in assessing not whether an answer “looks good,” but whether the task was genuinely advanced.

2. Key Signal Extraction: Turning Conversation Logs into Evaluation Metrics

Transforming raw conversations into quantifiable metrics requires signal extraction. In engineering practice, several high-value signal categories are typically monitored:

• Explicit Confirmation: Positive user confirmation of task results, such as verifying that a file was generated, a script ran successfully, or data was aligned. This is the most direct evidence of task completion.
• Praise and Complaints: Feedback with emotional tendencies can serve as an auxiliary dimension for quality assessment, though noise reduction is necessary to avoid misinterpreting pleasantries as praise.
• Correction Commands: Instructions from users asking the agent to adjust direction or fix errors, reflecting the model’s ability in intent understanding and execution correction.
• Command Execution Errors: Non-zero exit codes or exception outputs returned after the agent invokes a Shell command. This is a hard metric for execution reliability.

The key to signal extraction lies in structuring and denoising. Conversation logs are often a mix of unstructured natural language and tool invocation records. It requires session segmentation, role alignment, and intent classification, followed by assigning weights to each signal type. In engineering practice, it is recommended to design the signal extraction pipeline as a pluggable module:

# Pseudo-code: Conversation Signal Extraction Pipeline
def extract_signals(session):
    turns = segment_turns(session)          # Session segmentation and role alignment
    signals = []
    for turn in turns:
        signals += detect_confirmation(turn)  # Explicit confirmation
        signals += detect_sentiment(turn)     # Praise/Complaints
        signals += detect_correction(turn)    # Correction commands
        signals += detect_shell_error(turn)   # Command execution errors
    return normalize(signals)                 # Normalization and weighting

It must be emphasized that signal extraction must guarantee reproducibility: the same conversation should yield a consistent set of signals across different times and implementations, otherwise, subsequent statistical inferences will become meaningless.

3. Causal Tracing: Net Improvement Rate via Randomized Inference

Signals alone are not enough; evaluation needs to answer a causal question: How much net improvement does a specific model bring compared to the average level? Directly comparing raw signal frequencies introduces confounding factors—different models have varying conversation distributions, task difficulties, and user demographics. To address this, employing randomized inference through “causal tracing” is a more rigorous approach.

The basic logic is to perform randomized allocation within real traffic, ensuring that different models are as comparable as possible across dimensions like task type and user group. Subsequently, the relative performance of each model is calculated based on the extracted signals, and statistical inference is used to estimate its Net Improvement Rate relative to the average. Key aspects of this method include:

1. Selection of Randomization Unit: Random allocation should be done on a per-session or per-task basis to avoid interference from multiple rounds of conversation by the same user.
2. Covariate Balance: Balance tests should be conducted on covariates like task difficulty and session length to ensure a fair comparison.
3. Confidence Interval Estimation: The Net Improvement Rate must be accompanied by a confidence interval to prevent misinterpreting statistical noise as capability differences.

The Net Improvement Rate represents a “percentage point improvement relative to the average level of participating models.” It is a relative measure, not an absolute score, so interpretation should consider the overall distribution of evaluated models. A Net Improvement Rate of around +5% typically means the model consistently outperforms the average on most tasks, rather than relying on a few extreme outliers to inflate the mean.

4. Open-Weight Model Performance in Long-Horizon Tasks

Long-horizon agent tasks (e.g., multi-step code modifications, cross-file refactoring, continuous debugging) demand significantly more from models than single-turn Q&A. They require the model to maintain goal consistency across dozens of interaction rounds, correctly invoke tools, and self-correct upon errors. Evaluation data indicates that open-weight models are rapidly approaching closed-source models in these scenarios.

Take a recent large-scale evaluation as an example: a certain open-weight model was evaluated based on over 17,500 real long-horizon agent task conversations from global users. It ranked first among open-weight models and 13th among all participating models, securing the top spot for open models across Work, Coding, and Chat scenarios. Its overall Net Improvement Rate reached +5.1%, an increase of about 4.2 percentage points compared to its previous generation. More notably, the granular metrics reveal:

• Bash Reliability: Measures the success rate and error recovery capability of the agent invoking Shell commands. Open models ranking highly on this metric indicates a significant improvement in engineering maturity at the tool invocation level.
• Steerability: Measures whether the model can effectively adjust its execution direction upon receiving correction commands. This capability directly determines whether a long-horizon task can converge.

These results send a clear signal: the usability of open-weight models in real-world agent tasks has entered a new phase. Their advantages are reflected not just in benchmark scores, but also in “engineering perception” metrics like execution reliability and error correction.

5. Engineering Implementation Recommendations

If a team wishes to build a similar evaluation system, consider the following practical guidelines:

1. Establish signal specifications before building the pipeline. Clearly define each signal type’s definition, rules, and weights, forming a reviewable specification document to prevent repeated adjustments from making historical data incomparable.
2. Ensure the rigor of randomization. Random allocation units, covariate balance, and confidence intervals are all indispensable; otherwise, the Net Improvement Rate can easily become distorted.
3. Distinguish between “capability metrics” and “experience metrics”. Command execution error rates are hard capability metrics, while praise/complaints are experience metrics. The two should not be mixed into the same scoring formula.
4. Address sampling bias in long-horizon conversations. Long-task sessions represent a small proportion but contain a high volume of information. They require appropriate treatment during sampling and weighting.
5. Evaluation results should be drillable. Beyond overall rankings, the system should support drill-down analysis by task type, session length, and tool invocation count to help identify model shortcomings.

Conclusion

Real-conversation-driven agent evaluation is essentially an engineering approach to transforming “actual user behavior” into “comparable capability metrics.” It does not pursue the superficial consensus of voting; instead, through signal extraction and randomized causal inference, it answers the fundamental question: “Did the model actually make the task smoother to complete?” As open-weight models continue to improve their reliability in long-horizon tasks, this type of evaluation system will become a crucial infrastructure for model selection and engineering optimization.