Understanding Agent Evaluation in Real-World Tasks: From Conversation Signals to Net Improvement Rate

Why Real-World Task Evaluation Matters

Static benchmarks typically provide well-defined inputs and expected outputs, allowing models to complete reasoning or generation within a limited context. Real-world agent tasks, however, exhibit several distinct characteristics:

• Long-horizon execution: A task may span dozens of conversational turns, involving file reads/writes, command execution, and environment exploration.
• State dependency: Subsequent decisions depend on the results of prior actions, meaning errors can accumulate and amplify.
• Ambiguous objectives: Initial user instructions may be incomplete, requiring the agent to proactively ask clarifying questions or adjust direction mid-execution.
• High steerability requirements: Users may revise requirements midway, and the agent must correctly interpret these changes and alter its planned path.

These factors mean that single-turn question-answering accuracy is insufficient to reflect an agent’s true usability. Evaluation must go deeper into actual user–agent conversations and capture behavioral signals that indicate task success or failure.

Signal Extraction: Mining Key Evidence from Conversations

An evaluation mechanism oriented toward real conversations relies on automatically identifying high-value signals within sessions. These signals are not derived from explicit user votes on two responses but are inferred from natural interactions. Common signal types include:

1. Explicit confirmation: The user acknowledges task completion, such as “That works,” “No problem,” or “Exactly what I needed.”
2. Praise and complaints: Emotionally charged feedback like “This plan is very clear” or “You made the same mistake again.”
3. Corrective instructions: The user asks the agent to change course, for example, “No, you should check the configuration file first” or “Don’t use that command.”
4. Execution errors: In shell command scenarios, non-zero exit codes, syntax errors, and permission denials serve as negative signals.
5. Task restart: The user abandons the current session and initiates a similar task again, often implying previous attempts failed.

These signals rarely exist in isolation; they must be interpreted in context. For instance, an error message could result from the agent’s deliberate environment probing or from its own faulty operation. Therefore, the signal extraction module needs a degree of causal discrimination.

Causal Tracking: Estimating Net Improvement Rate via Randomized Inference

After collecting a substantial volume of session signals, the next step is converting them into comparable model scores. This is where a randomized inference method called “causal tracking” comes into play.

The core idea is to treat all sessions of the same real-world task type as a population, where different models appear at random. Since task assignment and model selection are statistically independent of users’ initial preferences, comparing models on the same signal dimensions allows estimation of each model’s net improvement rate relative to the population average.

Formally, let the observed success rate of a model on dimension $d$ be $p_m$, and the average success rate of all models on that dimension be $\bar{p}$. The net improvement rate can be expressed as:

Net Improvement Rate = (p_m - \bar{p}) / \bar{p} × 100%

In practice, confounding factors such as task difficulty, user activity level, and session length must be adjusted through stratification. By relying on randomized inference, robust capability estimates can be obtained without manual annotation.

This approach offers several advantages:

• Scalability: As the volume of sessions grows, estimation precision naturally improves.
• Resistance to gaming: It is difficult to manipulate rankings by constructing targeted sessions.
• Real-world fidelity: Signals are directly derived from actual user behavior.

Capability Stratification: Observations on Open-Weight Models

In a recent large-scale evaluation, one open-weight model participated in over 17,500 real-world, long-horizon agent task sessions and achieved the top ranking among open-weight models and 13th place across all evaluated models. Its overall net improvement rate was +5.1%, a 4.2-percentage-point gain over its predecessor’s +0.9%.

More notable are the fine-grained metric results. The model ranked at the forefront of open-weight models on the following dimensions:

• Bash Reliability: Measures the agent’s ability to generate and execute shell commands without syntax errors, path mistakes, or dangerous operations. High reliability implies fewer environment disruptions and smoother automation workflows.
• Steerability: Measures the agent’s capacity to correctly interpret corrective instructions and adjust subsequent behavior. This capability directly impacts the efficiency of multi-turn collaboration.

Additionally, the model ranked first among open-weight models across work, coding, and chat task categories, suggesting that its performance gains are not confined to a single scenario but generalize well.

Engineering Practice: Improving Real-World Agent Performance

Based on the evaluation signals discussed above, developers can optimize agent systems in the following areas:

1. Strengthen Robustness of Shell Command Execution

Shell commands are a primary means of agent–OS interaction and a common source of errors. Recommended practices include:

• Perform syntax checks before execution, such as using bash -n for static parsing.
• Add confirmation steps or sandbox isolation for dangerous operations like rm -rf or overwriting files.
• Capture exit codes and standard error, feeding them back into the next reasoning step as signals.

# Example: command execution wrapper with error capture
if ! output=$(some_command 2>&1); then
    echo "Command failed: $output"
    # Feed the error back to the agent for retry or correction
fi

2. Improve Steerability

Steerability requires the agent to recognize the user’s corrective intent and abandon its previous path. Key implementation points include:

• Explicitly mark “user correction” events in the context so the model does not mistake them for supplementary information.
• Maintain a task state machine that resets relevant subgoals when a direction change occurs.
• Confirm understanding in responses, for example, “Understood, I will check the configuration first and then start the service.”

3. Use Real Conversations for Continuous Evaluation

Enterprise users can build an internal evaluation pipeline by following the signal extraction approach described above:

• Collect de-identified real conversation logs.
• Automatically label signals such as confirmations, corrections, and execution errors.
• Stratify net improvement rates by task type and compare across models or versions.
• Feed evaluation results back into model selection and prompt optimization.

Conclusion

Agent evaluation oriented toward real-world tasks is shifting from manual voting to a combination of signal mining and causal inference. By automatically extracting key signals such as user confirmations, corrective instructions, and command execution errors, and applying randomized inference methods, it is possible to estimate a model’s net improvement rate across large-scale real conversations. Observed data indicate that open-weight models have made notable progress in dimensions such as Bash reliability and steerability, achieving an overall net improvement rate of +5.1%. For developers, understanding these evaluation dimensions and optimizing command execution robustness and direction-adjustment capabilities accordingly is a critical path to enhancing an agent’s practical usability.