Evaluating Open-Weight AI Agents: From Comprehensive Rankings to Granular Capabilities

1. Background: Why Agent Evaluation Needs “Real-World Long-Cycle Tasks”

As Agent technology transitions from lab demonstrations to production environments, evaluation methodologies are undergoing a fundamental shift. Early evaluations relied heavily on single-turn Q&A, short instruction following, or standardized test banks. While easy to reproduce, these methods fail to reflect agent performance in real-world scenarios. Real-world tasks typically exhibit the following characteristics:

• Long-cycle: A task may involve dozens or even hundreds of steps, including planning, execution, observation, and correction.
• Multi-tool collaboration: Agents need to invoke various capabilities such as command lines, file systems, and network requests.
• State dependency: Subsequent steps depend on the results of previous ones, meaning errors can accumulate and amplify.
• Open-endedness: User instructions may be ambiguous, requiring the agent to proactively clarify or make reasonable assumptions.

Against this backdrop, evaluation systems based on real user sessions have become the core reference for measuring agent capabilities. In a recent evaluation of open-weight models, a specific open-weight model achieved noteworthy results across multiple dimensions, including comprehensive ranking, programming tasks, and interactive capabilities. This ranking was based on over 17,500 real-world, long-cycle agent task sessions from global users. This data scale indicates that evaluation is no longer a closed lab test, but rather a sampling of real usage distribution.

2. Key Dimensions of the Evaluation System

To understand the meaning behind the ranking, we must first break down the dimensions focused on by the evaluation system. Based on public information, the evaluation covers at least the following capabilities:

1. Comprehensive Net Improvement Rate

The Net Improvement Rate is a comprehensive metric measuring whether an agent “makes things better” during a task. Its core logic can be abstracted as:

# Pseudocode: Calculating Net Improvement Rate
net_improvement_rate = (improved_tasks - degraded_tasks) / total_tasks

The value of this metric lies in the fact that it doesn’t just look at “how many tasks the agent completed,” but also whether it introduced new problems during the process. An agent that only pursues completion rate while frequently damaging the existing state might have a very low or even negative net improvement rate. In this evaluation, the open-weight model achieved a comprehensive net improvement rate of +5.1%, an increase of 4.2 percentage points compared to the previous generation’s +0.9%. This improvement is practically significant in long-cycle tasks—it means the agent genuinely helped users advance their work in more tasks, rather than creating additional burdens.

2. Programming and Command Execution Reliability

Programming tasks are among the most challenging scenarios in agent evaluation. They require the model not only to understand code semantics but also to execute commands in real environments, handle errors, and adjust strategies. The Bash Reliability metric involved in the evaluation measures the agent’s stability in a command-line environment:

• Whether it can construct commands correctly;
• Whether it can handle non-zero exit codes;
• Whether it can reasonably retry or switch to alternative approaches after a command failure;
• Whether it avoids dangerous operations (e.g., accidentally deleting files, overwriting critical configurations).

This model ranks at the top among open-weight models in granular metrics like Bash Reliability, indicating good robustness in the stages where engineering practice is most prone to failure. For developers, this is more valuable than simple code generation accuracy.

3. Steerability and Error Correction

Steerability measures whether an agent can quickly adjust its behavior after a user changes their intent or points out an error. In long-cycle tasks, users modifying requirements midway is the norm. An agent lacking correction capabilities will continue executing along the wrong path, leading to wasted resources or even state corruption.

This model’s performance in this metric also ranks at the top of open-weight models, meaning it can better accept feedback and revise plans during interactions. This is particularly important for development scenarios requiring human-machine collaboration—developers often want to intervene at critical nodes rather than letting go entirely.

4. Scenario-Based Rankings: Work, Programming, and Chat

In addition to comprehensive metrics, the evaluation also subdivided by scenarios. This model ranked first among open-weight models in three scenarios: Work, Programming, and Chat. This result reflects two characteristics:

• Balanced capabilities: No obvious shortcomings, maintaining stable performance across different task types;
• Good generalization: Does not rely on task-specific optimization for a certain category, but possesses more general agent capabilities.

For enterprise users, balanced capabilities are often more attractive than a single standout skill, because task types in production environments are diverse and dynamically changing.

3. From Rankings to Practice: What Developers Should Focus On

Evaluation rankings are references, not absolute answers. When selecting or optimizing an agent solution, developers are advised to establish their own evaluation frameworks from the following perspectives:

1. Build a Business-Aligned Task Set

The session distribution of general evaluations may not align with your business. It is recommended to extract real tasks from your own business to build a small, focused evaluation set. For example:

# Pseudocode: Building a business-aligned evaluation task set
tasks = [
    {"id": "task_001", "type": "Code Fixing", "steps": 25, "tools": ["shell", "editor"]},
    {"id": "task_002", "type": "Data Cleaning", "steps": 40, "tools": ["shell", "python"]},
    {"id": "task_003", "type": "Configuration Troubleshooting", "steps": 15, "tools": ["shell", "log"]},
]
# For each task, record: whether completed, whether new issues were introduced, number of manual interventions

2. Focus on “Failure Recovery” Rather Than Just “Success Rate”

In long-cycle tasks, failure is the norm. What truly impacts the experience is the agent’s ability to recover after a failure. Key observations include:

• Whether it blindly retries after a command failure;
• Whether it can read error messages and locate the cause;
• Whether it proactively asks when uncertain, rather than forcing execution.

3. Incorporate Steerability into Selection Criteria

If an agent is highly capable but difficult to steer, it may introduce risks in real collaboration. Steerability is reflected in:

• Whether it respects user interruptions and modifications;
• Whether it can explain its decision-making basis;
• Whether it can avoid repeating similar mistakes after being corrected.

4. Prioritize the Safety Boundaries of Command Execution

Shell command execution reliability is not just about success rate; it is also about safety. It is recommended to add the following constraints during integration:

• Intercept or require secondary confirmation for high-risk commands;
• Execute in a sandbox or container environment;
• Maintain complete operation logs for traceability.

4. Trend Analysis: The Next Stop for Open-Weight Agents

Several trends can be observed from the evaluation results:

1. Evaluation focus shifts from “knowledge” to “action”: Metrics like net improvement rate and command execution reliability measure the agent’s ability to act in an environment and produce actual impact.
2. Engineering capabilities of open-weight models are rapidly approaching: In dimensions with strong engineering attributes like programming and command execution, open-weight models already possess usable stability.
3. Steerability and collaboration become key differentiators: As foundational capabilities converge, the ability to collaborate efficiently with humans, accept feedback, and adjust will become the core competitiveness in the next phase.
4. Real session data becomes evaluation infrastructure: The scale of over 17,500 real long-cycle sessions indicates that evaluation is evolving from artificial construction to real distribution.

5. Conclusion

Agent evaluation is shifting from “how many questions can be answered correctly” to “how much work can actually be accomplished.” Metrics like comprehensive net improvement rate, command execution reliability, and steerability together form an evaluation perspective closer to engineering practice. For developers and enterprise users, understanding the meaning behind these metrics is more valuable than remembering specific rankings. When selecting and optimizing, it is recommended to base decisions on business task sets, focusing on failure recovery, steerability, and safety boundaries, thereby making the agent a truly trustworthy productivity tool.