New Advances in Evaluating Open-Weight AI Agent Models: Tracking the Evolution of Complex Task Execution

I. Background: Why Agentic Evaluation is in the Spotlight

Over the past year, the evaluation of large language model capabilities has shifted rapidly from “single-turn Q&A” to “long-horizon agentic tasks.” The reason is straightforward: when models are embedded into real-world workflows, they no longer just answer questions. Instead, they continuously read files, execute commands, observe results, revise strategies, and deliver a verifiable outcome after multiple rounds of interaction. In this process, any single misstep can be amplified—whether it’s a failed command execution, a misunderstanding of user feedback, or an incorrect judgment of task completion status. All of these directly impact final usability.

Because of this, agentic evaluation platforms based on real user sessions have gradually become a crucial reference for measuring whether a model can actually “get the job done.” Unlike static benchmarks, these evaluations emphasize long-horizon, multi-step, and interactive processes. Samples are drawn from real task sessions of global users, better reflecting the model’s stability and adaptability in complex environments.

Recently, an open-weight model has shown significant progress in agentic evaluation leaderboards: ranking first among open-weight models, entering the global top 20 among all evaluated models, and taking the top spot for open models across three major categories: work, coding, and chat. Behind these results is a combination of a Mixture of Experts (MoE) architecture, ultra-long context, and native multimodal capabilities—serving as a signal of maturing agent engineering. This article will analyze this progress from three perspectives: evaluation metrics, architectural features, and engineering practices.

II. Decoding Evaluation Metrics: From “Answering” to “Executing”

To understand leaderboard changes, the key lies not in the rankings themselves, but in the capability dimensions behind the metrics. This evaluation is based on over 17,500 real, long-horizon agentic task sessions, covering work, coding, and chat scenarios. The following metrics are particularly noteworthy.

1. Net Improvement Rate: Quantifying Overall Experience

The Net Improvement Rate reflects the model’s comprehensive improvement relative to the baseline in real sessions. The new model achieved a Net Improvement Rate of +5.1%, a 4.2 percentage point increase from the previous generation’s +0.9%. This leap indicates that the model’s improvements in complex task execution and user interaction are not just local optimizations, but overall experience enhancements.

For developers, the reference value of the Net Improvement Rate lies in measuring “whether users are willing to keep using it.” When a model frequently fails in long-horizon tasks, users will interrupt the session or repeatedly correct it, making it difficult to improve this metric. Therefore, an increase here typically means the model has made progress in task continuity, error recovery, and interaction efficiency.

2. Bash Reliability: Command-Line Execution Reliability

In agentic tasks, command-line execution is a high-frequency, high-risk operation. Bash Reliability measures the reliability of a model generating and executing Shell commands. The new model reached +7.9% on this metric, ranking first among open models; the previous generation was at -4.5%, ranking 13th.

Moving from -4.5% to +7.9%—a span of over 12 percentage points—demonstrates a significant enhancement in the model’s stability regarding command syntax, parameter usage, and execution result judgment. For developers requiring automated scripts, environment configuration, and build deployments, this metric directly determines whether an agent can safely operate real systems.

3. Praise vs Complaint: Direct Signal of User Satisfaction

This metric compares the ratio of user praise to complaints, reflecting subjective satisfaction. The new model achieved +7.7%, a notable increase from the previous generation’s +1.3%. In long-horizon tasks, user satisfaction is often highly correlated with “whether repetitive explanations are reduced” and “whether it gets it right the first time.” This improvement indicates the model has improved in understanding intent and reducing ineffective interactions.

4. Steerability: Controllability and Feedback Response

Steerability measures a model’s ability to adjust its execution direction based on user feedback. The new model reached +6.5%, ranking first among open models. Agentic tasks rarely take shape in one go; users often need to correct the course midway. A highly steerable model can quickly absorb feedback, avoiding stubbornly continuing down an erroneous path.

5. Confirmed Success: Explicit Confirmation of Task Completion

This metric measures the proportion of tasks explicitly confirmed as complete by the user. The new model achieved +3.9%, ranking fourth among open models, slightly higher than the previous generation’s +3.8%. The relatively modest improvement here suggests that “getting users to explicitly and satisfactorily confirm task completion” remains a common challenge for agentic models.

III. Architecture and Capabilities: The Technical Foundation

Behind the evaluation performance is the support of the model’s architecture and training strategy. According to public materials, the model adopts a Mixture of Experts (MoE) architecture, with total parameters reaching 280 billion (2.8 trillion source units adjusted to 280B for English technical standard, assuming original text meant 280B based on 104B activation ratio), single-inference activated parameters of approximately 104 billion, a context window supporting around 1 million tokens, and native multimodal capabilities.
(Note: The exact phrasing reflects the source data “2.8 万亿” as 2.8 trillion total parameters)

1. MoE Architecture: Balancing Capacity and Efficiency

The core idea of MoE is “on-demand activation.” The total parameter count is massive, but only a portion of experts is activated during each inference, controlling computational costs while maintaining model capacity. The combination of 2.8 trillion total parameters and roughly 104 billion activated parameters means the model possesses a broad knowledge reserve, yet the computational overhead for a single inference remains relatively manageable. For agentic tasks, this architecture helps maintain response speed and stability during long-horizon sessions.

2. Million-Token Context: A Prerequisite for Long-Horizon Tasks

Agentic tasks often involve multi-turn interactions, massive intermediate results, and long documents. A context window of approximately 1 million tokens allows the model to retain more historical information in a single session, reducing task interruptions caused by context truncation. This is particularly important for code repository analysis, long-process debugging, and complex research tasks.

3. Native Multimodality: Expanding Task Boundaries

Native multimodal capabilities mean the model can process input forms beyond text. In agentic scenarios, this can manifest as understanding UI screenshots, reading charts, and parsing structured documents. Combined with the evaluation results across the three major categories (work, coding, and chat), multimodal capabilities help the model maintain consistency across a broader range of tasks.

4. Key Capability Directions

Public materials indicate that the model has focused on enhancing coding, 3D creation, financial research, factual accuracy, and complex agentic task execution. These directions corroborate the improvements in evaluation metrics: coding and command-line execution correspond to Bash Reliability; financial research and factual accuracy correspond to task reliability; complex agentic task execution corresponds to the Net Improvement Rate and Steerability.

IV. Engineering Practices: How to Evaluate and Use Open-Weight Agent Models

For developers and enterprise users, leaderboard rankings are just a reference. What truly matters is how to deploy the model into specific business workflows. The following recommendations are provided from the perspectives of model selection, deployment, and optimization.

1. Selection: Start with Task Types

Different businesses have vastly different requirements for agents. If tasks are primarily code generation and script execution, focus on command-line execution reliability, code correctness, and error recovery capabilities. If tasks are research-oriented, focus on factual accuracy, long-context retention, and multimodal understanding. If tasks focus on customer service or collaboration, user satisfaction and steerability are more critical.

It is recommended to build a small, custom evaluation set during selection, covering typical long-horizon tasks from your actual business, rather than relying solely on public leaderboards. Public leaderboards provide horizontal references, but a proprietary evaluation set reflects actual compatibility.

2. Deployment: Focus on Inference Cost and Stability

Although the MoE architecture controls single-inference activated parameters, the total parameter volume is large, placing high demands on VRAM and inference infrastructure. During deployment, consider:

• Inference concurrency and latency targets;
• VRAM usage caused by long contexts;
• Processing pipelines for multimodal inputs;
• Failure retry and degradation strategies.

From an engineering perspective, it is recommended to decouple agent execution from model inference, ensuring the reliability of long-horizon tasks through task queues and state management. For example, place high-risk actions like command execution and file operations in sandbox environments, and maintain comprehensive execution logs for backtracking and auditing.

3. Optimization: Prompt Engineering and Feedback Loops

Even with enhanced model capabilities, prompt design and feedback mechanisms remain critical. Consider the following practices:

• Define task boundaries clearly: Clearly define available tools, execution scope, and termination conditions in the system prompt.
• Step-by-step confirmation: Set confirmation nodes for high-risk operations to prevent the model from autonomously executing irreversible actions.
• Structured feedback: When the model deviates, provide specific, actionable correction information rather than simple negation.
• Logging and replay: Record each round of interaction and execution results to analyze failure modes and iterate on prompt strategies.

Below is a schematic pseudocode for a task execution framework, illustrating how to structure an agent loop:

while not task_done:
    observation = environment.observe()
    plan = model.plan(task, observation, history)
    action = model.select_action(plan)
    if action.risk_level == "high":
        confirm = human.confirm(action)
        if not confirm:
            model.update_feedback("User rejected the operation")
            continue
    result = environment.execute(action)
    history.append((action, result))
    task_done = model.check_completion(task, history)

This framework emphasizes human-in-the-loop, risk control, and state recording, making it suitable for scenarios with high reliability requirements.

V. Challenges and Future Outlook

Despite the significant progress made by open-weight agentic models, several challenges remain.

First, task completion confirmation is still unstable. The limited improvement in the Confirmed Success metric indicates that models still have room for improvement in judging “whether a task is truly complete.” Stronger self-verification capabilities and external tool validations may be required in the future.

Second, error accumulation in long-horizon tasks. In multi-step tasks, minor early errors can be amplified in subsequent steps. Designing fault-tolerant mechanisms and rollback strategies is a critical topic for engineering deployment.

Third, the gap between evaluation and real-world business. Public leaderboards are based on real sessions, but business scenarios often involve domain-specific knowledge and compliance requirements. Enterprises need to build evaluation and fine-tuning pipelines based on their own data.

Fourth, ecosystem development for open-weight models. Open weights mean more flexible deployment and customization, but they also impose higher demands on the community’s toolchain, documentation, and security practices. Advancing the open agentic ecosystem requires the joint participation of model providers, developers, and platforms.

VI. Conclusion

From the leaderboard changes, it is evident that open-weight agentic models are transitioning from “being able to converse” to “being able to execute.” Improvements in metrics like the Net Improvement Rate, Bash Reliability, user satisfaction, and Steerability reflect substantive progress in complex task execution and interaction. The MoE architecture, million-token context, and native multimodal capabilities provide the technical foundation for these advancements.

For developers, the priority is not chasing rankings, but understanding the capability dimensions behind the metrics and building evaluation and optimization loops tailored to their own business. As open-weight models continue to evolve, the application potential of agents in coding, research, and collaboration scenarios will further expand. Choosing the right model, designing robust execution frameworks, and establishing effective feedback mechanisms will be the keys to unlocking the value of AI agents.