Open-Weight AI Agent Tops Leaderboard: Long-Horizon Tasks and Managed Agent Services Explained

1. The Evolution of AI Agent Evaluation

Evaluating large language models (LLMs) is shifting from basic text generation quality toward a comprehensive assessment of agentic execution capabilities. Traditional benchmarks typically focus on accuracy on static datasets, whereas agent scenarios emphasize the ability to continuously execute tasks, call tools, self-correct, and interact with users in open environments. Consequently, evaluation methods based on real user sessions are gaining traction.

Recent agent evaluation data reveals that an open-weight model has achieved significant progress in overall rankings. Across more than 17,000 real-world, long-horizon agent task sessions, the model demonstrated a net improvement rate of +5.1%, compared to just +0.9% for the previous generation. This signifies that the model not only completes tasks but also progressively optimizes results during execution, reducing invalid operations and error accumulation.

From an evaluation perspective, agent capabilities can be broken down into several quantifiable metrics. Command execution reliability reflects the model’s stability when generating Shell commands or tool calls. User satisfaction measures subjective user feedback post-task. Controllability indicates the model’s ability to adjust its execution trajectory based on user input. Confirmed success tracks whether the task was completed in a verifiable manner. Together, these metrics define an agent’s “usability” rather than merely evaluating text quality.

For developers, understanding these metrics is crucial for objectively assessing whether a model is ready for production. A model excelling at text generation may not maintain stability during long-running, multi-step operations and tool calls. Therefore, focusing on engineering metrics like command execution reliability, error recovery, and resource consumption is a vital step in model selection.

2. Key Improvements in Command Reliability and Controllability

In agentic tasks, Shell command execution is a common yet high-risk operation. The model must generate commands based on objectives, predict execution outcomes, and adjust strategies upon failure. Command execution reliability directly impacts task success and determines whether erroneous commands might cause environment corruption or data loss.

In recent evaluations, the open-weight model achieved a +7.9% score in command execution reliability, ranking first among open-weight models, a significant jump from the previous generation’s -4.5%. This shift from negative to positive indicates that the model no longer frequently makes critical errors in command generation and execution judgment. This improvement likely stems from reinforced training on command syntax, execution result feedback, and error recovery pathways.

Controllability is another critical metric. When executing long-horizon tasks, agents often need to adjust their plans based on mid-task user feedback. If a model is insensitive to instructions like “stop current operation,” “change the data source,” or “use a simpler method,” it can lead to task deviation or resource waste. This model achieved a +6.5% controllability score, demonstrating its effectiveness in translating user feedback into altered execution strategies.

From a technical implementation standpoint, improving controllability requires introducing more interaction trajectories with intermediate feedback into the training data, strengthening the model’s semantic understanding of such feedback. For instance, if a user says, “Forget the formatting, just get the process running first,” the model should recognize this as a priority adjustment rather than task abandonment. Such capabilities cannot be fully covered by static test sets; they require substantial real interaction data.

3. From Open Weights to Managed Agent Services

Releasing open model weights provides researchers and enterprises with the flexibility for autonomous deployment and secondary training. However, transforming a model into a stable agent service still presents significant engineering challenges. Running an agent requires a scheduling framework, tool-calling interfaces, sandboxed environments, logging, and exception recovery mechanisms. Building this infrastructure from scratch is time-consuming and requires continuous maintenance.

The newly introduced managed agent service provides the framework supporting agent operations as a cloud capability. Enterprises and developers can access an around-the-clock managed service without building or maintaining the runtime environment themselves. This service supports long-horizon task execution, meaning agents can run for extended periods, making them suitable for scenarios requiring multi-step processing or waiting for external events.

Multi-agent collaboration is another key capability of the managed service. In complex business workflows, a single agent often cannot cover all phases. For example, an investment research assistant might need to simultaneously perform data collection, report generation, and risk assessment. These tasks can be handled by different agents and consolidated through a collaboration mechanism. The managed service provides the runtime foundation for such architectures, significantly reducing the development costs of coordinating multiple agents.

Enterprises can configure agents on demand for internal business monitoring, IT operations response, or embed them into customer-facing systems as investment research or customer service assistants. This flexibility allows agents to transition from experimental tools to production environments. However, it is important to note that managed services do not replace business logic design; enterprises must still clearly define the agents’ responsibility boundaries, tool permissions, and exception handling strategies.

4. Security Mechanisms and Execution Auditing

When agents are granted the ability to execute commands, access networks, and call tools, security boundaries become a prerequisite for deployment. The managed service provides independent sandboxes, isolating the agent’s execution environment from core business systems. Even if an agent generates a dangerous command or makes an error, it will not directly impact core data and infrastructure.

Network access control allows enterprises to restrict the target ranges an agent can access. For instance, an internal IT operations agent might only be permitted to access specific intranet addresses, with public internet access disabled. Conversely, a user-facing investment research assistant requires controlled internet retrieval capabilities. This granular control helps mitigate risks without sacrificing functionality.

Execution tracing and audit export are indispensable capabilities in production environments. All tool calls, command executions, and intermediate results must be traceable for rapid root cause analysis during anomalies. Enterprise administrators can view the resource consumption and costs of various agent projects, providing data support for cost management and performance optimization.

Regarding security components, the managed service can integrate with independent security modules to inspect task inputs and tool return payloads before model inference. This mechanism allows enterprises to intercept privacy data and sensitive information based on custom rules, such as preventing the agent from sending ID numbers, API keys, or internal document contents to external interfaces. These checks occur pre- and post-inference, helping to control data leakage risks at the source.

5. Tool Ecosystem and Custom System Integration

The practical value of an agent largely depends on the breadth of tools it can invoke. Currently, the managed service offers over 140 skills and more than 30 plugins, covering common task types like web search, professional data query, office document processing, and content generation. These capabilities allow agents to quickly integrate into existing workflows without developing tools from scratch.

For enterprises with proprietary data sources or internal tools, the managed service supports custom interface integration. This means agents can invoke internal enterprise APIs, query dedicated databases, or trigger specific business workflows within a unified framework. For example, an IT operations agent can call an internal monitoring alert API to fetch real-time metrics and execute corresponding actions based on predefined rules.

Regarding integration approaches, enterprises typically need to focus on interface authentication, data format conversion, and error handling. The managed service provides the runtime framework and tool scheduling capabilities, while specific business logic remains the developer’s responsibility. It is recommended to clearly define the input and output contracts for tool calls before integration, and to define timeout, retry, and degradation strategies for each tool, preventing agents from getting stuck in infinite waits due to external interface anomalies.

6. The Evolution of an Open Agent Ecosystem

The combination of open-weight models and managed services reflects the evolution of agent technology toward a more engineering-focused and deployable direction. Open model weights lower the research barrier, enabling more teams to fine-tune and validate based on existing results. Meanwhile, managed services lower the deployment barrier, allowing small and medium-sized teams without infrastructure maintenance capabilities to leverage agentic capabilities.

Looking at evaluation trends, future competition among agent models will increasingly focus on long-horizon task stability, multi-tool collaboration efficiency, and security controllability. Pure text generation quality is no longer sufficient to support complex business scenarios. Developers need to focus on model performance in real-world interactions rather than relying solely on static benchmark scores.

For enterprises, adopting agent technology should begin with clearly defined task boundaries, prioritizing scenarios with relatively clear rules and verifiable outcomes, such as report generation, data inspection, and standardized customer service Q&A. During operation, execution logs and user feedback should be continuously collected to evaluate model performance and optimize prompts and tool configurations. Only by establishing a positive feedback loop between engineering practice and model capabilities can agents truly integrate into production systems and generate sustained value.