Breakthroughs in Agentic Capabilities and Deployment Challenges of Open-Weight Ultra-Large MoE Models

In recent years, the development of large language models (LLMs) has been evolving from simple text generation toward the agentic paradigm, which focuses on executing complex tasks. During this transition, the Mixture-of-Experts (MoE) architecture has become a critical technical direction, leveraging its ability to support massive parameter scales at lower inference costs. Recently, a research team officially open-sourced the weights of a 2.8-trillion-parameter MoE model. This model performed exceptionally well on an agentic leaderboard based on real user sessions, ranking first among open-weight models. This article provides an in-depth analysis of this release from four dimensions: technical architecture, evaluation mechanisms, core capability metrics, and engineering deployment challenges.

1. MoE Architecture Evolution and Long-Context Modeling Optimization

The newly released model employs a typical MoE architecture with a total parameter count of 2.8 trillion. However, it only activates approximately 104 billion parameters during a single inference pass. This sparse activation mechanism allows the model to possess vast knowledge capacity and complex reasoning potential while keeping the computational complexity of a single inference comparable to that of a 100-billion-scale dense model.

For long-context processing, the model supports a context window of approximately 1 million tokens. However, in ultra-long context scenarios, traditional full-attention mechanisms introduce quadratically growing computational and memory overhead. To overcome this bottleneck, the model implements a key optimization in its base architecture: replacing some full-attention layers with gated sparse attention layers. Through a dynamic routing mechanism, gated sparse attention ensures that each token only attends to a relevant subset of tokens when calculating attention weights, significantly reducing the overhead of long-context modeling. This improvement not only enhances the inference efficiency for million-token inputs but also provides the underlying architectural support required for agents to handle tasks such as ultra-long codebase analysis and multi-turn complex tool calls.

2. Agentic Evaluation Innovation: From Blind Tests to Causal Tracing

Traditional pairwise blind testing has shown limitations when measuring the agentic capabilities of large models. Relying on users to subjectively vote between two model responses fails to accurately reflect a model’s actual performance in executing multi-step tasks and invoking external tools. The recent evaluation introduced a signal extraction mechanism based on real user agent sessions and employed a “causal tracing” randomized inference method to evaluate the net improvement rate of the models.

Instead of requiring simple user votes, this evaluation mechanism extracts multi-dimensional behavioral signals from real agent sessions, primarily including:

1. Explicit confirmation signals of task completion;
2. User praise and complaint text;
3. User correction commands when task execution is interrupted;
4. Error messages during terminal shell command execution.

Using the “causal tracing” method, the evaluation system can estimate each model’s net improvement rate relative to the overall average level within real session data mixed with varying task difficulties and user habits. The top position on the leaderboard was calculated based on over 17,500 real, long-cycle agent task sessions from global users. Compared to the previous generation base model’s net improvement rate of only +0.9%, the new model achieved +5.1%, an increase of 4.2 percentage points. This marks a substantial leap in the model’s usability for real-world complex tasks.

3. Core Agentic Capability Metrics Analysis

The model demonstrates significant improvements in segmented agentic capability metrics, specifically across the following core dimensions:

1. Bash Command Execution Reliability

When executing programming or system operations tasks, agents frequently need to generate and run shell commands. This metric measures the correctness and execution success rate of the commands generated by the model. Data shows that this metric surged from -4.5% (ranking 13th) in the previous generation to +7.9%, ranking first among open-weight models. This indicates that the model has undergone deep optimization in code syntax understanding, system environment state perception, and command-line tool usage standards.

2. User Satisfaction

This metric analyzes the ratio of praise to complaints in user sessions using natural language processing technology. Improving from +1.3% in the previous generation to +7.7%, it shows that the model’s outputs are not only logically correct but also better aligned with user expectations in terms of interactive experience, format presentation, and tone.

3. Steerability

In long-cycle tasks, users often need to correct the model based on intermediate results. The steerability metric measures the agility with which the model adjusts its execution direction based on user feedback. This model achieved +6.5%, also ranking first among open-weight models, reflecting strong context adherence and instruction-following capabilities, enabling it to quickly understand user correction intents and reset task execution paths.

4. Task Confirmation Success Rate

The metric for explicit user confirmation of task completion was +3.9%, ranking fourth among open-weight models. This relatively conservative metric indicates that, although the model excels in command execution and interactive feedback, there is still a certain failure rate when facing highly challenging long-tail tasks. This remains a common technical challenge across the current agentic landscape.

4. Large-Scale Joint Reinforcement Learning and Post-Training Strategy

The enhancement in model capabilities is not solely dependent on base architecture fine-tuning; its post-training strategy has also played a decisive role. Using the previous generation’s 100-billion-scale model as a base, the team conducted large-scale joint reinforcement learning (RL) post-training.

This post-training covered hundreds of downstream domains, jointly optimizing capabilities such as tool calling, code execution, and mathematical reasoning. By constructing a high-quality multi-step task training set and combining it with reward model mechanisms in RL, the model learned to perform task decomposition, tool selection, parameter filling, and result parsing upon receiving a task. This paradigm shift from “predicting the next token” to “planning the next action” is the key reason for its high scores in agentic evaluations.

5. Engineering Deployment Challenges and Cloud Elasticity Practices

Although the release of model weights provides academia and industry with research and application opportunities, a massive engineering gap exists between “downloadable” and “locally runnable” in actual deployment.

Stored in BF16 precision, the 2.8-trillion-parameter model has weight files totaling approximately 5.6TB. Even with aggressive quantization schemes like INT4, the weight files still require about 1.4TB of storage space. For single-node consumer-grade GPUs (typically featuring 24GB to 80GB of VRAM), direct deployment is virtually impossible. Even if multi-node distributed inference frameworks are utilized, network communication overhead and memory bandwidth bottlenecks will severely degrade inference speed.

To address the deployment challenges of ultra-large MoE models, the industry typically relies on mature cloud-native infrastructures. In practice, enterprises can leverage general-purpose computing capabilities provided by cloud vendors to solve this pain point. For example, using cloud container platforms for the elastic orchestration of distributed inference nodes, combined with auto-scaling of GPU instances to handle peak and valley business traffic. At the storage layer, the 5.6TB weight files require high-throughput object storage services for hosting and must be mounted to compute nodes via high-speed networks to reduce cold start loading times. Furthermore, long-cycle agent tasks are often accompanied by massive intermediate state logs, requiring the system to integrate reliable logging services and monitoring/alerting components. This enables real-time tracking of command execution error rates and task success rates, forming a data loop to continuously optimize prompt engineering and post-training strategies.

In terminal agent benchmarks, the model achieved a score of 65.7%, not only surpassing a certain closed-source model’s 58.2% but also narrowing the gap with the industry’s top model (66.4%) to less than one percentage point. This proves that through the deep integration of architectural innovation and reinforcement learning, open-weight models now possess the capability to rival top closed-source models in complex agentic application scenarios. As cloud-based distributed inference technologies continue to mature, ultra-large MoE models will unlock greater value in enterprise-grade scenarios such as automated programming and complex data analysis.