Open-Sourcing Ultra-Large MoE Models: Long-Horizon Agent Architecture Evolution and Engineering Practices

Open-Sourcing Ultra-Large MoE Models: Long-Horizon Agent Architecture Evolution and Engineering Practices

As Large Language Models (LLMs) evolve from single-turn conversations to executing complex tasks, long-horizon agent capabilities have become a core metric for evaluating frontier models. Recently, the industry open-sourced an ultra-large Mixture-of-Experts (MoE) model with a total parameter count of 2.8T and 104B activated parameters. Beyond its sheer scale, this model introduces significant innovations in native multimodal understanding, a million-token context window, and long-horizon complex task planning. This article delves into the underlying architectural evolution, core capabilities, and engineering best practices for enterprise deployment.

1. Core Architectural Innovation: Deep Integration of Sparse Attention and MoE

Handling a context window of one million tokens imposes rigorous demands on computational complexity and memory footprint. Traditional Transformer architectures often face performance bottlenecks in this scenario. This open-source model boldly restructures its architecture, primarily introducing an improved sparse attention mechanism.

1.1 Hybrid Attention Layer Design
During the post-training and architectural fine-tuning phases, the model replaced its original 24 layers of Gated Multi-Head Latent Attention with Gated Dynamic Sparse Attention (DSA). This mechanism breaks the inherent pattern of global attention by dynamically selecting the 2,048 most relevant historical tokens to participate in the current attention computation. This “dynamic focus” strategy drastically reduces the computational complexity of long sequences, allowing the model to maintain sharp capture of core information when processing ultra-long documents or persistent agent interactions.

1.2 Supplementing with Delta Attention
In addition to the 24 DSA layers, the model retains a 69-layer Delta Attention structure. This hybrid stacking of “sparse + delta” attention layers preserves the global receptive field while optimizing information transmission loss. As a result, the model demonstrates enhanced robustness in ultra-long text retrieval and dependency tracking tasks.

1.3 Large-Scale Joint Reinforcement Learning Training
During the post-training phase, the model adopted a joint reinforcement learning (RL) strategy covering hundreds of downstream domains. Unlike simple instruction tuning, joint RL constructs multi-dimensional reward functions to align the model’s capabilities across heterogeneous tasks such as coding, tool invocation, and logical reasoning. Concurrently, the training process natively supports MXFP4 weight quantization, laying the groundwork for subsequent efficient inference deployment.

2. Deconstructing Long-Horizon Agent Capabilities and Scenario Practices

The core of a long-horizon agent lies in being “goal-driven” and capable of “multi-step execution.” The model offers low, medium, and high thinking budgets, allowing developers to dynamically allocate computational resources based on task complexity. This provides immense convenience for the implementation of complex workflows.

2.1 Software Engineering and 3D Structured Creation
In software engineering, long-horizon agents must possess repository-level understanding and repair capabilities. The model performs exceptionally well in terminal operations, code repairs, and automated software engineering evaluations. Developers can integrate it into intelligent programming tools to build a closed-loop workflow of “issue localization - code repair - terminal verification.” For example, in CAD generation and 3D design tasks, the model not only understands structured instructions but also completes multi-step operations from conceptual design to parametric modeling by combining external tools.

2.2 Knowledge Work and Multimodal Information Processing
The model natively supports image and text inputs, greatly expanding its application boundaries in scenarios like financial research and office document processing. When handling complex documents containing charts, scanned files, and mixed layouts, the model accurately extracts structured data and combines web search capabilities to conduct in-depth industry analysis.

2.3 Multi-Turn Tool Invocation and State Machine Management
Agents often need to chain multiple tools to execute a task. The model is natively compatible with mainstream OpenAI and Anthropic API protocols, supporting multi-turn tool invocation. In practical development, developers must ensure that the complete “thinking process” and “tool invocation messages” are retained across multi-turn conversations, which is crucial for maintaining the consistency of the agent’s state machine.

3. Enterprise Deployment and Inference Service Engineering Guide

Deploying a 2.8T-scale MoE model locally or in the cloud presents severe computing power challenges for enterprises. The model is open-sourced with MXFP4 quantized weights and is fully adapted to mainstream high-performance inference frameworks.

3.1 High-Performance Inference Framework Adaptation
Developers can deploy the model using industry-standard lightweight inference engines such as vLLM, SGLang, and TokenSpeed. MXFP4 quantization effectively compresses the memory footprint, making it possible to load a model with 100B-level activated parameters within a limited GPU cluster. Combined with memory optimization technologies like PagedAttention, inference throughput can be significantly improved.

3.2 Cloud-Native Architecture Integration
In enterprise deployments, model weights alone cannot solve all problems. Developers typically need to deploy inference services within container platforms, utilizing elastic scaling mechanisms to handle high-concurrency requests. Additionally, message queues should be introduced for asynchronous task decoupling (especially critical for long-running agent tasks), while object storage should be used to save intermediate states and log files. A comprehensive monitoring and alerting system is necessary to track GPU utilization, Time-to-First-Token (TTFT) latency, and context length distribution in real time.

3.3 Best Practices for Multi-Turn Conversation History Management
Because the model supports a million-token context, multi-turn interactions generate massive amounts of historical data. In engineering implementations, it is recommended to adopt a “bulk truncation + summary reinjection” strategy. The core thinking and tool invocation chains must be fully preserved; for casual chat or low-value intermediate states, compression management can be achieved through external caches and vector databases. This approach maintains the agent’s memory depth while controlling inference costs.

4. License Compliance and Model Usage Guidelines

It is important to note that the open-sourced model weights do not use the common Apache 2.0 or MIT licenses. Instead, they are governed by a proprietary license agreement provided by the model’s publisher.

4.1 Commercial Compliance Review
Before integrating this model into commercial products or internal workflows, enterprises must carefully review the terms of the proprietary license. Focus specifically on the scope of commercial authorization, daily request limits, open-source requirements for derivative models, and data privacy protection clauses to avoid business risks arising from compliance issues.

4.2 API Compatibility and Smooth Migration
The model provides API interfaces compatible with OpenAI and Anthropic protocols. This means enterprises can achieve a smooth switch of their model services by simply changing the API Endpoint, without modifying existing SDKs or business logic code. This standardized interface design significantly reduces the development costs associated with system migration and multi-model routing management.

Conclusion

The open-sourcing of a 2.8T parameter sparse MoE model marks a solid step forward for open-source LLMs in the fields of long-horizon agents and ultra-long context processing. The underlying technical evolution—from Gated Dynamic Sparse Attention to multi-stage joint reinforcement learning—provides highly valuable engineering references for developers. As high-performance inference frameworks and cloud-native infrastructures continue to mature, building super agents capable of autonomously planning and executing complex, long-cycle tasks is transitioning from a theoretical concept to large-scale engineering implementation.