Engineering Ultra-Large MoE Architectures for Long-Horizon Agents

Engineering Ultra-Large MoE Architectures for Long-Horizon Agents

As Large Language Models (LLMs) evolve, simple text conversations can no longer meet the demands of complex industrial applications. “Long-Horizon Agents,” capable of autonomous planning, tool calling, and executing multi-step tasks, have become the core direction of technological advancement. Recently, the industry open-sourced an ultra-large Mixture of Experts (MoE) model with 2.8 trillion total parameters and 104 billion activated parameters. This model supports million-token contexts and native multimodality, introducing deep innovations in underlying attention mechanisms and post-training strategies. This article strips away brand-specific attributes to analyze the core mechanisms and engineering practices of such ultra-large MoE models in long-horizon Agent scenarios.

1. Ultra-Large MoE Architecture and Sparse Attention Optimization

Handling a context of up to a million tokens is foundational for long-horizon Agents executing complex tasks, but it brings exponential growth in memory and compute overhead. Traditional Multi-Head Attention (MHA) or Multi-Head Latent Attention (MLA) often fall short under extreme context lengths. This open-source model makes key architectural compromises and innovations by introducing sparse attention mechanisms.

1.1 Hybrid Attention Layer Design
Instead of adopting a single attention mechanism across all network layers, the model uses a hybrid architecture. Specifically, it replaces 24 layers of gated latent attention with Gated Dynamic Sparse Attention (DSA), combined with 69 layers of Delta Attention.

• DSA (Dynamic Sparse Attention): The core idea is “focus on demand.” When processing long sequences, DSA does not compute correlations for all historical tokens. Instead, it dynamically selects the most critical 2,048 historical tokens to participate in the attention calculation. This mechanism drastically reduces complexity from O(N²), making it possible to process 1 million tokens with limited computing power.
• Delta Attention: Serving as a complement to DSA, this mechanism focuses on capturing detailed features of local or adjacent tokens, compensating for local strong correlation information that DSA might miss during global sparse filtering.

1.2 MoE Load and Activation
The proportional design of 2.8T total parameters and 104B activated parameters reflects an ultimate balance between capacity and efficiency in the MoE architecture. The model activates only about 3.7% of its parameters during inference. This requires the routing algorithm to be highly accurate, ensuring each token is dispatched to the most matching expert sub-network. This sparse activation allows ultra-large models to run efficiently on relatively mainstream hardware clusters.

2. Joint RL Post-Training and Quantization Deployment

While the base model’s pre-training provides broad world knowledge, transforming it into a reliable Agent heavily relies on reinforcement learning (RL) strategies during the post-training phase.

2.1 Cross-Domain Joint Reinforcement Learning
The model employs a joint RL strategy covering hundreds of downstream domains during post-training. Traditional single-task RL often leads to unbalanced model capabilities. In contrast, the joint RL strategy constructs complex reward models to simultaneously optimize multi-step reasoning, tool calling, and code generation. This ensures the model maintains consistent logical rigor and instruction-following capabilities across different Agent scenarios, such as terminal operations and financial research.

2.2 MXFP4 Weight Quantization and Deployment Compatibility
To further lower the deployment barrier, the model released a version using MXFP4 (Microscaling FP4) weight quantization. As an emerging low-precision floating-point format, MXFP4 offers advantages in dynamic range and precision retention compared to traditional INT4. It effectively reduces routing anomalies and output collapse in MoE models after quantization.
Regarding the deployment ecosystem, the model natively supports mainstream high-performance inference frameworks and provides API interfaces compatible with leading cloud providers. Developers can seamlessly integrate it with existing Agent scheduling systems. For example, combining container platforms with GPU compute clusters—such as Baidu AI Cloud—allows for the rapid setup of a highly available inference service environment, utilizing elastic scaling to handle high-concurrency requests.

3. Multi-Step Tool Calling and Inference Budget Control for Long-Horizon Agents

In real-world business scenarios, an Agent’s execution depth and thinking time determine its practicality. The model introduces tiered thinking budgets and strict multi-turn context management mechanisms.

3.1 Three-Tier Thinking Budgets (Low/High/Max)
The model allows developers to set compute overhead during inference:

• Low: Suitable for low-latency requirement scenarios like simple intent recognition and document summarization.
• High: A balanced mode, suitable for routine web searches and cross-referencing multiple documents.
• Max: The maximum thinking tier, used for complex code architecture design, CAD generation, or multi-step terminal operations. In this mode, the model generates a detailed internal chain of thought, deeply analyzing boundary conditions before outputting an action.

3.2 History Retention Mechanism for Multi-Turn Tool Calling
Executing tasks as a long-horizon Agent often requires multiple cycles of “think - call tool - observe result - think again.” To ensure logical coherence, the system requires the complete history of the thinking process and tool calling messages to be retained. Any truncation might cause the model to lose critical task context.

Below is a simplified example of a multi-step Agent calling logic, demonstrating how to construct a request body with complete history records:

import requests
import json
# Assume the API endpoint follows the standard OpenAI compatible format
api_url = "https://api.your-inference-platform.com/v1/chat/completions"
headers = {"Authorization": "Bearer YOUR_API_KEY"}
# Construct a message sequence containing the complete history of thinking and tool calls
messages = [
    {"role": "user", "content": "Analyze the recent financial statements and generate a summary."},
    # Model's first round of thinking and tool call (must be retained)
    {"role": "assistant", "content": "Thinking: Need to call the retrieval tool to get financial data.", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "search_doc", "arguments": "{\"query\": \"2024 Q3 financial report\"}"}}]},
    # Tool return result
    {"role": "tool", "tool_call_id": "call_1", "content": "Revenue increased by 15%, costs increased by 5%..."},
    # Based on the complete history, set the high thinking budget for the next round of inference
    {"role": "assistant", "content": "Thinking: Based on the retrieved data, I will generate the final summary report..."}
]
payload = {
    "model": "ultra-moe-28t",
    "messages": messages,
    "thinking_budget": "high",  # Set thinking budget
    "max_tokens": 4096
}
# Send request
response = requests.post(api_url, headers=headers, json=payload)
print(json.loads(response.text).get('choices')[0]['message']['content'])

4. Core Application Scenarios and Capabilities

Through the optimization of the aforementioned architecture and training strategies, the model demonstrates outstanding capabilities in several highly challenging Agent benchmarks, validating its engineering practicality.

4.1 Deep Software Engineering and 3D Creation
In the domain of code development, the model goes beyond basic code completion to possess long-horizon software engineering problem-solving capabilities. In evaluations, the model could understand the structure of large codebases and autonomously perform terminal operations, bug localization, and fixes. In 3D design, the model supports generating CAD drawings via tool calls, indicating its ability to transform natural language design intents into engineering-grade precise graphics, greatly empowering industrial design automation.

4.2 Complex Knowledge Work and Office Automation
For enterprises and research institutions, the model performs exceptionally well in financial research and office document processing. It can autonomously conduct web searches, extract key metrics from massive unstructured data, and generate structured reports using office automation tools. The tiered thinking budget allows the model to be highly efficient when processing simple email classifications, while maintaining rigorous depth when handling complex merger and acquisition analyses.

5. Engineering Deployment Considerations and Compliance

When introducing such ultra-large open-source models into enterprise production environments, developers need to pay special attention to deployment forms and licensing agreements.

5.1 Adaptation of Inference Frameworks
Although the model supports various mainstream inference frameworks, actual stress testing reveals differences in how frameworks optimize MoE routing and MXFP4 quantization. Developers are advised to conduct thorough performance benchmarks before going live, monitoring GPU memory fragmentation and KV Cache hit rates. Concurrently, pairing with mature logging services and monitoring alert systems will help track inference latency and error rates in real time.

5.2 Constraints of Proprietary Licenses
Unlike fully open-source licenses such as Apache 2.0 or MIT, this model adopts a proprietary license. This means that before commercializing it, legal and technical teams must carefully review the licensing terms to clarify whether commercial use is permitted, if there are modification restrictions, and the attribution of derivative works. This ensures that the technical dividends of the open-source model are maximized legally and compliantly.

Conclusion

The combination of ultra-large MoE architectures, sparse attention, and RL post-training is reshaping the technological boundaries of long-horizon Agents. Transitioning from single-turn conversations to multi-step autonomous execution, LLMs are accelerating their evolution from “smart assistants” to “intelligent productivity.” Developers should deeply understand the underlying mechanisms and make rational plans for compute scheduling, context management, and engineering architecture to address the increasingly complex challenges of deploying AI applications.