Ultra-Large Sparse MoE Architecture and Long-Horizon Agent Practices

Ultra-Large Sparse MoE Architecture and Long-Horizon Agent Practices

As large model applications expand into complex software engineering, automated terminal operations, and deep research, models need not only vast knowledge reserves but also long-context understanding and long-horizon task planning capabilities. Recently, an ultra-large sparse MoE (Mixture-of-Experts) architecture with 2.8T total parameters and approximately 104B activated parameters has emerged. By introducing dynamic sparse attention and large-scale joint reinforcement learning post-training, this architecture significantly improves the stability and accuracy of long-horizon agent task execution. This article strips away specific product attributes and breaks down the core mechanisms and engineering practices of this architecture from a general technical perspective.

1. Ultra-Large MoE Architecture and Native Multimodal Processing

When handling million-token contexts and native image-text inputs, traditional dense models face bottlenecks of excessive memory usage and exponentially rising inference latency. Sparse MoE architecture resolves this contradiction through conditional computation.

1.1 Parameter Allocation and Conditional Routing
The architecture has a total parameter scale of 2.8T, but only activates approximately 104B parameters in a single forward pass. This means that for each input token, the routing network assigns it to the best-matching expert sub-network based on its features. This design allows the model to maintain inference speed comparable to a tens-of-billions-parameter dense model while possessing massive knowledge capacity.

1.2 Native Multimodal Fusion
In long-horizon agent scenarios (such as reading PDF documents containing charts and generating code), the model must natively support image-text inputs. This architecture unifies the representation of text and visual features at the input layer, avoiding the information loss caused by traditional external vision encoders. Within a million-token context window, the model can continuously maintain global awareness of multimodal information, providing precise contextual evidence for subsequent tool calls.

2. Engineering Breakthroughs in Dynamic Sparse Attention Mechanisms

Processing a 1-million-token context with traditional self-attention mechanisms incurs extremely high computational complexity. The new architecture deeply reforms the underlying attention mechanism, combining Gated Dynamic Sparse Attention (Gated DSA) with Delta Attention.

2.1 Gated Dynamic Sparse Attention (Gated DSA)
Compared to earlier versions that fully used Gated MLA (Gated Multi-Layer Attention), the new architecture replaces it with Gated DSA in specific 24 layers of the network. The core idea of DSA is to dynamically select historical tokens for computation. Under this mechanism, each attention computation selects only the 2,048 most relevant historical tokens. This “hard” filtering dramatically reduces redundancy in long-sequence computation, enabling the model to focus on key information when processing ultra-long contexts and avoiding attention dispersion.

2.2 Synergistic Supplement of Delta Attention
The other 69 layers adopt a mechanism combining Delta Attention with Gated DSA alternately. Delta Attention is typically used to capture local or relative positional change information. The combination of both ensures precise capture of local details while maintaining the establishment of global long-range dependencies. From a systems engineering perspective, this hybrid attention mechanism effectively balances computational efficiency and context quality.

3. Joint Reinforcement Learning and Quantized Deployment Practices

Once the base model’s capabilities are established, the post-training stage determines its performance in specific agent tasks.

3.1 Large-Scale Joint Reinforcement Learning Post-Training
This architecture adopts a joint reinforcement learning (RL) post-training strategy covering hundreds of downstream domains. Unlike single-task fine-tuning, joint RL enables the model to find a balance point across heterogeneous tasks such as multi-step reasoning, tool calling, and code generation. By introducing complex environment feedback during the reinforcement learning process, the model learns how to self-correct when encountering errors and how to plan multi-step task paths — an indispensable “reflection” capability for long-horizon agents.

3.2 MXFP4 Weight Quantization and Inference Framework Adaptation
To lower the deployment barrier for a 2.8T-parameter model, weights adopt MXFP4 (Microscaling FP4) quantization technology. MXFP4 groups multiple weight values into blocks that share a scaling factor, substantially compressing model size (reducing memory usage by approximately 75% compared to FP16) while reasonably preserving the precision of the MoE routing network and expert weights.

On the deployment side, the model supports mainstream open-source inference engines (such as vLLM, SGLang, etc.). Below is illustrative code using a general compatible API for multi-turn tool calling, demonstrating how to retain complete historical thinking and tool-call messages:

import requests
# Assume deployment on a local or cloud container platform
api_url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Fi": "application/json"}
# Build multi-turn conversation history, retaining complete thinking process and tool call records
messages = [
    {"role": "user", "content": "Check recent CPU utilization logs and generate optimization suggestions"},
    # Retain the model's thinking process
    {"role": "assistant", "content": "Thinking: Need to call the monitoring alert API to get CPU data.", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_cpu_usage"}}]},
    # Retain the tool's returned result
    {"role": "tool", "tool_call_id": "call_1", "content": "Average CPU utilization reached 85%"},
    # Model continues reasoning based on history
    {"role": "user", "content": "Based on the above logs, provide a code-level optimization plan"}
]
payload = {
    "model": "ultra-moe-model",
    "messages": messages,
    "thinking_budget": "max"  # Set thinking budget to the highest tier
}
response = requests.post(api_url, json=payload, headers=headers)
print(response.json())

4. Three-Tier Thinking Budget and Long-Horizon Agent Task Adaptation

In complex agent tasks, not all steps require maximum-depth reasoning. This architecture introduces a three-tier thinking budget mechanism — low, high, and max — allowing developers to dynamically allocate compute based on task node difficulty.

4.1 Application Scenarios for Low and High Budgets
For routine web searches, simple office document format conversion, or single API parameter extraction, using low or high budget can greatly reduce time-to-first-token and improve interaction experience. In dialogue scenarios requiring rapid response, this mechanism effectively controls inference costs.

4.2 Necessity of Max Budget in Complex Engineering
When facing long-horizon code agent tasks (such as fixing complex bugs with cross-file dependencies), terminal operation planning, or deep financial research, the model needs to construct long-chain thought trees. Enabling max budget at this point allows the model to fully utilize the implicit knowledge of 2.8T parameters, combined with historical thinking and tool-call messages, to conduct deep multi-path exploration and self-verification.

Taking a typical software engineering task as an example, under max budget the model first reads through the global code context and plans a multi-step process of “locate the issue -> write test cases -> modify core logic -> verify tests.” If an error occurs during tool calls, the model retains the error information in the historical context and adjusts subsequent strategies until the task is completed. This mechanism requiring multi-turn tool calls to preserve complete historical messages is the core safeguard ensuring long-horizon tasks do not deviate from their goals.

5. Enterprise-Grade Deployment and General Cloud-Native Architecture Integration

In enterprise-grade implementation, inference services for ultra-large MoE models typically require deep integration with cloud-native infrastructure. Taking a general cloud platform architecture as an example, model deployment and invocation typically involve the following core components:

1. Container Platform and Elastic Scaling: Different experts are activated under different inputs for MoE models, causing fluctuations in GPU memory and computational load. Through container platforms combined with monitoring and alerting services, the number of inference node replicas can be dynamically adjusted based on request queue depth.
2. Object Storage and Model Weight Distribution: Even under MXFP4 quantization, 2.8T parameters remain volumetrically large. Using object storage services to store model shard weights, combined with high-speed internal networks for parallel pulling during node startup, can significantly shorten service cold-start time.
3. Message Queues and Asynchronous Task Decoupling: For time-consuming long-horizon agent tasks (such as 3D CAD generation or full codebase refactoring under max budget), synchronous front-end waiting is highly prone to timeout triggers. The best practice is to use message queues to make tasks asynchronous — the inference engine consumes queue messages, completes multi-step tool calls, and then writes results to log services or databases for client polling or WebSocket streaming.

Summary

The ultra-large sparse MoE architecture with 2.8T total parameters and 104B activated parameters breaks through the compute bottleneck of million-token contexts through the combination of Gated DSA and Delta Attention. Large-scale joint reinforcement learning post-training endows the model with powerful long-horizon task planning and multi-step tool-calling capabilities, while the three-tier thinking budget provides flexible cost and latency control for practical engineering applications. After integrating with general cloud-native infrastructure, this technical solution provides a solid technical foundation for enterprise-grade code agents, automated research analysis, and other complex scenarios.