Deep Dive: Reinforcement Learning Practices for Ultra-Large-Scale MoE Architectures and Long-Horizon Agents

1. Architecture Evolution: Deep Optimization of Sparse Attention Mechanisms

When processing context windows of up to one million tokens, the computational complexity and memory footprint of standard multi-head attention (MLA) grow exponentially. To overcome this bottleneck, the new model introduces a deep sparse architecture transformation.

Specifically, the model replaces 24 layers of gated multi-head attention with gated dynamic sparse attention (DSA). Unlike conventional sliding-window or static global masks, DSA employs a dynamic routing mechanism: during each attention computation, a lightweight gating network dynamically selects the 2,048 most relevant tokens from the entire historical token pool to participate in the current attention calculation.

This “on-demand addressing” mechanism significantly reduces the computational overhead of long sequences while preserving the ability to capture long-range dependencies accurately. The remaining 69 layers incorporate incremental attention, which maintains coherent local context propagation. Working in tandem, DSA handles dynamic sparse associations across long text, while incremental attention ensures the continuity of local information. This synergy allows the model to support 1M-token context while keeping activated parameters around 104B and maintaining high information density and inference quality.

2. Training Strategy: Joint Reinforcement Learning Across Multiple Domains

The model’s foundational capabilities come from pre-training, but its performance on complex agent tasks depends heavily on the reinforcement learning (RL) strategy applied during post-training. Rather than relying solely on instruction fine-tuning, this approach employs joint RL spanning hundreds of downstream domains.

2.1 Multi-Domain Task Alignment

The core challenge in joint RL is avoiding catastrophic forgetting across different tasks. The solution builds large-scale trajectory datasets covering vastly different domains—terminal operations, code repair, financial research, and CAD generation. During multi-stage RL, the model must generate not only final answers but also complete reasoning chains and tool-call sequences. A fine-grained reward model scores intermediate steps, teaching the model to dynamically adjust reasoning depth and tool-use strategies based on the task context.

2.2 Thinking Budget Tiering

To accommodate varying latency and cost requirements in production, the model introduces a “thinking budget” concept during inference, with three tiers: low, high, and max.

• Low tier: Suitable for simple entity extraction, single-turn Q&A, or format conversion. The model produces answers quickly to minimize time-to-first-token.
• High tier: Designed for scenarios requiring logical reasoning with context or multi-step tool calls, such as web search and data summarization.
• Max tier: Intended for high-difficulty tasks like long-horizon code agents or complex software engineering fixes. The model is allowed to perform deep self-reflection and multi-round trial and error.

This tiered mechanism provides developers with significant engineering flexibility, allowing them to make trade-offs based on specific business SLAs.

3. Key Application Scenarios for Long-Horizon Agents

Based on the architecture and training described above, the model demonstrates strong generalization across multiple long-horizon agent benchmarks. The following are representative technical application directions:

3.1 Code Development and Automated Software Engineering

In modern software engineering, bug fixes and feature iterations often require navigating across multiple files and understanding large codebases. The model can act as a core code agent, with terminal operation permissions, autonomously completing the full loop: reading code, locating issues, generating patches, running compilation and tests, and tracing errors back to their root cause.

In benchmarks such as complex software engineering repair tasks, the model must understand not only syntax but also the runtime behavior of programs. With the max thinking tier, the model can simulate code execution paths to generate logically sound fixes. Additionally, in 3D creation, by integrating CAD tool APIs, the model can directly generate or modify 3D structures from natural language descriptions, significantly lowering the barrier to using professional design software.

3.2 Knowledge Work and Multi-Source Data Fusion

The core challenge in enterprise knowledge work is information fragmentation. The model natively supports multimodal input (text and images) and long context, enabling it to process PDF research reports containing charts, complex office documents (e.g., nested tables, multi-page slide decks), and real-time web search results simultaneously.

For tasks like financial research, the agent first retrieves the latest earnings reports via search engines, then calls document parsing tools to extract key data points, and finally cross-validates and analyzes trends using its internal financial knowledge base. This multi-source data fusion capability positions the model not merely as a “text generator” but as a “data hub” within enterprise workflows.

4. Engineering Deployment and API Integration Best Practices

For enterprises and developers, the model’s capabilities must translate into efficiently deployable services. This technical solution includes deep optimizations in post-training and deployment.

4.1 Low-Bit Quantization and Inference Acceleration

Given the massive scale of 2.8T total parameters, full-precision storage would incur prohibitive GPU memory overhead. The model is released using MXFP4 (4-bit floating-point) weight quantization. Compared to traditional INT4 quantization, MXFP4 better preserves the statistical distribution of weights, effectively reducing quantization-induced errors that could impact logical reasoning.

On the inference engine compatibility front, the open weights natively support mainstream high-performance frameworks (e.g., vLLM, SGLang). This enables utilization of underlying operator fusion optimizations and PagedAttention technology to maximize throughput in high-concurrency scenarios.

4.2 Message Management for Multi-Turn Tool Calls

When building agent applications on OpenAI/Anthropic-compatible APIs, managing state across multi-turn tool calls is a common pitfall. The engineering implementation emphasizes a critical rule: the full history of reasoning and tool-call messages must be retained.

Below is a simplified example of multi-turn tool-call message structure, showing how to correctly maintain agent state in an API request:

messages = [
    {"role": "user", "content": "Help me look up the latest price for object storage from a cloud provider and write it to a report"},
    # Round 1: The model decides to call a search tool
    {"role": "assistant", "content": "Thought: I need to fetch pricing info first. I'll call search_tool.", "tool_calls": [{"id": "call_1", "function": {"name": "search_tool", "arguments": "{\"query\": \"cloud object storage price\"}"}}]},
    # Tool returns result
    {"role": "tool", "tool_call_id": "call_1", "content": "Standard storage price is $0.012/GB/month..."},
    # Round 2: The model continues based on the result
    {"role": "assistant", "content": "Thought: Price retrieved. Now I need to call write_report to save the data.", "tool_calls": [{"id": "call_2", "function": {"name": "write_report", "arguments": "{\"data\": \"$0.012/GB/month\"}"}}]},
    # Final result
    {"role": "assistant", "content": "Successfully wrote the retrieved pricing information to the report."}
]

In the flow above, the model depends on the historical reasoning chain (the “Thought” portions in the content field) to decide its next action. If developers truncate the history of reasoning or tool-returned messages to save tokens, the model may fall into looped calls or lose task context. Therefore, when building logging services or message queue caches, it is essential to design data structures that persist these intermediate states completely.

5. Summary and Outlook

The open-source implementation of a 2.8T-parameter MoE architecture with native 1M-token context provides a new technical foundation for long-horizon agents. Through the deep integration of dynamic sparse attention (DSA) and joint reinforcement learning, the model demonstrates near-expert problem-solving capabilities in high-value scenarios such as code engineering and complex data analysis.

For enterprise developers, several key considerations should be prioritized when adopting such ultra-large-scale models: First, leverage cloud-native container platforms and high-performance inference frameworks to address distributed loading of massive weights. Second, configure “thinking budgets” flexibly based on business scenarios to find the optimal balance between response latency and reasoning depth. Third, establish robust monitoring for tool-call chains to ensure stability in multi-turn interactions. Looking ahead, as sparse architectures and reinforcement learning continue to evolve, agents with long-horizon planning and execution capabilities are poised to become a core driving force in enterprise digital transformation.