The Evolution of Reinforcement Learning: Scaling MoE Models for Long-Horizon Agents

The Evolution of Reinforcement Learning: Scaling MoE Models for Long-Horizon Agents

In recent years, as large language models (LLMs) take on increasingly central roles in complex tasks, traditional single-turn dialogues or simple chain-of-thought invocations are no longer sufficient for terminal operations, multi-step code repairs, and deep industry research. Models are evolving to support ultra-long contexts, native multimodal capabilities, and long-horizon autonomous planning. Recently, an open-source project released an ultra-large-scale Mixture of Experts (MoE) model that has garnered significant industry attention. Through large-scale joint reinforcement learning (RL) post-training and deep modifications to sparse attention mechanisms, it significantly improves the execution efficiency and accuracy of long-horizon agents. This article explores this technical trend across dimensions of technical architecture, attention mechanism innovation, agent engineering practices, and deployment integration.

1. Ultra-Large-Scale MoE Architecture and Long Context Processing

When handling complex agent tasks, models must maintain long-term memory and state tracking, placing extremely high demands on parameter scale and context windows. The open-source model utilizes an MoE architecture with a total parameter count of 2.8 trillion (2.8T) and approximately 104B activated parameters.

1.1 Balancing Parameter Sparsity and Computational Efficiency
The core of the MoE architecture lies in “sparse activation.” Although the total parameter count is massive, during a single forward pass, the routing network selects only a small subset of expert networks to participate in the computation. This design ensures the model possesses extremely high capacity and knowledge reserves while keeping actual inference computation at the level of a 100B-parameter model, drastically reducing computational consumption during long-horizon tasks.

1.2 Native Multimodality and Million-Token Context Window
The model supports an ultra-long context input of 1 million tokens and natively supports image-text inputs. In long-horizon agent scenarios, this means the model can ingest an entire code repository’s history logs, multiple lengthy financial research reports, or complex CAD design blueprints at once. It can perform global context understanding without relying on complex external chunking and Retrieval-Augmented Generation (RAG) strategies, effectively mitigating information loss and context fragmentation.

2. Innovative Evolution of Sparse Attention Mechanisms

The bottleneck of processing million-token contexts is the quadratic growth in computational complexity of self-attention mechanisms relative to sequence length. To break this bottleneck, the new generation model features significant structural innovations in the attention layer, transitioning from traditional Gated MLA (Multi-head Latent Attention) to a more efficient sparse attention scheme.

2.1 Hybrid Architecture of Gated DSA and Delta Attention
The new architecture employs Gated DSA (Dynamic Sparse Attention) in 24 network layers and combines Delta Attention mechanisms in the remaining 69 layers. This hybrid architecture aims to balance capturing global long-horizon information with processing local high-frequency details.

2.2 Dynamic Historical Token Selection Strategy
In the Gated DSA layers, attention computation is no longer globally dense; instead, it dynamically selects 2,048 of the most relevant historical tokens for each computation. This importance-based dynamic routing strategy drastically reduces the complexity of long-sequence attention computation from O(N^2), making it possible to process 1 million tokens with limited VRAM while retaining precise recall of critical historical information.

3. Joint Reinforcement Learning and Thinking Budgets for Complex Scenarios

The leap in model capability relies not only on the underlying architecture but also on post-training strategies. The model adopts a joint reinforcement learning (RL) mechanism covering hundreds of downstream domains.

3.1 Multi-Domain Joint Reinforcement Learning
Unlike fine-tuning for a single vertical domain, joint RL requires the model to simultaneously optimize policies across heterogeneous tasks such as terminal operations, code engineering, web browsing, and financial research. This multi-task joint training helps the model learn more generalized tool invocation logic and task decomposition capabilities, rather than overfitting to a specific data distribution.

3.2 Tiered Thinking Budgets
To balance inference depth and response latency, the model provides three thinking budget configurations during the inference phase: low, high, and max.

• Low: Suited for simple intent recognition or quick document Q&A, reducing internal chain-of-thought generation to achieve millisecond-level responses.
• High: Appropriate for routine code repairs or multi-step tool invocations, allowing the model to perform moderate self-reflection and plan correction.
• Max: Designed for extremely complex long-horizon tasks (e.g., SWE marathon-level code repairs, CAD generation). The model fully utilizes computing power for deep reasoning, exploring multiple solution paths. This flexible budget control provides developers with great convenience for cost optimization across different business scenarios.

4. Engineering Practices and Task Evaluation for Long-Horizon Agents

In long-horizon agent applications, the model needs not only to “think” but also to “act.” Official evaluation data and engineering specifications provide best practice references.

4.1 Core Benchmark Analysis
With the default Max thinking budget and the assistance of partial agent frameworks, the model performed excellently in several rigorous benchmarks:

• Terminal Operations: Scored 65.7 on Terminal-Bench 4.0, demonstrating robust command-line interaction and system-level task execution capabilities.
• Software Engineering: High scores on SWE Marathon v1.1 (51.7) and DeepSWE v1.1 (75.5) prove its practical utility in locating bugs, fixing logic, and refactoring within complex codebases.
• Retrieval and Knowledge Work: BrowseComp (92.1) and OfficeQA Pro v2 (64.4) reflect its advantages in filtering massive web information and deep understanding of office documents.
• 3D Creation: Scored 92.7 on BenchCAD with tools, indicating the model can effectively combine professional tool APIs to accomplish complex 3D design tasks.

4.2 History Message Management for Multi-Turn Tool Invocations
A critical engineering detail when integrating such agent models is message queue management. The model requires retaining the complete “historical thinking process” and “tool invocation messages” across multi-turn tool calls. This means developers cannot arbitrarily truncate intermediate reasoning steps to save the context window. Because each action of a long-horizon agent depends on the reasoning rationale of the previous step, truncating historical thoughts causes the model to lose its logical grasp of the task context, leading to hallucinations or infinite loops.

5. Deployment, Integration, and Quantization Optimization Practices

To lower the deployment barrier for ultra-large models, the model utilizes MXFP4 weight quantization technology and has released the relevant weights.

5.1 Engineering Advantages of MXFP4 Quantization
MXFP4 (Microscaling FP4) is an advanced low-bit floating-point quantization format. Compared to traditional INT4, MXFP4 maintains an extremely high compression ratio while better preserving the dynamic range of model weights, reducing the impact of quantization errors on long-horizon task inference. This allows a 2.8T parameter model to be deployed and run on more mainstream hardware clusters.

5.2 Inference Framework Compatibility and API Design
The quantized model natively supports various mainstream open-source inference frameworks (such as vLLM, SGLang, and TokenSpeed) and provides API interfaces compatible with OpenAI and Anthropic standards. This design drastically reduces migration costs for developers. Enterprises can directly reuse existing agent middleware logic and integrate the new model simply by changing the API endpoint.

5.3 Cloud-Native Deployment Recommendations
For actual enterprise-grade deployments, it is recommended to combine such large-parameter MoE models with mature cloud-native components:

• Container Platforms: Utilize elastic container instances to handle high-concurrency fluctuations during long-horizon tasks.
• Object Storage: Used for storing massive historical conversation logs and intermediate thinking steps.
• Monitoring and Alerting: Track GPU VRAM usage, KV-Cache hit rates, and token generation latency in real-time to dynamically adjust thinking budget configurations.

6. Licensing Compliance Considerations

Finally, it must be emphasized that although the model has open weights, it uses a proprietary license rather than the industry’s most permissive Apache 2.0 or MIT licenses. Before integrating it into commercial products or internal workflows, enterprises must carefully review the commercial use restrictions, derivative work open-source requirements, and data compliance terms within the proprietary license to ensure compliant business operations.

Conclusion

From the parameter scaling of large-scale MoE architectures to the underlying innovations in dynamic sparse attention mechanisms, and the engineering implementation of joint reinforcement learning and tiered thinking budgets, ultra-large-scale models are rapidly evolving along the path of “smarter reasoning, more autonomous action.” When building next-generation long-horizon agents, developers should focus on the completeness of context management, dynamic control of inference costs, and performance tuning of quantized deployments to fully leverage the potential of these cutting-edge technical architectures.