Large-Scale MoE Architecture Explained: Reinforcement Learning Practices for Building Long-Horizon Agents

As AI moves toward complex, real-world applications, building long-horizon agents remains constrained by compute overhead, context memory decay, and the stability of multi-step logical reasoning. A recently open-sourced model with 2.8 trillion total parameters and 104 billion activated parameters tackles these challenges head-on. Beyond supporting million-token context and native image-text input at the architectural level, its large-scale joint reinforcement learning (RL) post-training significantly improves performance on complex tasks such as terminal operation, software engineering, and 3D design. This article strips away product-specific details and provides a deep technical analysis of the core path to building long-horizon agents, covering the underlying architecture, attention mechanism innovations, RL strategies, and engineering deployment guidance.

1. Large-Scale MoE Architecture and Multimodal Processing

Building an agent capable of long-horizon operation requires, first and foremost, sufficiently wide information bandwidth and knowledge capacity. The model family discussed here uses a sparse Mixture-of-Experts (MoE) architecture with 2.8T total parameters and 104B activated parameters. The core advantage of MoE is “compute decoupling”: a routing network dynamically assigns input tokens to the best-matching expert subnetworks, so the model retains a massive parameter scale—and therefore rich world knowledge—while keeping the compute cost of each forward pass on par with a 104B-parameter dense model.

On the input side, the architecture supports a 1-million-token context window and native image-text input. This means that when an agent analyzes large code repositories, parses extremely long API documentation, or runs vision-driven UI automation tests, it does not need to rely on an external retrieval-augmented generation (RAG) system to slice and stitch context—the full global context can be fed directly into the model for native processing. Combining native multimodality with long context greatly reduces the risk of “hallucination” caused by information fragmentation and provides a solid memory foundation for long-horizon tasks.

2. The Evolution of Dynamic Sparse Attention

Maintaining efficient attention computation over ultra-long contexts is key to controlling inference latency and memory usage. Compared with the previous generation, which used gated multi-head latent attention (MLA) throughout, the new version makes a key upgrade deep in the architecture: 24 layers are replaced with gated dynamic sparse attention (DSA) and combined with 69 layers of delta (incremental) attention.

The essence of this design is dynamic filtering of historical token information. In each DSA layer, only the 2,048 most relevant historical tokens are selected to participate in the attention computation at each step. This breaks the “everyone participates” compute bottleneck of traditional Transformer global attention, reducing the computational complexity of long-text reasoning from near-quadratic to linear growth. Meanwhile, the gating mechanism keeps the dynamic selection process differentiable and training stable, while Delta Attention handles high-frequency updates of local context. The alternating fusion of the two allows the model to accurately capture long-range logical dependencies even at million-token lengths, avoiding the common “lost in the middle” pain point.

3. Joint Reinforcement Learning Post-Training for Complex Scenarios

A model’s base capabilities are only the starting point; excellent long-horizon agent performance depends even more on high-quality post-training alignment. The model adopts a joint reinforcement learning strategy covering hundreds of downstream domains. Unlike single-task fine-tuning, joint RL combines a cross-domain general reward model with task-specific reward functions, enabling the model to improve across multiple capability dimensions simultaneously.

For example, in code repair tasks, the model is rewarded not only for the syntactic correctness of the final code but also for the logical rigor of the debugging process; in office document processing, it is rewarded for the accuracy of information extraction and the compliance of tool calls. Through this multi-dimensional policy optimization, the model gains the ability to adapt to task complexity. At inference time, three thinking budgets—Low, High, and Max—are available, and developers can configure them flexibly based on real-time requirements and the depth of reasoning needed. In high-budget mode, the model can unfold deep internal chains of thought to effectively handle multi-step planning and trial-and-error correction.

4. Long-Horizon Agent Application Scenarios

With its architectural advantages and deep alignment, this type of model demonstrates practical value for long-horizon agents in the following core scenarios:

1. Code development and software engineering: In terminal operations, large codebase refactoring, and defect fixing, an agent must understand the structure and dependencies of an entire project. Ultra-long context allows the model to load core modules in one pass, and the tool-calling capability enabled by joint RL lets it autonomously complete the full loop of “locate the error - read the relevant files - modify the code - run tests - fix again.” Official evaluations show that this type of model achieves strong pass rates on industry-standard software engineering and terminal operation benchmarks.

2. 3D creation and CAD generation: 3D design tasks typically involve precise parameter constraints and spatial topology logic. By parsing natural language instructions or reference sketches and combining them with CAD tool APIs, the model can perform multi-step geometry generation, Boolean operations, and parameter fine-tuning. This capability breaks the limitation of traditional large models to text or 2D generation and represents an important step toward interaction with the physical world.

3. Automated knowledge work: In financial research, web retrieval, and office document processing, information is highly fragmented and unstructured. The model can plan retrieval paths, extract multi-source heterogeneous data, and invoke computational tools to generate final reports, greatly increasing the information throughput of knowledge workers.

5. Engineering Deployment and Technical Practice Guide

For enterprise deployment, the open release of model weights brings significant convenience, but real-world engineering still requires attention to inference frameworks and calling conventions.

1. Weight precision and inference backends: The model offers an MXFP4 quantized weight version. MXFP4 is a microscaling floating-point format that substantially compresses memory usage—allowing the very large model to be deployed on limited hardware—while preserving the precision of MoE routing and the numerical dynamic range of expert weights. For deployment, mainstream high-performance inference engines such as vLLM or SGLang are recommended; these frameworks are deeply optimized for MoE expert parallelism and continuous batching.

2. Context management for multi-turn tool calls: The core of a long-horizon agent is the multi-turn “think-act-observe” loop. In actual API calls, developers must retain the complete message history, including the model’s internal thinking process and the external tool call/return messages. Truncating or omitting history causes the model to lose awareness of task state, leading to infinite loops or incorrect calls. The following is a simplified example of the multi-turn calling logic:

import json
from openai import OpenAI
# Assume an OpenAI-compatible inference backend
client = OpenAI(base_url="http://localhost:8000/v1", api_key="demo_key")
messages = [
    {"role": "user", "content": "Analyze the code structure in the current directory and fix the null pointer exception in utils.py."}
]
def run_agent_loop():
    while True:
        response = client.chat.completions.create(
            model="large-moe-agent-model",
            messages=messages,
            # Set the max thinking budget to handle complex code logic
            extra_body={"thinking_budget": "max"}
        )
        msg = response.choices[0].message
        messages.append(msg)
        # Check whether the model has issued tool calls
        if msg.tool_calls:
            for tool_call in msg.tool_calls:
                # Execute the tool logic (e.g., reading files, running scripts)
                result = execute_tool(tool_call.function.name, 
                                      json.loads(tool_call.function.arguments))
                # The tool result must be appended back to the context as a tool-role message
                messages.append({
                    "role": "tool",
                    "tool_call_id": tool_call.id,
                    "name": tool_call.function.name,
                    "content": str(result)
                })
        else:
            # The model requested no tools, meaning the task is complete
            print("Agent Final Answer:", msg.content)
            break

1. Enterprise-grade infrastructure support: In real production environments, deploying a long-horizon agent depends not only on the model itself but also on the underlying cloud-native infrastructure. It is advisable to use container platforms for elastic scaling of inference nodes, object storage to manage large volumes of model weights and multi-turn conversation logs, and a comprehensive monitoring and alerting system to track MoE routing hit rates, token generation latency, and memory utilization in real time, ensuring service stability under high concurrency.

6. Compliance and License Considerations

Finally, it is important to emphasize that although the model weights are open, developers must review the dedicated license before use. Very large models of this kind often adopt specific non-standard open-source licenses (rather than permissive ones such as Apache 2.0 or MIT), which may impose restrictions on commercial use, the openness of derivative models, and permitted use cases. Before integrating such a model into core business lines, enterprises should conduct a strict compliance review to ensure that deployment complies with applicable laws, regulations, and license terms.

In summary, the combination of a 2.8T sparse MoE architecture, gated DSA attention, and joint reinforcement learning represents an important trend in the evolution of long-horizon agent technology. By deeply understanding the underlying mechanisms and following sound engineering practices, developers can effectively harness this powerful technical foundation to build highly reliable intelligent applications in complex code engineering and automated knowledge work.