Open-Sourced Ultra-Large Sparse MoE Model: Architecture and Inference Practices for Long-Horizon Agents

Model Overview: A Sparse Expert Architecture for Long-Horizon Agents

In agentic workflows, a model typically has to carry a complex task across many steps: understanding the user’s goal, invoking external tools, reading intermediate results, correcting its execution path, and finally delivering the result. Tasks of this kind place four core demands on a model: long-context capacity, tool-oriented reasoning, multimodal perception, and efficient inference under a limited compute budget.

The newly released model adopts a sparse Mixture-of-Experts (MoE) architecture with 2.8T total parameters, of which 104B are activated per forward pass. Rather than engaging all of its parameters during inference, the model selectively activates a subset of expert modules according to the features of each input token. This preserves the capacity of a very large model while keeping computational cost under control. Compared with the previous version, the model maintains overall architectural continuity while specifically strengthening reasoning ability and tool coordination for long-horizon agent tasks.

The model natively supports a context window of 1 million tokens — a capability that matters enormously for long-horizon agents. In multi-turn tool-calling scenarios, the system must continuously retain the complete thinking process, tool call requests, tool results, and intermediate reasoning states. When the context window is too small, the system is forced to truncate history or compress it through summarization, which can lose critical information and degrade task quality. A 1-million-token window gives complex tasks ample “working memory.”

On the input side, the model natively accepts combined image and text. An agent can therefore process not only textual instructions but also visual information such as screenshots, UI layouts, and design drawings. In terminal operation tasks, for example, the agent can perceive the current interface state from a screenshot; in 3D design tasks, the model can reason jointly over engineering drawings and parameter text.

Attention Upgrade: From Gated MLA to Gated DSA

Attention is the core component that lets large language models handle long sequences. In the previous generation, a portion of the model family’s layers used Gated MLA (Multi-head Latent Attention), a structure that compresses the Key-Value cache through low-rank latent projection and, to some extent, alleviates the memory pressure of long-context inference.

The new version replaces 24 of these Gated MLA layers with Gated DSA (Dynamic Sparse Attention), which works alongside the existing 69 layers of Delta Attention. The core idea of DSA is straightforward: in each layer’s attention computation, the current token is not required to attend to every historical token. Instead, the layer dynamically selects the 2048 most relevant tokens from the historical sequence to participate in the attention calculation.

This design has clear engineering value in long-horizon agent scenarios. As context grows, the computational complexity of standard full attention rises quadratically, and the memory consumed by the KV cache climbs linearly. Sparse attention limits the number of keys and values each query token actually attends to, reducing per-layer computation from a function of the full sequence length to a function of a fixed window size. More importantly, DSA is not a simple sliding window. A sliding window keeps only the most recent fixed number of tokens, whereas DSA’s relevance-based selection can retrieve the information most relevant to the current reasoning step from much earlier in the history — something that is critical for long-horizon tasks with cross-step dependencies.

Consider an agent task where the model calls a tool in step 3 and obtains a key parameter, but does not need to reference that parameter until step 50. A sliding-window approach would have discarded the information long ago; DSA’s selection mechanism can still retrieve it from history. This is the core advantage of sparse attention over window attention, and an important technical foundation for this version’s progress on long-horizon agent capabilities.

Joint Reinforcement Learning Post-Training Across Hundreds of Domains

Pretraining and basic instruction tuning alone rarely prepare a model for complex, multi-step agent tasks that depend on environmental feedback. One of the key improvements in this release is a large-scale Joint Reinforcement Learning (Joint RL) post-training stage covering hundreds of downstream domains.

Supervised fine-tuning is constrained by the scale and diversity of high-quality labeled data. Reinforcement learning, by contrast, lets the model learn policies autonomously through reward signals while interacting with the environment. In a code-repair task, the model must understand code structure, locate the defect, generate a fix, and verify whether the fix works. In a terminal operation task, the model must decide the next action based on the current interface state, observe the result, and continue deciding. These tasks are naturally closed loops of “action and feedback,” which makes them well suited to an RL training framework.

The “joint” in Joint RL means the training data does not come from a single task type. It mixes code development, software engineering, 3D design, web retrieval, financial research, office document processing, and more. This cross-domain mixing helps the model learn more general reasoning patterns and tool-use strategies instead of overfitting to one category of tasks. Training also used MXFP4 weight quantization, which reduces memory footprint and communication overhead during training while preserving model accuracy, making it engineering-feasible to train a 2.8T-parameter model.

Tool Calling and Agent Framework Integration

With open weights, the model can be deployed through mainstream inference engines and exposes API interfaces compatible with OpenAI and Anthropic. Developers can therefore switch existing agent systems built on general-purpose chat interfaces over to this model for testing and use with relatively little friction.

The tool-calling protocol follows the common industry pattern: developers define the names, descriptions, and parameter schemas of available tools; the model issues tool call requests as needed during inference; the system executes the tools and returns results; and the model continues reasoning based on those results. One practical point deserves attention: in multi-turn tool-calling scenarios, you must retain the complete history of thinking and tool-call messages in the request. Because the model supports a 1-million-token context, this requirement is satisfiable in engineering terms, but you should avoid simple truncation or coarse-grained summarization of history.

The model offers three thinking-budget settings: low, high, and max. This lets developers trade off response speed against reasoning depth according to the scenario. In interactive Q&A, the low tier reduces response latency; in complex code repair or multi-step decision tasks, the max tier provides more thorough reasoning steps. The thinking budget essentially controls how deeply the model performs “internal reasoning” before producing its final answer. For agent scenarios, a higher thinking budget usually means more reliable planning and more accurate tool selection.

Benchmark Results and How to Interpret Them

The officially published evaluation results cover several benchmarks closely tied to long-horizon agent capability:

• Terminal operation benchmark: 65.7
• Software engineering marathon tasks: 51.7
• Deep software engineering tasks: 75.5
• Comprehensive browsing and retrieval evaluation: 92.1
• Computer-aided design (CAD) tasks with tools: 92.7
• Office document question answering: 64.4

All results were measured with the thinking budget set to max, and some tests used an intelligent coding tool as the agent framework.

Several caveats apply when interpreting these numbers. First, the benchmarks differ in maximum score, difficulty, and scoring criteria, so a single absolute number cannot be compared across benchmarks; that said, the model overall performs notably well on tasks that involve tool calling. Second, results on agent tasks depend heavily on the execution framework and tool environment used — the same model may score differently under different frameworks. These figures are therefore better treated as a capability reference than as an absolute guarantee of deployment outcomes.

Deployment Practices and Engineering Considerations

From a deployment perspective, the model ships as MXFP4-quantized weights and runs on multiple mainstream inference engines, with the TokenSpeed acceleration solution also supported. Before deployment, enterprise users and developers should focus on evaluating the following areas.

Memory and compute planning. Although only 104B parameters are activated per token, the 2.8T total means that fully loading the weights requires substantial storage and GPU memory. MXFP4 quantization significantly reduces the storage footprint of the weights, but the KV cache at inference time remains a non-trivial memory cost, especially for ultra-long contexts approaching 1 million tokens. Plan your hardware based on your actual context-length requirements, and make full use of techniques such as paged KV caching offered by your inference framework.

License compliance. The model weights are released under a dedicated license — not a permissive open-source license such as Apache 2.0 or MIT. Before commercial deployment, enterprise users should carefully review the license terms, confirm that their use case falls within the permitted scope, and avoid compliance risks.

Service integration. Because the model exposes an API compatible with common chat protocols, developers can wrap it as a standard chat endpoint for upper-layer agent frameworks. When building the service, introduce a monitoring and alerting system to continuously track metrics such as inference latency, token generation rate, and tool-call success rate.

A System-Level View of Long-Horizon Agent Design

However capable a single model is, it must be embedded in a complete agent system to deliver value. A typical long-horizon agent system contains the following layers: the model layer, which handles reasoning and decision-making; the tool layer, which supplies external capabilities such as code execution, web retrieval, and file operations; the memory layer, which maintains long-term state and accumulated experience; and the orchestration layer, which manages multi-step workflows and recovers from exceptions.

The model described in this article provides the foundational capability at the model layer for long-horizon tasks: the 1-million-token context window reduces the design complexity of the memory layer; DSA sparse attention keeps inference efficient across ultra-long sequences; and the reasoning ability instilled by Joint RL makes the model better at multi-step planning and tool coordination. That said, the model does not replace system design. Developers still need to design an appropriate tool set for their business scenario, choose suitable thinking budgets, and build reliable error-handling and retry mechanisms.

When integrating with Baidu AI Cloud or other cloud infrastructure, developers can use general cloud products such as container platforms, logging services, and monitoring and alerting to build an observable, scalable agent inference service — for example, deploying the inference service in containers, collecting tool-call chains through the logging service, and tracking inference latency and error rates in real time through the monitoring system. Combined with the model’s own reasoning capability, these general-purpose capabilities help enterprises move long-horizon agents from experiment to production faster.