Large-Scale Sparse Model Open-Sourced: Architecture and Reinforcement Learning Practices for Long-Horizon Agents

1. From Dense Models to Sparse Expert Architecture

Large language models face two core tensions in long-horizon agent scenarios: attention computation costs rise rapidly as context length grows, and a single inference pass must simultaneously handle code generation, tool calling, document analysis, and multi-step decision-making across highly dispersed capability dimensions.

The open-sourced model uses a Mixture-of-Experts (MoE) architecture with 2.8T total parameters and roughly 104B activated parameters per token. The significance of this design is that model capacity can continue to expand to cover more domain knowledge, while inference-time computation does not scale linearly with total parameter count. For multi-turn agent tasks that must maintain million-level context, decoupling activated parameters from total parameters is a key prerequisite for controlling latency and cost.

In terms of engineering implementation, the model combines gated attention layers with sparse attention layers. Some layers handle coarse-grained modeling of the global sequence, while others select only a small subset of historical tokens to participate in attention computation at each inference step. Compared with conventional dense attention, this approach significantly reduces memory access and compute pressure in long-sequence scenarios.

2. Sparse Attention and Long-Context Modeling

The core difficulty of long-horizon agents is not “how many tokens can be read in,” but “whether relevant information can be accurately recalled from a long history at any given moment.” Although fully dense attention can theoretically access the entire history, when context reaches hundreds of thousands or even millions of tokens, its computational cost often forces engineering teams to truncate or summarize, which in turn loses information.

The model introduces sparse attention mechanisms in some layers. At each attention computation step, it dynamically selects 2,048 tokens from the history to participate in computation, working in concert with another type of attention layer to form hierarchical modeling capability: some layers maintain a stable representation of sequence structure, while others perform retrieval-style extraction of key segments from long history. This combination avoids the risk of context loss that a single sparsification strategy can introduce.

For developers, this means that when building agents, you should not assume the model will “fully read” all historical messages. More robust practices include:

• Place key constraints, user goals, and environment state in the system prompt or high-priority context;
• Periodically write checkpoint summaries during long tasks;
• Move fields from tool return results that are strongly relevant to the current step to the front, reducing the difficulty of retrieval across long history;
• Avoid accumulating large amounts of intermediate output in historical messages that has already been superseded by subsequent operations.

The presence of sparse attention further reinforces the necessity of these engineering practices: although the model supports million-token context, access to historical information is not indiscriminate full access, and reasonable context organization directly determines task success rates.

3. Joint Reinforcement Learning Across Multiple Domains

During training, the model adopts a joint reinforcement learning strategy covering hundreds of downstream domains. Unlike isolated optimization on a single task, joint training enables the model to share the same set of decision-making and reflection capabilities across code, reasoning, tool calling, document processing, financial analysis, and other scenarios.

The advantage of multi-domain joint reinforcement learning lies in stability. Long-horizon agent tasks are typically composed of multiple subtasks in sequence—for example, first retrieving information, then reading documents, then generating code, and finally revising the plan based on execution results. If the model is only reinforced on code generation while tool calling and error recovery capabilities are not optimized in sync, the overall task chain can easily break at some intermediate link.

On the inference budget side, the model offers three reasoning intensity levels: low, medium, and high. The low level suits simple information extraction, format conversion, and short code generation; the medium level suits scenarios requiring multi-step reasoning but not complex tool chains; the high level targets long-horizon planning, complex debugging, and agent tasks requiring repeated verification. In practice, routing can be performed at the request level based on task complexity, thereby controlling overall cost without sacrificing effectiveness on critical tasks.

It is worth emphasizing that higher reasoning levels are not always better. Excessive reasoning can cause agents to produce redundant steps on simple tasks and also introduces additional latency. A reasonable approach is to set the low level for high-frequency but simple intent recognition and structured tasks, and concentrate computational budget on task planning, error recovery, and result verification.

4. Quantization, Deployment, and Compatible Interfaces

The open weights use the MXFP4 quantization format. Compared with common integer quantization, floating-point quantization retains more information in the mantissa representation, making it better suited to MoE models with wide weight distribution ranges. For a model with 2.8T total parameters, quantization is a necessary means of lowering the VRAM threshold and improving inference throughput.

On the deployment side, the model can run on mainstream distributed inference frameworks. Because MoE architectures are naturally suited to tensor parallelism and expert parallelism, deployers can flexibly adjust expert distribution strategies based on cluster topology, VRAM capacity, and request concurrency. For scenarios that must serve multiple agent sessions simultaneously, the following dimensions deserve attention:

• Throughput differences between long-sequence prefill and incremental decoding;
• The impact of continuously growing historical messages on KV cache during multi-turn tool calling;
• Index construction and cache reuse strategies for sparse attention layers;
• The balance between request-level reasoning-level routing and batching efficiency.

At the interface level, the model provides calling methods compatible with mainstream chat protocols. In agent scenarios, multi-turn tool calling must preserve the complete history of reasoning and tool-call messages. That is, you cannot use only the final answer as context for the next turn; you should retain the reasoning process generated at each step, tool request parameters, and tool return data. Only in this way can the model correctly determine which subgoals have been completed and which failures need retrying in subsequent steps.

For production systems that need to control costs, a history compression strategy can be introduced: under the premise of preserving key tool-call boundaries and result summaries, perform lossy compression on earlier verbose reasoning content. However, the compression boundary should be set on the premise of not breaking task-state consistency. Especially for strongly stateful information such as file paths, variable naming, and environment configuration, over-summarization that leads the model to misjudge should be avoided.

5. Agent Capability Evaluation and Result Interpretation

The officially published evaluation results cover terminal operations, software engineering, deep code understanding, browsing and retrieval, assisted design, and office Q&A, among other directions. Numerically, the model performs strongly on tasks requiring long-horizon planning and tool composition, while there is still room for improvement on some procedural Q&A tasks.

Three points should be noted when interpreting these evaluations. First, some tasks are not completed by the model alone but combine an agent framework for tool scheduling and execution. This means the scores reflect the combined capability of the model and toolchain working together, not isolated reasoning ability. Second, most results default to the highest reasoning level, which means that in actual deployment with lower reasoning levels, results may differ from the leaderboard. Third, different evaluations have different difficulty distributions, context lengths, and failure penalty mechanisms, so a single score should not be directly compared across tasks.

For enterprise users, a more reasonable evaluation approach is to build a private task set closely aligned with their own business. Typical real production problems can be sampled, covering the following dimensions:

• The ratio of short-horizon to long-horizon tasks;
• Whether external retrieval or internal knowledge bases are relied upon;
• Tool-call chain length and failure rates;
• Whether multi-turn history contains large amounts of noise;
• Requirements for output format and factual consistency.

Through a private evaluation set, a clearer feedback loop can be established among model selection, reasoning-level choice, and prompt engineering, avoiding deployment decisions based solely on public leaderboards.

6. Licensing and Engineering Practice Recommendations

The open weights use a model-specific license rather than a common permissive open-source license. Before using it in commercial products or internal systems, enterprises need to evaluate license boundaries in light of their own usage patterns. In particular, when the model is embedded as an agent capability foundation in customer-facing products, the scope of the license, redistribution restrictions, and compliance obligations should be reviewed as separate matters.

From an engineering implementation perspective, the stability of long-horizon agents depends not only on model capability but also on surrounding engineering infrastructure. It is recommended to build supporting infrastructure in the following areas:

• Context lifecycle management: clarify which history should be retained and which can be removed or summarized;
• Tool-call state machine: design standard recovery paths for tool failures, timeouts, and abnormal return structures;
• Reasoning budget routing: dynamically select reasoning levels based on task type to avoid runaway costs;
• Cache strategy under multi-session concurrency: make full use of shared prefixes and repeated tool definitions;
• Result evaluation and regression mechanisms: perform regression validation with private task sets after every change to the model, prompts, or toolchain.

Large-scale sparse models provide long-horizon agents with more ample context windows and stronger backbone capabilities, but what ultimately determines production effectiveness is still the depth of coordination between model capability and engineering practice. Only when task design, history management, tool calling, and reasoning budget form a complete closed loop can the model’s long-context and multi-step reasoning advantages be translated into stable, usable business capabilities.