Massive Sparse Model Open-Sourced: Long-Horizon Agent Reasoning Architecture and Reinforcement Learning in Practice

1. Model Positioning and Overall Capabilities

This open-weight, ultra-large sparse model adopts a mixture-of-experts (MoE) architecture, with a total parameter count of 2.8T and 104B activated parameters. Compared with dense models, MoE architectures can expand model capacity without significantly increasing per-inference compute, giving the model stronger expressive potential on tasks such as code, mathematics, knowledge question answering, long-document understanding, and tool calling.

The model supports a 1M-token context window and natively supports image-text input. For long-horizon agent tasks, context length is not an isolated metric—it directly determines whether an agent can continuously retain environment feedback, historical actions, intermediate reasoning, and code snippets across multiple tool-calling rounds. A 1M-token window means the model can handle composite work within a single task, such as cross-file code engineering, long-document review, and multi-round web retrieval with result integration.

On the inference strategy side, the model offers three tiers of thinking budget: low, medium, and high. Thinking budget here can be understood as the upper bound on compute allocated during inference, affecting how deeply the model performs internal reasoning and planning before producing a final answer. The low tier suits high-frequency, low-latency interactive scenarios; the medium tier suits routine complex tasks; and the high tier targets scenarios requiring multi-step verification, long-chain reasoning, or high-difficulty code repair. In practice, developers can choose based on task difficulty, latency requirements, and cost constraints.

2. Architecture Design: Fusing Sparse Attention with Deep Attention

In terms of attention mechanisms, the model makes significant adjustments to the prior architecture. The previous version’s 24 layers of gated multi-head latent attention have been replaced with gated dynamic sparse attention, which works in concert with 69 layers of deep attention. The core goal of this change is to improve computational efficiency and key-information recall under long contexts.

Dynamic sparse attention does not treat all historical tokens equally. Instead, at each inference step, it dynamically selects a certain number of historical tokens from the full context to participate in attention computation. In the specific implementation, 2,048 historical tokens are selected each time. The advantages of this approach are:

1. Controllable compute. As context grows, the cost of dense attention typically grows quadratically, whereas sparse attention can keep compute within an acceptable range.
2. More focused information selection. Through a gating mechanism, the model learns which historical tokens are more critical to the current reasoning step, avoiding interference from redundant information in long contexts.
3. Complementarity with deep attention. Deep attention layers capture global dependencies and long-range semantics, while sparse attention layers handle efficient retrieval and local activation.

For developers, this means that when building long-horizon agents, they can more confidently retain the full task history without frequently compressing or truncating context. That said, in actual deployment, attention still needs to be paid to GPU memory usage, caching strategies, and inference throughput.

3. Joint Reinforcement Learning and Post-Training Strategy

The model does not rely solely on knowledge acquired during pre-training; it undergoes post-training via large-scale joint reinforcement learning. Joint reinforcement learning here means optimizing not for a single task or a single reward signal, but across hundreds of downstream domains, including code generation, tool calling, mathematical reasoning, document understanding, web retrieval, and 3D design.

This training strategy yields at least three benefits:

First, it improves cross-task generalization. Because the model encounters feedback from multiple tasks simultaneously during training, the internal representations it learns must reconcile the commonalities and differences among tasks, resulting in stronger adaptability when facing unseen composite tasks.

Second, it strengthens long-horizon planning. In single-task reinforcement learning, a model easily overfits to a specific reward structure; multi-domain joint training forces the model to learn more general planning and error-correction capabilities, such as adjusting subsequent actions based on intermediate results during multi-step tool calling.

Third, it reduces reward hacking risk. Reward hacking refers to a model obtaining high reward through unintended means while actual output quality is poor. Multi-domain joint training can reduce the probability of a single reward signal being exploited through task diversity.

In long-horizon agent tasks, reward design for reinforcement learning is particularly critical. Typically, it is necessary to consider final result correctness, intermediate step soundness, tool-call format validity, and task completion efficiency simultaneously. A reasonable reward combination can guide the model to focus on both outcomes and process quality.

4. Weight Quantization and Multi-Platform Inference Support

The model weights are released in quantized MXFP4 format. MXFP4 is a quantization representation aimed at large-scale model inference. Its core idea is to allocate exponent bits and mantissa bits more flexibly, reducing GPU memory usage and memory bandwidth pressure while maintaining inference precision.

The quantized model supports multiple inference engines and deployment frameworks, and offers an API form compatible with mainstream chat interfaces. This means developers can choose an appropriate inference solution based on actual resource conditions across different hardware environments, without needing to adapt the model structure from scratch.

Note that in multi-round tool-calling scenarios, the full history of thinking and tool-call messages must be retained. The so-called full history includes not only user input and the model’s final output, but also the model’s internal reasoning process at each round, tool request parameters, tool return results, and intermediate states. If historical messages are over-compressed or truncated, the model may lose critical context in subsequent rounds and make incorrect decisions.

A common practice is to maintain a structured message list on the server side, storing each round’s role, content type, tool-call ID, and return result in order. At inference time, the full sequence is reassembled into model input. For particularly long tasks, moderate summarization can be considered under the premise of preserving semantic integrity, but the risk of information loss introduced by summarization must be evaluated.

5. Typical Task Performance and Evaluation Dimensions

The model’s performance on multiple evaluation sets reflects its design goal of serving long-horizon agents. Below, we categorize by task type to avoid merely piling up numbers while ignoring the capability implications behind them.

Terminal operation tasks mainly assess the model’s ability to complete file management, program execution, error troubleshooting, and similar operations in a command-line environment. The difficulty of such tasks lies in the fact that the model needs to understand the current environment state and dynamically adjust subsequent operations based on command execution results, rather than generating a complete script in one shot.

Software engineering marathon tasks are more comprehensive, typically involving code repository understanding, defect localization, cross-file modification, and test verification. The model needs to complete multi-step reasoning within a single task while ensuring that each step’s tool-call format is correct and input parameters are reasonable.

Deep software engineering evaluations further emphasize the complexity of real engineering tasks, such as simultaneously modifying multiple modules, handling build dependencies, or analyzing runtime logs. The MoE architecture and long-context capability offer clear advantages in such tasks.

Browsing and retrieval tasks assess the model’s ability to perform information queries, result filtering, and synthesis in an open web environment. The model needs to judge which retrieval results are trustworthy, which information is missing, and whether to conduct supplementary retrieval.

3D design and computer-aided design tasks reflect the model’s joint understanding of spatial structures, geometric constraints, and tool parameters. Such tasks place high demands on image-text input capability and further validate the model’s multimodal processing ability.

Office document and knowledge work tasks involve table processing, report generation, information extraction, and process automation. The model needs to convert between structured and unstructured information while ensuring output formats meet business requirements.

6. Engineering Practice Recommendations for Long-Horizon Agents

When actually building long-horizon agents, beyond the model’s own capabilities, the following engineering elements also require attention:

The first is context management. Although the model supports 1M tokens, not all scenarios require the full context. A reasonable context layering strategy should be designed—for example, treating task definitions, tool descriptions, and long-term goals as persistent information, and treating intermediate tool results and temporary drafts as compressible information.

The second is the tool-calling protocol. Multi-round tool calling requires strict consistency in each round’s message format, including tool name, parameter structure, call ID, and return content. It is recommended to validate the tool-call sequence on the server side to avoid task interruption caused by format errors.

The third is thinking-budget control. Different tasks vary greatly in their required reasoning depth. For example, a simple factual query may only need low-tier thinking, whereas multi-file code repair or complex data cleaning should switch to high-tier thinking. This can be dynamically adjusted via a task classifier or explicit user specification.

Finally, there is error recovery. A failure at any step in a long-horizon task can cause the entire task to fail. Checkpointing, retry, and rollback mechanisms should be designed so that the agent can recover to the most recent stable state when encountering tool exceptions, parsing failures, or reasoning deviations, rather than starting over from scratch.

Overall, the model’s open weights and architecture design provide a foundation model with high capacity, long context, and sparse attention advantages for long-horizon agents, code development, and knowledge work scenarios. In engineering practice, how to reasonably configure inference resources, manage context, and design tool protocols in line with specific business requirements is the key to determining final application effectiveness.