Large-Scale Sparse Model Open Sourcing: A Technical Analysis for Long-Horizon Agents and Multi-Step Tool Calling

1. Background: From Single-Turn QA to Long-Horizon Agents

Historically, the evaluation of large language models has primarily focused on single-turn QA, text comprehension, and code completion. These tasks share common characteristics: relatively closed inputs, shorter contexts, and no need for the model to make continuous decisions across multiple tools.

However, with the evolution of agent applications, demands are shifting. A typical long-horizon agent task might involve reading multiple documents, retrieving external information, writing and executing code, revising approaches based on execution results, maintaining task state, and finally generating an outcome. This scenario introduces new requirements for models:

• Support for sufficiently long contexts to accommodate historical thoughts, tool call logs, and intermediate results.
• Stable reasoning capabilities to maintain goal consistency across long chains.
• Support for tool calling protocols, enabling the model to interact with environments rather than just outputting text.
• Adjustable inference tiers to balance reasoning cost and quality.

The design goals of this newly open-sourced large-scale sparse model directly target these needs. It adopts a Mixture of Experts (MoE) architecture, controlling the activated parameters at each inference step despite a large total parameter count, effectively balancing capability and computational cost.

2. Overall Architecture: Large-Scale Sparse Activation and Long Context

The model features an MoE architecture with approximately 2.8T total parameters and around 104B activated parameters per inference. The core concept of an MoE model is setting up multiple expert networks in each layer (or every few layers) while only activating a subset during each pass. This expands model capacity without the linear increase in computational overhead associated with activated parameters.

Equally noteworthy is the context length. The model supports a context window of approximately 1 million tokens, meaning it can handle:

• The complete execution trajectory of a long task.
• Multi-turn tool calls and their returned results.
• Continuous segments of long documents or code repositories.
• Historical thoughts generated during multi-step reasoning.

Long context is not simply about “inputting more content.” In agent scenarios, context also serves as “working memory.” If a model loses early information in a long chain, it may repeat completed actions or ignore critical constraints in subsequent steps. Therefore, long-context capabilities must be paired with attention mechanisms and positional encoding designs to form genuinely usable long-horizon working memory.

3. Sparse Attention: Key to Reducing Long-Sequence Computation Costs

The direct challenge of long contexts is the increased computational cost of attention. Traditional dense attention sees significant computational growth as sequence length increases. To control this cost, the model introduces a sparse attention design.

Specifically, the model replaces the attention mechanism in certain layers with a gated sparse attention format. During each computation, the model selects a specific number of key historical tokens to participate in the attention calculation, rather than performing full computation on all historical tokens. For example, it might select around 2048 historical tokens to interact with current information.

The basis for this selection is learned through a gating mechanism. The model determines which historical tokens are more critical to the current step, enabling “selective recall.” Simultaneously, another type of attention layer is retained within the model, allowing information to pass and fuse across different levels.

Sparse attention is well-suited for long-horizon agents because agent tasks typically exhibit distinct locality and phasing. For instance, in a code repair task, the model’s current focus might be the error message returned by the most recent tool call, rather than all the details from hundreds of steps ago. Yet, certain early constraints still need to be remembered. Through the “selective recall” mechanism, sparse attention seeks to balance memory retention and computational cost.

4. Inference Tiers: Allocating Thinking Budgets Based on Task Complexity

The model offers low, medium, and high inference budget tiers. This design addresses a practical issue: vast differences in task complexity.

For simple tasks, the model can quickly provide results without lengthy reasoning chains. For complex multi-step planning tasks, the model requires more reasoning steps to decompose goals, evaluate intermediate results, and adjust strategies.

The tiered inference mechanism allows developers to set thinking budgets based on task types:

• Low tier: Suitable for simple queries, lightweight classification, and short text generation, offering faster response times.
• Medium tier: Ideal for general code generation, document processing, and information extraction tasks.
• High tier: Designed for complex agent tasks, multi-step tool calling, and reasoning tasks requiring repeated verification. It delivers higher quality but takes more time.

In practical engineering, this tiered design can be integrated with agent frameworks. For example, a high tier can be used during the task planning phase, while a low tier can be applied during simple tool parameter population, achieving cost control across the overall task.

5. Tool Calling and Long-Horizon Agent Practices

Tool calling is a core capability of agents. The model needs to output structured tool calling requests—such as specifying an API to call and passing parameters—then receive the tool’s returned results and continue reasoning based on them.

In long-horizon agent tasks, the design of the tool calling protocol is particularly crucial. A common issue is managing historical information across multi-turn tool calls. If only the most recent call’s result is retained, the model may lose early constraints; if all history is retained, context length grows rapidly.

A viable approach is to retain the complete history of thoughts and tool call messages. The benefits of this include:

• The model can review previous decision bases.
• When a tool returns an exception, the source of the problem can be traced.
• If a task branches, the model can backtrack or re-plan based on history.

Naturally, this imposes higher requirements on the deployment environment. The inference service must support long-context caching and incremental computation; otherwise, multi-turn calls will incur repeated computational overhead. Common optimization directions include reusing the Key-Value (KV) cache of historical tokens and saving the state of completed steps to reduce redundant processing in subsequent steps.

A typical code repair agent workflow can be described as follows:

1. Receive task description and code repository information;
2. Planning: Locate potentially faulty modules;
3. Tool Call: Read relevant file contents;
4. Analysis: Determine the cause of the error based on file contents;
5. Tool Call: Run tests or static checks;
6. Adjust the repair plan based on tool returned results;
7. Generate a patch and verify it;
8. Output the final repair result.

In this workflow, each step may depend on information from several previous steps. The model must maintain an understanding of the task goal across a long chain and continuously correct its behavior based on tool returned results.

6. Multi-Domain Joint Reinforcement Learning

During the post-training phase, the model employs joint reinforcement learning (RL) covering multiple downstream domains. Compared to single-domain RL, the significance of joint training lies in enhancing the model’s generalization capabilities.

If a model is only fine-tuned with RL for a specific class of tasks, it might perform well on those tasks but easily suffer capability degradation when switched to new scenarios. Joint RL mixes data from different domains, allowing the model to learn more general reasoning and decision-making patterns rather than memorizing templates for specific tasks.

For example, code repair, document processing, information retrieval, and 3D design differ vastly on the surface, but they share underlying capabilities: understanding task goals, decomposing steps, calling tools, verifying results, and adjusting plans. Joint training strengthens these common capabilities.

It is worth noting that the effectiveness of RL highly depends on the design of reward signals. For agent tasks, reward signals may come from:

• Whether the final result is correct.
• Whether intermediate steps are reasonable.
• Whether tools were successfully called.
• Whether the task was completed within a limited number of steps.
• Whether the generated content meets formatting and constraint requirements.

Designing stable and scalable reward mechanisms remains a core engineering challenge in training long-horizon agents.

7. Quantization and Deployment: MXFP4 Weights and Inference Framework Compatibility

The model releases weights in the MXFP4 format. FP4 is a low-bit-width floating-point representation. Compared to common FP16 or BF16, it offers significant improvements in model size and inference speed, though it demands higher numerical precision.

The key concept of MXFP4 is “block-wise scaling.” For a block of values, a shared scaling factor is used to maintain the dynamic range post-quantization. Compared to element-wise quantization, this method better preserves model precision at low bit widths.

For large-scale sparse models, quantization has another significance: reducing storage and transmission costs. A model with hundreds of billions of parameters requires immense hardware resources if stored in full precision. Post-quantization, the weight volume drops significantly, enabling more teams to perform local deployment or research under limited hardware conditions.

Regarding inference frameworks, the model supports common open-source inference engines and provides APIs compatible with mainstream interface styles. This lowers the barrier to entry for developers and allows existing agent applications to migrate relatively smoothly.

In actual deployment, attention should also be paid to:

• Load balancing for sparse experts, preventing some experts from being overloaded while others sit idle.
• KV cache management for long contexts.
• Batching efficiency in multi-user concurrent scenarios.
• Consistency between quantized decoding and full-precision decoding.
• Serialization and deserialization of tool call messages within the API.

8. Evaluation Metrics: From Static Scores to Process Evaluation

A series of benchmark test results have been released, covering scenarios such as terminal operations, code repair, deep software engineering, browsing and retrieval, 3D design, and office document QA. A common trait of these benchmarks is that tasks usually require multi-step operations and interaction with external tools or environments.

Evaluating static QA is relatively simple because answers can be clearly judged as right or wrong. In agent tasks, however, evaluation standards are more complex:

• A task might have multiple feasible paths, with the same final result but different processes.
• Intermediate steps might fail, but the model could still complete the task through retries or adjustments.
• Tool calls might fail, but the model’s ability to recover from these failures is itself a crucial capability.
• Efficiency matters: whether the model completes the task within a reasonable number of steps.

Therefore, evaluating long-horizon agents cannot rely solely on final scores; it must incorporate process metrics, such as step count, number of tool calls, failure recovery rate, and task completion rate.

For developers, public benchmark scores serve as a reference, but it is more important to build targeted evaluation sets within their own business scenarios. Because different applications vary greatly in context distribution, tool types, and fault tolerance, general evaluations cannot fully reflect the model’s performance in specific scenarios.

9. Licensing and Engineering Considerations

The model weights use a specific license rather than a commonly permissive open-source license. This means developers must carefully read the terms before use to confirm whether commercial use, distribution, modification, or derivative development based on this model is permitted.

Proprietary licenses are not uncommon in the current model ecosystem. Key points developers need to focus on include:

• Whether commercial use is allowed.
• Whether fine-tuning or continued training based on these weights is permitted.
• Whether derivative models must be open-sourced.
• Whether there are geographical, industrial, or usage restrictions.
• Whether model outputs need to be labeled or watermarked.

From an engineering perspective, before selecting a model for a production system, license compliance should be treated as an independent phase separate from technical evaluation. Otherwise, subsequent risks may arise regarding usage scope.

10. Summary and Outlook

The open-sourcing of this large-scale sparse model provides a new technical foundation for long-horizon agent applications. The MoE architecture expands model capacity while controlling activation costs; sparse attention offers a more economical computation method for long contexts; the tiered inference mechanism allows developers to flexibly allocate computing resources based on task complexity; and multi-domain joint RL enhances the model’s adaptability across diverse tasks.

However, it should also be recognized that long-horizon agents still face many unresolved challenges:

• How to more stably maintain goal consistency in long-chain tasks.
• How to design more efficient memory and state management mechanisms.
• How to further reduce precision loss under low-bit-width quantization.
• How to establish unified and reproducible agent evaluation standards.
• How to find a better balance between cost, latency, and task completion quality.

For developers, rather than chasing single-point scores, it is more practical to start from actual business needs, build custom task chains, evaluation sets, and tool protocols, and measure whether the model genuinely solves problems through iteration. The engineering value of a large-scale model is ultimately reflected in its stability, controllability, and sustainable delivery within specific scenarios.