Large Sparse MoE Model Goes Open Source: A Technical Breakdown of Long-Horizon Agent Capabilities

1. Background: Why Long-Horizon Agents Need a New Model Architecture

Over the past year, agent applications have evolved rapidly from single-turn Q&A to multi-step autonomous execution. A typical long-horizon task may involve dozens of tool calls, cross-file code edits, web retrieval, and cross-validation of results. Such tasks impose three hard requirements on the underlying model:

1. Sufficiently long context: It must accommodate the full conversation history, tool return results, code repository snippets, and intermediate reasoning traces;
2. Adjustable reasoning budget: Simple tasks should not waste compute, while complex tasks need deeper thinking;
3. Stable tool calling: Historical reasoning and tool messages must be preserved across multiple rounds; otherwise, broken state leads to task failure.

A recently open-sourced ultra-large sparse MoE model is designed around these requirements. Its core specifications are: 2.8T total parameters, 104B activated parameters, support for 1 million tokens of context, and native image-text input. This article provides a technical breakdown across four dimensions: architecture, training, evaluation, and deployment.

2. Architecture Analysis: A Hybrid Design of Sparse Attention and Linear Attention

2.1 MoE Skeleton: Separating Total Parameters from Activated Parameters

The model adopts a MoE (Mixture of Experts) architecture with 2.8T total parameters, but only 104B parameters are activated per forward pass. The engineering significance of this design is: memory capacity determines whether the model can be loaded, while activated parameters determine inference speed. The total parameter scale supports knowledge capacity, and the activated parameter scale controls per-token computation. By decoupling the two, the model can achieve large knowledge reserves while maintaining relatively high throughput.

2.2 Attention Layer Replacement: From Gated MLA to Gated DSA

Compared with the previous generation, the new model makes a key replacement in the attention structure: replacing 24 layers of Gated MLA with Gated DSA (Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention.

The working mechanism of DSA can be summarized as follows: for each attention computation, 2048 historical tokens are selected to participate, rather than performing dense attention over the entire history. This yields two direct benefits:

• The attention computation cost under long context drops significantly from O(n²);
• The “selection” mechanism preserves the historical information most relevant to the current query, avoiding information dilution in long contexts.

The Kimi Delta Attention layers play the role of linear attention, responsible for maintaining long-range information flow at low cost. The two are stacked alternately, forming a hybrid attention scheme of “sparse selection + linear retention.”

Input sequence
   │
   ├─ Gated DSA layers (24 layers): dynamically select 2048 historical tokens for sparse attention
   │
   ├─ Kimi Delta Attention layers (69 layers): linear attention maintains long-range state
   │
   └─ Output to MoE feed-forward layers

2.3 Thinking Budget Tiers: low / high / max

The model offers three tiers of reasoning thinking budget. This design is particularly important for agent scenarios:

• low: suitable for simple tool calls and format conversion;
• high: suitable for multi-step reasoning and code repair;
• max: suitable for complex planning, CAD generation, and deep research.

Developers can dynamically set the budget based on task complexity, avoiding the cost waste of “running full budget on every request.”

3. Training Strategy: Joint Reinforcement Learning and Low-Precision Quantization

3.1 Joint Reinforcement Learning Covering Hundreds of Downstream Domains

The model undergoes large-scale joint reinforcement learning post-training based on the previous generation. “Joint” here means the training objective covers hundreds of downstream domains rather than fine-tuning on a single task. The value of this strategy lies in:

• Avoiding overfitting on any single task category;
• Enabling agent capabilities (tool calling, multi-step planning) to improve alongside domain knowledge;
• Forming capability transfer across heterogeneous tasks such as code, office work, finance, and design.

3.2 MXFP4 Weight Quantization

Weights are released in MXFP4 quantized format. MXFP4 is a low-precision floating-point format that significantly reduces memory usage while maintaining numerical stability. For a model with 2.8T total parameters, quantization is a prerequisite for deployment on small-to-medium-scale clusters. Developers need to confirm the level of support for this format in their inference frameworks.

4. Capability Evaluation: Interpreting Official Benchmarks

The officially published evaluation results cover multiple dimensions, using the max thinking tier by default, with some evaluations using intelligent coding tools as the agent framework:

Benchmark	Score	Capability Focus
Terminal-Bench 4.0	65.7	Terminal operations
SWE Marathon v1.1	51.7	Long-horizon software engineering
DeepSWE v1.1	75.5	Code repair
BrowseComp	92.1	Web retrieval
BenchCAD (with tools)	92.7	3D/CAD generation
OfficeQA Pro v2	64.4	Office document processing

It is important to view these objectively: benchmark scores reflect performance under specific evaluation conditions. Real-world business results also depend on prompt design, toolchain quality, and context management strategy. This is especially true for long-horizon tasks, where differences in evaluation frameworks can significantly affect results.

5. Deployment and Integration: Engineering Best Practices

5.1 Inference Framework Support

The model releases MXFP4 weights and supports integration with mainstream inference frameworks through compatible APIs. The official offering includes OpenAI/Anthropic-compatible APIs, meaning existing applications built on these two interface types can migrate at relatively low cost.

5.2 History Management for Multi-Turn Tool Calling

This is an easily overlooked but critically important engineering detail: multi-turn tool calling must preserve the complete thinking messages and tool-call messages. The reason is that in subsequent rounds, the model relies on the prior reasoning trajectory to determine the next action. If developers truncate historical thinking to save context, the model may:

• Repeatedly call the same tool;
• Lose the task objective;
• Pass incorrect parameters.

The recommended approach is to maintain a structured message queue that distinguishes five message types: system, user, assistant thinking, tool call, and tool return. When approaching the context limit, use summarization compression rather than outright discarding.

5.3 License Considerations

The weights are released under a dedicated license, not Apache 2.0 or MIT. Enterprises should carefully review the license terms before commercial use to confirm scope of use, distribution conditions, and compliance requirements.

6. Suitable Scenarios and Selection Advice

Given the model’s capabilities, the following scenarios are a good fit:

• Long-horizon code agents: cross-file edits, terminal operations, multi-step repair;
• Software engineering automation: combining tools to complete the chain from requirements to commit;
• 3D design and CAD generation: geometry generation tasks with tools;
• Knowledge work: web retrieval, financial research, office document processing.

When selecting, consider three points: first, the maturity of inference framework support for MXFP4 and sparse attention; second, the strategy for balancing thinking budget and cost; third, the state management approach for multi-turn tool calling. For scenarios with extremely high context requirements, the 1 million token window combined with sparse attention is a core advantage, but actual latency performance should also be evaluated.

7. Summary

The open-sourcing of this ultra-large sparse MoE model provides a new foundation option for long-horizon agent applications. Its technical highlights center on three areas: long-context efficiency brought by the hybrid of Gated DSA and linear attention, inference cost flexibility brought by three thinking budget tiers, and task generalization brought by joint reinforcement learning covering hundreds of domains. For engineering deployment, history message management, quantization format compatibility, and license compliance are three areas that must be confirmed in advance.