Open-Source 2.8T MoE Model Released: Long-Horizon Agent Capabilities and Sparse Attention Architecture Explained

1. Background: What Long-Horizon Agents Demand from Model Architecture

Over the past year, agent applications have rapidly evolved from “single-turn Q&A plus a single tool call” to “multi-step planning, long tool chains, and cross-session memory.” These tasks impose hard requirements on the underlying model across three dimensions:

1. Context length: A single software engineering task may involve dozens of files and hundreds of rounds of tool-call records, easily pushing context beyond several hundred thousand tokens.
2. Adjustable reasoning budget: Simple retrieval and complex code repair differ enormously in required thinking depth. A fixed reasoning overhead either wastes compute or underperforms on difficult tasks.
3. Sparsification efficiency: The longer the context, the more unacceptable the quadratic complexity of standard attention becomes. Sparse attention must be introduced at the architectural level.

A recently open-weighted large-scale MoE model was iterated precisely around these three directions. Built on a previous-generation architecture with large-scale joint reinforcement learning post-training, it adopts a mixture-of-experts structure with 2.8T total parameters and 104B activated parameters, supporting 1M-token context and native image-text input. This article analyzes it across four dimensions: architecture, training, evaluation, and deployment.

2. Architecture: The Sparsification Path from Gated MLA to Gated DSA

2.1 Overall Structure

The model continues the MoE (Mixture of Experts) approach, with 2.8T total parameters and 104B activated parameters. This means that in each token’s forward computation, only about 3.7% of parameters are activated, keeping per-token compute cost acceptable while maintaining extremely large capacity for knowledge storage.

Compared with the previous generation, the core architectural change centers on the attention layers:

• Previous scheme: 24 layers of Gated MLA (Multi-head Latent Attention) + 69 layers of Kimi Delta Attention
• Current scheme: The 24 layers of Gated MLA are replaced with Gated DSA (Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention

2.2 How DSA Works

The core idea of DSA is to dynamically select historical tokens to participate in attention computation. Specifically:

• At each attention computation, DSA selects 2,048 tokens from the historical context to participate;
• The selection process is driven by a gating mechanism, dynamically determined by the relevance between the current Query and historical Keys;
• Tokens not selected do not participate in attention computation, reducing the computational complexity of long contexts from quadratic to approximately linear.

This design differs from the industry-common “fixed sliding window + global tokens” approach: a fixed window cannot adapt to changes in task structure, whereas dynamic selection allows the model to focus on relevant function definitions when processing code files, or on key passages when processing long documents.

2.3 Why Kimi Delta Attention Is Retained

The 69 layers of Kimi Delta Attention handle another portion of attention responsibility. The core advantage of Delta Attention lies in its friendliness to incremental updates—when context is appended in a streaming fashion, Delta-style attention can efficiently update state without recomputing the entire history. This complements DSA:

• DSA is responsible for “precisely selecting relevant information from history”;
• Delta Attention is responsible for “efficiently maintaining streaming state.”

Together, they allow the model to maintain usable inference speed at 1M-token context.

3. Training Strategy: Joint Reinforcement Learning and Quantization

3.1 Joint RL Covering Hundreds of Downstream Domains

The post-training adopts joint reinforcement learning, covering hundreds of downstream domains. “Joint” here has two layers of meaning:

1. Task jointness: Code repair, terminal operations, CAD generation, web retrieval, financial research, office document processing, and other tasks are optimized together in a single training pipeline, avoiding the capability fragmentation that results from separately fine-tuning each task.
2. Reward jointness: Reward signals from different tasks are normalized and used jointly, enabling the model to learn how to allocate reasoning budget across tasks.

This training approach is particularly important for agent scenarios: real tasks are often combinations of multiple capabilities—for example, “retrieve web pages → extract data → write to spreadsheet → generate report”—and single-task fine-tuning struggles to cover the complete chain.

3.2 MXFP4 Weight Quantization

Weights are quantized in MXFP4 format. MXFP4 is a 4-bit floating-point quantization format used with microscaling factors, significantly reducing VRAM usage while maintaining numerical stability. For a model with 2.8T total parameters, quantization is key to deployment feasibility:

• Stored in BF16, 2.8T parameters require approximately 5.6TB of VRAM;
• Stored in MXFP4, the theoretical VRAM requirement drops to approximately 1.4TB, making multi-GPU cluster deployment possible.

It should be emphasized that quantization applies to weight storage, while activation computation still occurs at higher precision to ensure inference quality.

4. Capability Coverage: Three Core Scenarios

4.1 Code Development and 3D Creation

Targeting long-horizon code agents, software engineering, and 3D design tasks, official evaluations cover terminal operations, code repair, and CAD generation. Common characteristics of these tasks include:

• Multi-step: A single task may involve dozens of tool calls;
• Strong state dependency: Later steps depend on the execution results of earlier steps;
• Tool integration required: The model must actively invoke external capabilities such as terminals, file systems, and CAD tools.

4.2 Agent and Knowledge Work

Supports web retrieval, financial research, office document processing, and tool calling. These tasks also place high demands on context length: a research report may be dozens of pages, a single retrieval may return multiple web pages, and the model needs to track task objectives within long contexts.

4.3 Three Thinking Budget Tiers

Inference offers low, high, and max thinking budget tiers, configurable per task. This is a highly practical engineering feature:

Tier	Use Case	Characteristics
low	Simple retrieval, format conversion	Low latency, low token consumption
high	Routine code repair, document processing	Balanced latency and quality
max	Complex software engineering, CAD generation	Quality-first, highest consumption

Developers can dynamically switch tiers within an agent framework based on task type, achieving a balance between overall cost and effectiveness.

5. Official Benchmark Performance

The officially published benchmark results are as follows (max thinking tier by default, with some evaluations using an agent framework):

• Terminal-Bench 4.0: 65.7
• SWE Marathon v1.1: 51.7
• DeepSWE v1.1: 75.5
• BrowseComp: 92.1
• BenchCAD (with tools): 92.7
• OfficeQA Pro v2: 64.4

From the task distribution, the evaluations cover five directions: terminal operations, software engineering, web retrieval, CAD generation, and office Q&A, broadly corresponding to the capability scenarios described above. It should be noted that benchmark results are heavily influenced by evaluation frameworks, prompts, and tool sets; actual deployment performance must be verified against specific tasks.

6. Deployment and Engineering Practices

6.1 Weight Format and Inference Frameworks

The released weights are in MXFP4 format, supported by mainstream inference frameworks (such as vLLM, SGLang, and TokenSpeed-class solutions), with OpenAI/Anthropic-compatible APIs provided. This means:

• Existing applications built on compatible APIs can migrate at relatively low cost;
• Multi-turn tool calling must retain the complete history of thinking and tool-call messages, otherwise the model loses context state.

6.2 Notes on Multi-Turn Tool Calling

In agent scenarios, a common mistake is to retain only the final answer while discarding intermediate thinking processes. For this model, the correct approach is:

messages = [
    {"role": "user", "content": "Fix the null pointer exception in this repository"},
    {"role": "assistant", "content": "...", "tool_calls": [...]},
    {"role": "tool", "content": "..."},
    {"role": "assistant", "content": "..."},
    # Subsequent turns must fully retain the above history
]

Discarding intermediate messages causes the model to repeat already-completed steps in subsequent turns, or to fail to correctly understand the current state.

6.3 License Information

The weights use a proprietary license, not Apache 2.0 or MIT. Enterprises should carefully review the license terms before commercial use to confirm the scope of use, distribution conditions, and restrictions.

7. Summary

The core iteration directions of this open-sourced 2.8T MoE model can be summarized in three points:

1. Architectural sparsification: Replacing part of Gated MLA with Gated DSA, selecting only 2,048 historical tokens per attention computation, combined with Kimi Delta Attention to achieve efficient inference at million-token context.
2. Training jointness: Joint reinforcement learning covering hundreds of downstream domains, maintaining capability consistency across multi-step agent tasks.
3. Deployment practicality: MXFP4 quantization lowers the VRAM threshold, three thinking budget tiers provide a cost-adjustment lever, and compatible APIs reduce migration costs.

For teams building long-horizon agent applications, this model offers an option worth evaluating. It is recommended to start from specific tasks: first verify the upper bound of effectiveness at the max tier, then gradually lower the tier to find the balance between cost and quality, while paying careful attention to fully retaining historical messages in multi-turn tool calling.