Open-Source Community Releases 2.8T-Parameter MoE Model with Breakthrough Long-Horizon Agent Capabilities

1. Background: New Architectural Demands Posed by Long-Horizon Agent Tasks

As agent technology moves from laboratory settings into production environments, the core challenge facing developers is no longer the quality of single-turn conversations, but rather model stability and consistency across long-horizon tasks. A long-horizon task typically requires dozens or even hundreds of tool calls, spans multiple rounds of reasoning loops, and demands precise preservation of intermediate state — examples include terminal automation, bug fixing in large code repositories, multi-step CAD modeling, and cross-document knowledge integration.

These tasks impose three hard requirements on the underlying model:

1. Ultra-long context windows: Every step an agent takes, every tool return, and every reasoning trace must be retained in context. Otherwise, the model “forgets” the basis for earlier decisions.
2. Efficient attention computation: The computational cost of attention scales quadratically with context length. Without sparsification, inference costs become unacceptable.
3. Controllable reasoning depth: Different tasks demand vastly different levels of reasoning depth. Simple retrieval and complex code repair should not consume the same computational budget.

Recently, an open-source community released the weights of a large model designed specifically for long-horizon agent scenarios. Built on a previous-generation architecture with large-scale joint reinforcement learning post-training, the model uses a Mixture-of-Experts (MoE) design with 2.8T total parameters and 104B activated parameters, supports a 1-million-token context window, and natively accepts image and text inputs. This article analyzes the model from four dimensions: architecture design, training strategy, evaluation performance, and engineering deployment.

2. Architecture Analysis: The Evolution of Sparse Attention Mechanisms

2.1 Basic MoE Architecture Design

The model adopts an MoE architecture with 2.8T total parameters, but activates only 104B parameters per forward pass. The core advantage of this design is that it uses larger total capacity to hold broader knowledge while using sparse activation to control per-inference cost. For agent scenarios, this means the model can simultaneously support code generation, document understanding, and tool-use decision-making without these capabilities interfering with each other through parameter sharing.

The key to MoE lies in the routing mechanism — how to select the most suitable experts for each token. Common industry approaches include Top-K routing, expert capacity limiting, and load-balancing losses. The specific routing strategy for this model has not been fully disclosed, but given its activation ratio of roughly 3.7%, it falls into the highly sparse category.

2.2 From Gated MLA to Gated DSA

Compared with the previous version, the new model makes an important substitution in its attention mechanism: replacing 24 layers of Gated MLA (Gated Multi-head Latent Attention) with Gated DSA (Gated Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention.

Two key concepts need clarification:

• MLA (Multi-head Latent Attention): Reduces KV Cache memory consumption through low-rank compression, a common optimization for long-context scenarios.
• DSA (Dynamic Sparse Attention): Selectively includes only a subset of historical tokens in attention computation each step.

The model’s DSA mechanism selects 2,048 historical tokens for attention computation at each step. This means that even with a 1-million-token context window, the actual computation per attention operation is constrained to roughly 2,048 tokens. This design significantly reduces computational overhead while preserving the most critical historical information through dynamic selection.

2.3 Engineering Implications of Sparse Attention

From an engineering perspective, the value of sparse attention manifests in three ways:

• Memory consumption: KV Cache no longer grows linearly with context to unmanageable levels.
• Inference latency: Attention computation drops from O(n²) to approximately O(n×k), where k is the number of selected tokens.
• Long-horizon consistency: By retaining key historical tokens, the model can still reference early decisions during long-running tasks.

3. Training Strategy: Joint Reinforcement Learning and Quantization

3.1 Joint Reinforcement Learning Across Hundreds of Downstream Domains

During post-training, the model uses joint reinforcement learning covering hundreds of downstream domains. “Joint” here means optimization is performed across multiple domains simultaneously rather than through single-task RLHF, giving the model cross-domain generalization capability.

This training strategy is particularly important for agent scenarios because a real agent task often spans multiple domains. For example, “fixing a bug in a web application” may involve code understanding, terminal operations, log analysis, and document retrieval. A model trained with reinforcement learning in only a single domain would struggle to maintain stability across such cross-domain tasks.

3.2 MXFP4 Weight Quantization

The model’s open weights are quantized in MXFP4 format. MXFP4 is a 4-bit floating-point quantization format belonging to the MX (Microscaling) family of standards. Compared with traditional INT4 quantization, MXFP4 is more robust to outliers while preserving accuracy.

The direct benefit of quantization is significantly reduced deployment cost: a 2.8T-parameter model stored in FP16 would require approximately 5.6TB of memory, while MXFP4 quantization compresses this to around 1.4TB, making deployment on mid-sized clusters feasible.

4. Capability Coverage: From Code Development to Knowledge Work

4.1 Code Development and 3D Creation

The model is specifically optimized for long-horizon coding agents, software engineering, and 3D design tasks. Official evaluations cover terminal operations, code repair, and CAD generation scenarios, with support for multi-step tasks involving tool use.

Using code repair as an example, a typical long-horizon task chain might include:

1. Read issue description → 2. Locate relevant code files → 3. Analyze dependencies
→ 4. Generate a fix plan → 5. Apply changes → 6. Run tests → 7. Iterate based on failures

Each step requires the model to retain memory of previous steps and make decisions on that basis. This is precisely where long-context capability and sparse attention mechanisms deliver value.

4.2 Agent and Knowledge Work

For agent and knowledge work scenarios, the model supports web retrieval, financial research, office document processing, and tool calling. Inference provides three reasoning budget levels — low, high, and max — allowing flexible configuration based on task complexity.

The three-tier reasoning budget design reflects the principle of matching inference cost to task difficulty:

• low: Suitable for simple retrieval and format conversion tasks with fast response and low cost.
• high: Suitable for analytical tasks requiring multi-step reasoning.
• max: Suitable for complex code repair, multi-tool coordination, and other long-horizon tasks.

5. Evaluation Results and Deployment Practices

5.1 Official Benchmark Data

The officially released evaluation results are as follows (using the max reasoning tier by default, with some benchmarks using an intelligent coding tool framework):

• Terminal-Bench 4.0: 65.7
• SWE Marathon v1.1: 51.7
• DeepSWE v1.1: 75.5
• BrowseComp: 92.1
• BenchCAD (with tools): 92.7
• OfficeQA Pro v2: 64.4

These data cover terminal operations, code repair, web retrieval, CAD generation, and office document Q&A, reflecting the model’s comprehensive capabilities in long-horizon agent scenarios.

5.2 Deployment and API Compatibility

The model’s MXFP4 weights are openly available and support deployment on mainstream inference frameworks. OpenAI/Anthropic-compatible APIs are also provided, making it easier for developers to migrate existing agent systems.

One important note: multi-turn tool calling must preserve complete historical reasoning traces and tool call messages. This is because agent decision-making depends on the full context chain. If historical messages are dropped between turns, the model loses awareness of task state.

A typical multi-turn call message structure looks like this:

[
  {"role": "user", "content": "Fix this bug"},
  {"role": "assistant", "content": "I need to check the logs first", "tool_calls": [...]},
  {"role": "tool", "content": "Log contents..."},
  {"role": "assistant", "content": "Based on the logs, the issue is in...", "tool_calls": [...]},
  {"role": "tool", "content": "Test results..."}
]

5.3 License Note

The weights are released under a model-specific license, not Apache 2.0 or MIT. Developers should carefully review the license terms before commercial use to confirm the scope of permitted use and compliance requirements.

6. Summary and Outlook

The release of this 2.8T-parameter MoE model signals that long-horizon agent capabilities are evolving from “usable” to “reliable.” Several technical directions worth watching include:

• Dynamic selection strategies for sparse attention: How to maximize information retention within a 2,048-token budget is a key area for future optimization.
• Domain coverage in joint reinforcement learning: Avoiding negative transfer when training jointly across hundreds of domains remains an engineering challenge.
• The balance between quantization and deployment: MXFP4’s trade-off between precision and cost provides a reference for large-scale model deployment.

For developers, a practical starting path would be:

1. Validate the basic tool-calling chain at the low reasoning tier.
2. Gradually introduce multi-turn history retention and observe consistency on long-horizon tasks.
3. Dynamically switch reasoning budgets based on task complexity to control inference costs.
4. Complete license compliance review and stress testing before production deployment.

Long-horizon agent deployment is not a matter of model capability alone, but a systems engineering problem involving models, frameworks, toolchains, and task design. The release of this model provides a new foundation worth studying for this systems engineering effort.