An Open-Source 2.8T MoE Model Release: Long-Horizon Agent Capabilities and Sparse Attention Explained
A 2.8T-parameter open-source Mixture-of-Experts model introduces dynamic sparse attention, configurable reasoning budgets, and multi-turn tool-calling message retention rules to address context bloat and consistency challenges in long-horizon agentic tasks.
An Open-Source 2.8T MoE Model Release: Long-Horizon Agent Capabilities and Sparse Attention Explained
1. Background: Why Long-Horizon Agents Have Become the New Frontier
Over the past year, the evaluation focus for large language models has shifted from single-turn question accuracy toward multi-step task completion. Whether the task is terminal operation, repository-level code repair, or cross-software 3D modeling workflows, models are now expected to maintain goal consistency, contextual coherence, and error recovery across dozens or even hundreds of tool-calling steps.
These tasks are commonly referred to as Long-Horizon Agentic Tasks, and they present three core challenges:
- Context bloat: Every tool call generates new observations, and historical messages can quickly accumulate to hundreds of thousands of tokens. With conventional full attention, compute cost grows quadratically.
- Uncontrollable inference budgets: If simple and complex tasks consume the same amount of reasoning compute, the result is either wasted capacity or insufficient quality.
- Broken tool-calling chains: If earlier reasoning steps and tool-call messages are discarded during multi-turn interactions, the model loses its decision-making basis and its behavior can drift.
An open-source Mixture-of-Experts (MoE) model has now released a 2.8T-parameter version with open MXFP4 quantized weights. This release addresses the issues above through a systematic design: replacing part of the full-attention layers with sparse attention, introducing a configurable reasoning budget, and defining explicit message retention rules for multi-turn tool calls. This article analyzes the release from four perspectives: architecture, training, evaluation, and deployment.
2. Model Architecture: Sparse Design at 2.8T Total Parameters
2.1 MoE Base Structure and Active Parameters
This version uses a Mixture-of-Experts architecture with 2.8T total parameters, but each forward pass activates only about 104B parameters. This “large capacity, small activation” approach preserves knowledge storage while keeping per-token compute cost within an acceptable range. For developers, this also means inference memory usage depends on active parameters rather than total parameters, making the deployment footprint significantly smaller than a dense model of equivalent size.
2.2 Key Attention Layer Change: From Gated MLA to Gated DSA
Compared with the previous generation, this version makes two important adjustments to the attention mechanism:
- 24 Gated MLA layers are replaced with Gated DSA (Dynamic Sparse Attention) layers
- These are used alongside 69 Kimi Delta Attention layers
The core behavior of DSA is straightforward: during each attention computation, the model selects 2048 tokens from the historical context instead of attending to all tokens. This design directly targets long-horizon agent scenarios. When context reaches the million-token level, full attention causes compute and memory usage to spike dramatically. A fixed selection budget keeps per-step attention cost relatively stable.
It is important to note that “2048” here is a selection budget, not a context window size. The context window still supports 1M tokens. DSA only determines how many historical tokens actually participate in each attention computation. The combination of a large window with sparse retrieval is becoming one of the mainstream engineering approaches for long-context models.
2.3 Native Image-Text Input and 1M Token Context
This version supports a 1M-token context window and native image-text input. Native image-text input means the model encountered interleaved image-text data during pretraining, rather than relying on an external vision encoder added later for alignment. This capability is foundational for long-horizon 3D design and software engineering agents that need to read design drawings, interface screenshots, and CAD previews.
3. Training Strategy: Joint Reinforcement Learning and Quantization
3.1 Joint RL Across Hundreds of Downstream Domains
This version is post-trained on top of the previous generation using large-scale joint reinforcement learning across hundreds of downstream domains. “Joint” here means the model is optimized across multiple task families simultaneously, rather than applying RLHF to a single task. For agent scenarios, this training approach helps the model transfer decision-making strategies across different tool sets and task types.
3.2 MXFP4 Weight Quantization
The weights are quantized in MXFP4 format. MXFP4 is a 4-bit floating-point format with micro-scaling, which preserves numerical stability while significantly reducing memory consumption. For a 2.8T-parameter model, quantization is a key prerequisite for realistic cluster deployment. Releasing MXFP4 weights also allows the community to run inference and validation on relatively limited hardware.
4. Capability Matrix: From Coding Agents to Knowledge Work
4.1 Code Development and 3D Creation
This version targets long-horizon coding agents, software engineering, and 3D design tasks. Official evaluations cover terminal operation, code repair, and CAD generation, with support for multi-step tool-assisted workflows. A typical workflow looks like this:
User goal → Environment exploration → Problem localization → Patch generation →Run verification → Rollback and retry on failure
Each step can produce substantial tool output, and the model must track where it is in the process, which attempts have failed, and what it should do next across a long context.
4.2 Agent Tasks and Knowledge Work
This version supports web retrieval, financial research, office document processing, and tool calling. Inference provides three reasoning budget tiers: low, high, and max. This is a practical engineering choice: use low for simple queries and fast responses, use max for complex analysis and deep reasoning. Developers can explicitly trade off cost against quality.
4.3 Official Benchmark Results
The following results are from official releases. The max reasoning tier is the default unless otherwise noted, and some evaluations use a general-purpose agent framework:
| Benchmark | Score |
|---|---|
| Terminal-Bench 4.0 | 65.7 |
| SWE Marathon v1.1 | 51.7 |
| DeepSWE v1.1 | 75.5 |
| BrowseComp | 92.1 |
| BenchCAD (with tools) | 92.7 |
| OfficeQA Pro v2 | 64.4 |
It is worth noting that agent evaluations are highly sensitive to framework, prompt design, and tool implementation details. When comparing across models, it is important to check whether evaluation configurations are consistent rather than relying on a single number.
5. Deployment and Engineering Considerations
5.1 Inference Framework Support
The MXFP4 weights are open, mainstream inference frameworks are supported, and OpenAI- and Anthropic-compatible APIs are provided. The value of API compatibility is that existing agent code written against these interfaces can be migrated at a lower cost without rewriting tool-calling and message orchestration logic.
5.2 Message Retention Rules for Multi-Turn Tool Calls
The official guidance is explicit: multi-turn tool calls must retain complete historical reasoning and tool-call messages. This is an easy-to-overlook but critical engineering constraint. Many agent frameworks trim historical messages after each round to save context, keeping only the final answer. In short conversations this has limited impact, but in long-horizon tasks it causes the model to lose its decision chain, which often shows up as repeatedly calling the same tool, ignoring intermediate results it has already obtained, or retrying failed paths.
The recommended context management strategy is:
- Retain: historical reasoning content, tool-call requests, and tool-returned results
- Can compress: verbose raw tool output can be summarized, but key fields should be preserved
- Can discard: chat turns unrelated to the current task
5.3 License Considerations
The weights use a version-specific license, not Apache 2.0 or MIT. Before commercial use, organizations should carefully review the license terms to confirm permitted use, distribution conditions, and compliance requirements. Do not assume the same usage rights granted by permissive open-source licenses.
6. Summary and Practical Recommendations
This 2.8T MoE release reflects three major trends in large model development:
- Sparsity: MoE controls active parameters, while dynamic sparse attention controls long-context overhead;
- Agent orientation: Models are moving from single-turn Q&A to multi-step tool calling, and evaluation benchmarks are shifting toward long-horizon tasks;
- Controllable reasoning: Reasoning budget tiers let developers allocate compute according to task complexity.
For teams planning to adopt this model, the recommended sequence is: first verify basic API calls and multi-turn tool-call message retention against the compatible API; then test the cost-quality curve across the three reasoning tiers for your specific business scenarios; finally evaluate the actual impact of sparse attention on long-context consistency in real long-horizon tasks. Only with careful engineering validation can model-level capabilities be turned into stable business results.
