A 2.8T-Parameter MoE Model Goes Open Source with a 1M-Token Context Window for Long-Horizon Agent Workloads
A newly open-sourced large language model combines a 2.8T-parameter Mixture-of-Experts architecture, native 1M-token context, and joint reinforcement learning post-training to support long-horizon coding, CAD, and knowledge-work agents. This article covers its core capabilities, sparse attention design, benchmark results, and deployment considerations.
A New Open-Source Model Foundation for Long-Horizon Agents and Complex Tasks
A next-generation large language model has officially released its model weights, marking a notable step forward for complex task handling and long-horizon agent applications. Built on a Mixture-of-Experts (MoE) architecture with 2.8T total parameters and 104B activated parameters, the model balances capability and inference cost through large-scale joint reinforcement learning post-training. Its most distinctive features are native support for a 1M-token context window and built-in multimodal image-text processing, providing a foundation for very large code repositories, long-document analysis, and multimodal design tasks.
In current large model development, massive parameter counts and long context windows alone do not solve real-world complexity. The harder challenge is whether a model can maintain reasoning coherence, instruction-following accuracy, and reliable tool use across interactions spanning tens or even hundreds of thousands of tokens. This open-source release targets that specific pain point through changes in both architecture and training strategy.
Core Focus Areas: Code Development and 3D Creation
The model positions code development and 3D creation as primary application scenarios. In software engineering, a long-horizon coding agent must understand project structure, cross-file dependencies, and commit history before it can effectively fix defects or iterate on features.
The model shows strong results across several specialized evaluations:
- Terminal operations: Beyond generating code snippets, the model can interpret runtime environments, logs, and error stacks, then execute commands independently, analyze outcomes, and adjust its strategy.
- Code repair and refactoring: When working with complex multi-file codebases, the model can locate root causes and generate patches without disrupting existing architecture, covering a pipeline from static analysis to dynamic verification.
- CAD generation: In 3D design, the model can understand design constraints and coordinate with design tools to complete multi-step tasks from parametric modeling to structural optimization. This extends generative capabilities beyond text or simple images into professional engineering software.
These capabilities come not only from model scale but also from training on code structure, version-control logic, and tool-calling protocols, giving the model a degree of engineering intuition.
Agent and Knowledge Work: From Retrieval to Decision-Making
Beyond vertical domains, the model supports general knowledge work and agent applications. In financial research, deep web search, and office document processing, it can plan task steps, call external search tools and data-processing utilities, and fuse multi-source heterogeneous information.
The official inference design introduces a Reasoning Budget with three adjustable levels: low, high, and max. This gives developers flexibility. For simple summarization or format conversion, low reduces latency and cost. For complex mathematical proofs, multi-step logical reasoning, or long-range planning, max guides the model into deeper internal reasoning during the thinking phase, producing more rigorous intermediate reasoning chains and improving final decision accuracy. This explicit and adjustable reasoning mechanism is a key design element for practical usability.
Technical Architecture: Sparse Attention and Reinforcement Learning
Compared with its predecessor, the core architectural change replaces the original 24-layer Gated MLA mechanism with Gated DSA (Deep Sparse Attention), working alongside the remaining 69 layers of Kimi Delta Attention. The main intent is to reduce computational bottlenecks in long-context scenarios.
In conventional dense attention, every token attends to all historical tokens. At 1M tokens, computational complexity grows quadratically, creating large memory and latency costs. With Gated DSA, each attention layer dynamically selects only 2,048 historical tokens for computation. This sparsification is not simple window truncation. A gating mechanism learns which historical information matters more, preserving key information while sharply reducing computation. As a result, per-token cost on very long sequences drops significantly, making long-horizon agent workloads more commercially viable.
For training, the model uses a joint reinforcement learning scheme covering hundreds of downstream domains. Compared with models relying mainly on supervised fine-tuning, this approach better aligns performance across diverse tasks and improves reward-signal optimization in long-chain decision-making. At deployment, MXFP4 weight quantization compresses model weights to roughly one quarter of their original size while maintaining accuracy, reducing memory overhead and lowering deployment barriers.
Benchmark Results: Capability Boundaries in Numbers
Published benchmarks cover terminal interaction, code generation, and financial knowledge tasks:
| Evaluation Suite | Score | Focus Area |
|---|---|---|
| Terminal-Bench 4.0 | 65.7 | Terminal operations and interaction |
| SWE Marathon v1.1 | 51.7 | Cross-file software engineering |
| DeepSWE v1.1 | 75.5 | Deep software engineering benchmark |
| BrowseComp | 92.1 | Web browsing and information retrieval |
| BenchCAD with tools | 92.7 | 3D CAD design and tool use |
| OfficeQA Pro v2 | 64.4 | Multimodal office document understanding |
These results use the max reasoning budget by default, and some evaluation suites rely on external agent frameworks, such as common code-agent scaffolds, to enhance execution on complex tasks. This suggests that real-world performance depends not only on model parameters but also on the external scaffolding around the model.
Deployment and Ecosystem Integration
The model is adapted for mainstream inference frameworks such as vLLM and SGLang, with API styles compatible with common formats. This reduces migration costs. Developers can use an OpenAI-compatible interface for quick replacement in existing applications, or an Anthropic-compatible interface for tool-calling patterns.
One development guideline deserves attention: in multi-turn tool-calling interactions, the system requires preserving the full history of reasoning steps and tool-call messages. The reasoning chain, including thought and observation steps, must be passed back completely as context for subsequent decisions. Truncating or cleaning historical messages at the application layer can cause the model to lose execution context and destabilize long-horizon planning.
For model weights, the official release uses a custom Kimi K3.1 Pro license rather than a permissive license such as Apache 2.0 or MIT. Organizations should review the license terms carefully before commercial use or private deployment to avoid compliance risks.
Conclusion
This open-source release represents more than a jump in parameter scale. It is an important practical step in combining long-horizon task execution, multimodal understanding, and tool-use coordination. Sparse attention addresses computational bottlenecks, joint reinforcement learning improves reasoning quality, and adjustable reasoning budgets make costs more controllable. Together, these design choices offer a valuable technical pattern for building reliable agent applications. For engineering teams, the 1M-token context window and agent capabilities open up possibilities in code generation, financial analysis, and intelligent hardware interaction that were previously difficult to realize.
