Open-Source Breakthrough: A 2.8T-Parameter MoE Model for Long-Horizon Agents and Complex Tasks

Model Architecture: Balancing Scale and Efficiency

The new open-source large language model adopts a Mixture of Experts (MoE) architecture, which has become a leading approach for maintaining extremely high parameter capacity while controlling inference-time computation. The model has a total of 2.8T parameters, but each forward pass activates only 104B parameters. This means the model can store a massive amount of knowledge and recognize complex patterns while running at a relatively economical computational cost.

This design philosophy — large total parameters, small activated parameters — allows the model to approach the performance ceiling of dense ultra-large models without incurring the same energy consumption and latency penalties.

The model also supports a native context window of up to 1 million tokens and accepts mixed image-text input directly. These capabilities are critical for tasks involving long documents, complete code repositories, multi-turn interaction histories, or complex multimodal data. Combined with the MoE architecture, developers can build agents that “remember” an entire project context and act on it without frequent external retrieval or context truncation.

Core Innovation: Sparse Attention and a Restructured Architecture

Compared with its predecessor, this open-source model introduces a major change to its attention mechanism. According to the technical report, the model replaces the previous 24 layers of Gated Multi-head Latent Attention (Gated MLA) with Gated Dense Sparse Attention (Gated DSA), which works alongside the remaining 69 layers of Kimi Delta attention.

The core idea behind sparse attention is dynamically selecting which tokens participate in computation. In each attention layer, the model selects 2,048 of the most critical tokens from the historical token sequence. This strategy mirrors how humans skim and highlight while reading: computational resources are concentrated on high-value information while irrelevant text segments are ignored.

Compared with global attention, this approach significantly reduces memory usage and computational complexity for long sequences. Compared with fixed-window attention, it avoids the risk of critical information being truncated outside the window. In practice, this hybrid architecture means that when the model processes a 100,000-token codebase, it can remain highly sensitive to recent changes while still remembering key dependency declarations at the top of a file.

Training Strategy: Joint Reinforcement Learning for General Capabilities

The training process is equally significant. On top of the base model, the team performed post-training with joint reinforcement learning across hundreds of downstream domains. Unlike traditional supervised fine-tuning, joint reinforcement learning optimizes multiple capabilities — code generation, mathematical reasoning, tool calling, and content understanding — under a unified reward signal.

The advantage of this approach is that it avoids the “seesaw effect,” where improving performance on one task degrades performance on another. Through cross-domain joint optimization, the model learns to allocate its “thinking budget” autonomously for complex tasks: it responds quickly to simple queries, while activating deeper reasoning chains for ambiguous engineering problems or agent tasks that require multi-step planning. The model provides three thinking budget levels — low, high, and maximum — allowing developers to specify the desired trade-off between latency and reasoning depth through the API.

Application Scenarios: From Code to 3D Design

Long-Horizon Code Agents and Software Engineering

In terminal-oriented benchmarks, the model achieves a score of 65.7, demonstrating an ability to understand terminal environments, execute commands, and adjust strategies based on output feedback. In comprehensive software engineering evaluations, the model leverages its understanding of the global state of a codebase to locate cross-file bugs and generate patches.

The key enabler is the 1-million-token context window. An agent can directly “read” the abstract syntax tree or key files of an entire repository, then use the sparse attention mechanism to identify logic chains related to a failure — without relying on vector retrieval, which can introduce semantic drift.

For developers, this means an AI assistant is no longer limited to code completion. It can take on the role of a junior engineer: understanding requirement issues, planning file modification strategies, writing test cases, and executing test commands to verify fixes.

3D Design and CAD Generation

The model achieves a score of 92.7 on CAD generation benchmarks with tool use. This performance is driven by its native image-text input capability and multi-step tool-calling logic. In a typical workflow, the model parses a hand-drawn sketch or reference image, decomposes it into geometric constraints, and then generates three-dimensional geometry step by step through parametric modeling tools. The model must strategically decide which modeling commands to call and which parameters to pass at each step, while continuously comparing preview render outputs against the original input.

Knowledge-Intensive Work and Office Automation

In benchmarks focused on knowledge work, the model shows strong multi-dimensional capabilities. It scores 92.1 on web browsing benchmarks, indicating an ability to understand complex browsing and information extraction instructions. It achieves 64.4 on office document processing tests, extracting key clauses from lengthy PDF contracts or spreadsheets and performing format conversion.

Combined with tool calling, the model can support automated financial research assistants that retrieve the latest earnings reports, extract data, generate summaries, and produce comparison charts — all without human intervention.

Technical Details and Deployment Considerations

Practical Gains from Sparse Attention

Replacing 24 layers of Gated MLA with Gated DSA is a targeted fix for the long-sequence inference bottleneck. In the MLA mechanism, full key-value caches must be maintained for all historical tokens. The DSA mechanism, by contrast, only needs to store caches for the 2,048 selected tokens. This design drastically reduces KV cache memory usage. For a 1-million-token input, the DSA mechanism can reduce KV cache memory requirements by roughly an order of magnitude, depending on token sparsity distribution, making it possible to deploy higher-precision models on a single GPU.

Inference Optimization and Quantization

To support large-scale deployment, the team provides model weights quantized in MXFP4 format. MXFP4 is a microscaling format that assigns dynamic scaling factors to each weight group, preserving much of the dynamic range of FP16 with only 4-bit precision. Quantization reduces the model size by approximately four times and is supported by mainstream inference frameworks. Testing shows that this quantization scheme significantly improves per-node concurrent throughput while keeping natural language understanding and code generation quality nearly unchanged.

API Compatibility and Calling Patterns

The model provides API interfaces compatible with industry standards, supporting both OpenAI and Anthropic calling protocols. Developers can switch services without rewriting low-level interaction code. However, in multi-turn tool-calling scenarios, developers must preserve the full history of reasoning steps and tool call records in the message body — not just the visible dialogue between the user and the assistant. The model’s reasoning chain depends on contextual understanding of previously executed actions and observations. If this process is omitted, the agent loses state-tracking ability and may make incorrect decisions.

Open Source Licensing and Compliance

It is important to note that the released weights use a custom license rather than a permissive license such as Apache 2.0 or MIT. While the model weights can be obtained free of charge for research purposes, teams planning commercial deployment or redistribution should carefully review any additional clauses in the license — for example, whether certain industries are restricted from commercial use, whether derivative models must maintain equivalent open-source obligations, or whether monthly active user thresholds apply. Development teams should treat licensing compliance as an evaluation dimension equal in importance to performance when planning production architectures.

Summary and Outlook

This open-source 2.8T-parameter model demonstrates that extremely large models can support complex agent tasks at manageable cost through the combination of MoE architecture, dynamic sparse attention, and joint reinforcement learning. Its targeted optimizations for long-horizon tasks, image-text understanding, and tool calling provide a solid foundation for building the next generation of intelligent agents.

As inference frameworks further optimize sparse computation and more mature quantization toolchains emerge, this class of models will unlock greater value in automated programming, digital twins, and intelligent decision support. For technical teams, now is an appropriate time to evaluate how well this architecture fits their business scenarios and to begin proof-of-concept testing.