New Progress in Large-Scale MoE Open Source: 2.8T Parameters and Long-Horizon Agent Capabilities

The open-source LLM ecosystem is shifting from chasing individual benchmark scores to competing on systems engineering capability. On November 4, a new model version built on a previous-generation architecture and post-trained with large-scale joint reinforcement learning officially released its weights. The version adopts a MoE (Mixture of Experts) architecture with 2.8T total parameters and 104B activated parameters, supports 1M-token context and native image-text input, and explicitly targets long-horizon agents as its core scenario. This article analyzes the model across four dimensions—architecture modifications, training strategy, evaluation performance, and deployment practice—to provide a reference for developers planning to run it in their own environments.

1. Core Specifications: Balancing Capacity and Cost Under a Sparse Architecture

This version continues the MoE approach: total parameters reach 2.8T, but only about 104B parameters are activated per forward pass. The engineering significance of this design is that model capacity—knowledge storage and representation capability—is determined by total parameters, while inference compute cost is mainly determined by activated parameters. For long-context scenarios, controlling activated parameters is especially critical: with a dense architecture of equivalent total parameter count, inference cost at 1M-token context would be unsustainable.

Key specifications can be summarized as follows:

• Total / activated parameters: 2.8T / 104B, MoE sparse architecture;
• Context length: supports 1M tokens, targeting long documents, large code repositories, and multi-turn agent trajectories;
• Input modalities: native image-text input, with no need for an external vision encoder adapter layer;
• Quantization format: MXFP4 weights released, reducing VRAM usage and deployment barriers;
• License: a version-specific license rather than permissive terms such as Apache 2.0 or MIT; commercial use requires careful review of the terms.

It is worth emphasizing that a 1M-token context does not mean “if it fits, it works.” Attention computation complexity at long context, KV cache VRAM usage, and effective information recall rate together determine practical usability. This is precisely why the version makes structural modifications to the attention mechanism.

2. Sparse Attention Redesign: From Gated MLA to Gated DSA

Compared with the previous-generation K3, the most notable architectural change is in the attention layers: 24 layers of Gated MLA (Multi-head Latent Attention) are replaced with Gated DSA (Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention.

How DSA works deserves elaboration: at each attention computation, it selects 2,048 tokens from the historical context to participate, rather than performing full attention over all historical tokens. This “dynamic sparse selection” mechanism yields two benefits:

1. Controllable computation complexity: at the 1M-token context level, full attention computation cost grows rapidly with sequence length, whereas sparse selection with a fixed budget constrains per-step attention cost within a manageable range;
2. Reduced VRAM pressure: with fewer tokens participating in computation, KV cache read and compute pressure decreases accordingly, improving the feasibility of consecutive multi-turn tool calls in long-horizon tasks.

From an engineering perspective, the essence of such modifications is “trading compute for a selection strategy.” The quality of the selection strategy directly determines information recall at long context—if key historical information is not selected, the model degrades into “seeing the length but not the content.” Therefore, sparse attention must be paired with joint optimization during training to perform reliably in real tasks.

3. Training Strategy: Joint Reinforcement Learning and Quantization

This version is post-trained with joint reinforcement learning covering hundreds of downstream domains. Unlike single-task RLHF, joint reinforcement learning emphasizes simultaneous optimization across multiple domains (code, terminal operation, retrieval, document processing, design tasks, etc.), enabling the model to maintain policy consistency in multi-step, cross-tool tasks.

This choice aligns closely with the long-horizon agent positioning. The core difficulty of long-horizon tasks is not single-step accuracy but error accumulation: one wrong tool-call parameter or one misread retrieval result can cause subsequent steps to drift from the goal. Joint reinforcement learning improves the model’s error-correction and replanning ability over long chains by introducing multi-domain, multi-step trajectory feedback during training.

Meanwhile, weights use MXFP4 quantization. MXFP4 is a low-bit floating-point quantization format whose value lies in significantly reducing weight VRAM usage, making a model with 2.8T total parameters deployable on reasonable hardware configurations. Note that quantization introduces precision loss, so deployment should evaluate it against task type—numerically sensitive tasks warrant comparative validation.

4. Capability Coverage: Code Agents, 3D Creation, and Knowledge Work

The official capability positioning for this version falls into two main directions.

Code development and 3D creation: targeting long-horizon code agents, software engineering, and 3D design tasks. Official evaluations cover terminal operation, code repair, and CAD generation, supporting multi-step tasks with tools. A common trait of these tasks is “environment interaction + long-chain execution”—for example, locating a build failure cause in the terminal, modifying code, re-validating, or generating geometric structures under constraints in a CAD scenario.

Agents and knowledge work: supporting web retrieval, financial research, office document processing, and tool calling. On the inference side, three thinking budgets are provided—low, high, and max—which can be set dynamically based on task complexity. This design has practical value for cost control: simple tasks can use the low tier to reduce latency and token consumption, while complex multi-step tasks can use the max tier in exchange for greater reasoning depth.

5. Official Benchmark Data and Interpretation Boundaries

Official benchmark results use the max thinking tier by default, and some evaluations use external agent frameworks. The data is as follows:

• Terminal-Bench 4.0: 65.7
• SWE Marathon v1.1: 51.7
• DeepSWE v1.1: 75.5
• BrowseComp: 92.1
• BenchCAD (with tools): 92.7
• OfficeQA Pro v2: 64.4

Three points deserve attention when interpreting this data: first, benchmark scores are strongly correlated with the thinking tier, and results at the max tier do not represent performance at default settings; second, some tasks depend on agent frameworks and toolchains, and framework differences affect final results; third, different benchmarks have different task distributions and difficulty levels, so directly comparing numbers across benchmarks is of limited value. Readers are advised to run small-scale validation on their own business data rather than making model-selection decisions directly from public leaderboards.

6. Deployment and Integration Practices

This version releases MXFP4 weights, supports mainstream inference frameworks (vLLM, SGLang, and TokenSpeed), and provides OpenAI/Anthropic-compatible APIs. For teams with existing inference infrastructure, compatible APIs mean lower integration and modification cost.

One easily overlooked detail in engineering practice: multi-turn tool calling must retain the full history of thinking and tool-call messages. A long-horizon agent trajectory typically contains a loop of “think—tool call—tool return—think again.” If intermediate messages are truncated or dropped in context management, the model loses accurate perception of task state, leading to repeated calls or logical breaks. The following is an illustrative structure for message retention:

messages = [
    {"role": "user", "content": task},
    # Retain thinking and tool-call records for every turn
    {"role": "assistant", "content": thought_1, "tool_calls": [call_1]},
    {"role": "tool", "content": result_1},
    {"role": "assistant", "content": thought_2, "tool_calls": [call_2]},
    {"role": "tool", "content": result_2},
]

In addition, KV cache management at 1M-token context requires supporting strategies: properly configuring session isolation, storing historical trajectories in a structured manner, and performing summarization-based compression instead of direct truncation when necessary are all prerequisites for stable long-horizon agent operation.

7. License and Model-Selection Recommendations

This version uses a version-specific license, not Apache 2.0 or MIT. This means users must review the license terms clause by clause for scenarios such as commercial use, redistribution, and derivative model training, and cannot assume permissive open-source conventions apply.

Overall, the technical value of this version centers on three points: the sparse attention redesign makes million-token context engineering-feasible; joint reinforcement learning improves policy stability in long-chain tasks; and MXFP4 quantization plus compatible APIs lower the deployment barrier. For teams building code agents, terminal automation, or document processing pipelines, it is a candidate worth validating in their own environments—but model-selection decisions should still be based on measured business data, not public leaderboards.