An Open-Source LLM Release Adds 2.8T MoE Architecture for Long-Horizon Agent Tasks

1. Release Overview: From General Chat to Long-Horizon Task Execution

A recent open-source large language model release has made its latest model weights publicly available. Built on large-scale joint reinforcement learning post-training from the previous generation, this version shifts its core focus from general conversational ability to long-horizon agent task execution, covering code development, software engineering, 3D design, knowledge work, and other professional domains.

From an architectural standpoint, the new version uses a Mixture of Experts (MoE) design with 2.8T total parameters and 104B activated parameters. It supports a 1M-token context window and native image-text input. This gives the model a significant capacity advantage when processing very long documents, multi-turn tool-calling chains, and large code repository understanding.

For developers, the most notable change in this release is not simply the increase in parameter scale, but the systematic optimization of long-horizon reasoning stability and tool-calling reliability. The following sections examine the release from four angles: architecture evolution, capability matrix, evaluation results, and engineering practices.

2. Architecture Evolution: Sparse Attention and Gating Mechanism Upgrades

2.1 From Gated MLA to Gated DSA

The new version makes a key replacement in its attention mechanism: the previous 24 layers of Gated MLA (Multi-head Latent Attention) are replaced by Gated DSA (Dynamic Sparse Attention), used alongside 69 layers of Kimi Delta Attention.

The core design of DSA is dynamic sparse selection. During each attention computation, only 2048 historical tokens are selected for attention calculation instead of the full context. This design preserves long-context understanding while substantially reducing computational overhead. For contexts approaching the 1M-token level, the engineering value of sparse attention is especially clear: it makes long-range dependency modeling feasible in terms of inference cost.

2.2 Activation Efficiency in the MoE Architecture

The configuration of 2.8T total parameters with 104B activated parameters continues the MoE design philosophy of large capacity with low activation. Each token activates only part of the expert network, maintaining model capacity while controlling per-inference computational load. Combined with MXFP4 weight quantization, the model further reduces memory usage and inference latency during deployment.

2.3 Training Strategy: Joint Reinforcement Learning

At the training level, the new version adopts a joint reinforcement learning scheme covering hundreds of downstream domains. The advantage of this multi-domain joint training is that the model does not overfit to any single task, but instead learns to transfer reasoning strategies across different tasks. For agent scenarios, this means the model can combine tool calls, plan steps, and handle exceptions more flexibly.

3. Capability Matrix: Three Thinking Budgets and Multi-Scenario Coverage

3.1 Thinking Budget Control

The new version provides three thinking budget levels: low, high, and max. Developers can set the budget dynamically based on task complexity:

• Low: suitable for simple Q&A, format conversion, and other low-latency scenarios.
• High: suitable for moderately complex code completion and document summarization.
• Max: suitable for long-horizon agent tasks, multi-step tool calls, and complex reasoning chains.

This tiered design allows developers to make explicit tradeoffs between latency and quality rather than accepting a fixed inference cost.

3.2 Code Development and 3D Creation

For code agent scenarios, the model covers terminal operations, code repair, and CAD generation. In official evaluations, it scores 65.7 on Terminal-Bench 4.0, 51.7 on SWE Marathon v1.1, and 75.5 on DeepSWE v1.1. For CAD generation tasks, it scores 92.7 with tool-based evaluation, indicating strong tool coordination capability in structured design tasks.

3.3 Agent and Knowledge Work

In knowledge work scenarios, the model supports web retrieval, financial research, office document processing, and tool calling. It scores 92.1 on BrowseComp and 64.4 on OfficeQA Pro v2. These capabilities allow the model to serve as the reasoning core of an agent system, completing end-to-end workflows from information retrieval to document generation.

4. Engineering Practices: Deployment and Tool-Calling Considerations

4.1 Quantization and Inference Frameworks

The model provides MXFP4 weights, supports mainstream inference frameworks, and offers OpenAI/Anthropic-compatible APIs. MXFP4 quantization significantly reduces memory requirements while preserving model quality, making single-node multi-GPU deployment feasible.

4.2 Key Details for Multi-Turn Tool Calling

In multi-turn tool-calling scenarios, the full reasoning history and tool-calling messages must be preserved. This is because the model’s reasoning chain and tool-calling results have a strong dependency relationship, and truncating history can cause subsequent reasoning to lose context. It is recommended to explicitly maintain a message queue in the agent framework to ensure that every call carries the complete interaction history.

4.3 License Notes

It should be noted that the model weights in this release use a custom license, not a permissive license such as Apache 2.0 or MIT. Review the license terms carefully before commercial use to confirm compliance.

5. Developer Recommendations

For teams planning to use this model in long-horizon agent scenarios, the following dimensions are worth evaluating:

1. Context management: Although the 1M-token window is large, effective use requires reasonable retrieval and compression strategies.
2. Thinking budget tuning: Choose the appropriate level based on task type to avoid cost escalation from overusing the max budget.
3. Tool-calling protocol: Design clear tool descriptions and error-handling mechanisms to improve agent robustness.
4. Evaluation baseline: Establish baselines for your own tasks using the official benchmarks as a reference to facilitate iterative comparison.

Overall, this release makes clear progress in three directions: architectural sparsification, tiered reasoning budgets, and systematic agent capabilities. It provides an open-source option worth evaluating for long-horizon task scenarios.