MoE Sparse Attention and Long-Horizon Agent Architecture Explained

1. Background: New Architectural Demands from Long-Horizon Agents

As agent applications evolve from single-turn Q&A toward multi-step task orchestration, models must maintain stable context understanding across high-frequency tool calls, long-document retrieval, and multi-turn reasoning chains. Traditional dense attention suffers from quadratic compute growth as context length increases, while long-horizon agent tasks often involve hundreds of thousands to millions of tokens of conversation history, tool outputs, and intermediate reasoning states. This poses three challenges for model architecture:

• Compute efficiency: How to control attention compute overhead in ultra-long contexts;
• Information fidelity: How to compress the attention scope without losing critical historical information;
• Multimodal fusion: How to unify heterogeneous inputs such as text and images within a single framework.

A recently open-sourced large-scale MoE model (2.8T total parameters, 104B activated parameters) offers a systematic engineering answer to these problems. Its core approach can be summarized as: replacing some dense layers with sparse attention, covering the downstream task distribution with joint reinforcement learning, and lowering the deployment barrier with low-precision quantization.

2. Architecture Design: Sparse Attention and Mixture-of-Experts in Concert

2.1 MoE Foundation

The model adopts a standard Mixture-of-Experts (MoE) architecture with 2.8T total parameters, activating only about 104B parameters per forward pass. This design keeps single-inference compute cost acceptable while preserving very large capacity. The core of MoE is expert routing via the gating network—each token is assigned to Top-K experts, and the remaining experts do not participate in computation.

2.2 Sparse Attention Mechanism

Compared with the previous generation, the new version makes a key substitution at the attention layer: 24 layers of Gated MLA (Multi-head Latent Attention) are replaced with Gated DSA (Dynamic Sparse Attention), used in combination with 69 layers of Kimi Delta Attention. The core mechanism of DSA is: each attention computation selects only 2048 historical tokens to participate, rather than performing dense attention over the full context.

The engineering significance of this design:

• Controllable memory footprint: The retrieval scope of the KV Cache drops from O(n) to O(k), where k is a fixed selection window;
• Preserved long-range dependencies: A gating mechanism dynamically selects the historical segments most relevant to the current token, avoiding the information fragmentation of fixed sliding windows;
• Complementarity with Delta Attention: Delta Attention excels at capturing local incremental changes, while DSA handles global sparse retrieval; together they form multi-scale attention coverage.

For developers, understanding this mechanism helps in setting context windows and retrieval strategies appropriately in real deployments. For example, when building a long-horizon code agent, the full repository index can serve as an external retrieval source, while the model’s internal DSA locates key modification records within the conversation history.

2.3 Native Multimodal Input

The model supports native image-text input, meaning the vision encoder and the language backbone are aligned during pretraining rather than stitched together afterward. This matters for agent tasks involving UI screenshots, CAD drawings, and scanned documents—the model can directly understand the spatial relationships of interface elements without relying on an external OCR pipeline.

3. Training Strategy: Joint Reinforcement Learning and Low-Precision Quantization

3.1 Joint Reinforcement Learning Across Hundreds of Downstream Domains

The model underwent large-scale joint reinforcement learning post-training on top of K3. “Joint” here means the training signal covers hundreds of downstream domains simultaneously, including terminal operations, code repair, web retrieval, financial research, and office document processing. The advantages of this strategy:

• Task generalization: The model does not overfit to a single task distribution but shares reasoning capabilities across tasks;
• Tool-calling consistency: Multi-turn tool calling requires retaining complete thinking and tool-call messages; joint training teaches the model to maintain state consistency when switching between tools;
• Controllable reasoning budget: The model offers three thinking budget levels—low, high, and max—so developers can set them dynamically based on task complexity to balance latency and quality.

3.2 MXFP4 Weight Quantization

Weights are quantized in MXFP4 format, a 4-bit floating-point micro-scaling format that preserves model performance at very low precision. The impact on deployment:

• Significantly lower memory requirements: At 4-bit quantization, 2.8T parameters require roughly 1.4TB of weight storage, achievable via multi-GPU or layered loading;
• Inference framework compatibility: Supports the quantized loading paths of mainstream inference engines, so developers do not need to implement quantization operators themselves;
• Controlled accuracy loss: Official evaluations show that benchmark results after quantization are within an acceptable range of the full-precision version.

4. Capability Evaluation: From Terminal Operations to CAD Generation

Official benchmark results cover the main application scenarios of long-horizon agents. Below are the results at the default max thinking level:

Benchmark	Score	Scenario
Terminal-Bench 4.0	65.7	Terminal operations and environment interaction
SWE Marathon v1.1	51.7	Long-horizon software engineering tasks
DeepSWE v1.1	75.5	Deep code repair
BrowseComp	92.1	Web retrieval and information extraction
BenchCAD (with tools)	92.7	CAD generation and 3D design
OfficeQA Pro v2	64.4	Office document understanding and processing

The score distribution shows that the model performs strongly in tool-augmented scenarios (BenchCAD with tools 92.7, BrowseComp 92.1), indicating solid tool-calling and external information integration capabilities. Pure terminal operations (Terminal-Bench 65.7) and long-horizon software engineering (SWE Marathon 51.7) still leave room for improvement—consistent with the general difficulty distribution of current long-horizon agent tasks.

Some evaluations used an intelligent coding tool as the agent framework, meaning the model needs to work with external frameworks in practice rather than completing all tasks independently. When selecting a model, developers should consider the adaptation cost between the model and their own agent framework.

5. Deployment and Integration: API Compatibility and Multi-Turn Tool Calling

5.1 Inference Engine Support

The model releases MXFP4 weights and supports mainstream inference engines such as vLLM, SGLang, and TokenSpeed. It also provides OpenAI/Anthropic-compatible APIs, which means:

• Existing applications built on the OpenAI interface can migrate at low cost;
• Standard interface semantics such as streaming output and function calling are supported;
• Multi-turn tool calling requires retaining complete historical thinking and tool-call messages—this is key to correct long-horizon agent operation, and developers must explicitly maintain these fields in conversation management.

5.2 License Notes

The weights use a dedicated license, not Apache 2.0 or MIT. Enterprises should carefully read the license terms before commercial use to confirm the scope of use, distribution conditions, and requirements for derivative works.

6. Practical Recommendations for Developers

1. Context management: Although the model supports a 1M-token context, DSA selects only 2048 historical tokens each time. Implement an external memory module in the agent framework to persist key information in a vector database or structured storage rather than relying entirely on the model’s internal attention.
2. Thinking budget tuning: Use the low level for simple tasks to reduce latency and the max level for complex reasoning tasks. A/B testing can determine the best level for each task type.
3. Quantized deployment planning: Confirm that the inference engine version supports MXFP4 quantization, and plan memory and bandwidth resources in advance.
4. Multi-turn tool-calling state maintenance: Ensure that historical thinking and tool-call results are passed in full with each request to avoid state loss that causes abnormal agent behavior.

7. Conclusion

Through a combination of sparse attention, joint reinforcement learning, and low-precision quantization, this model offers a practical technical solution for long-horizon agent scenarios. Its core value lies not in leading any single metric, but in integrating very large parameter capacity, million-level context, and tool-calling capabilities into a unified inference framework. For teams building complex agent systems, understanding its architectural principles and deployment constraints matters more than focusing solely on benchmark scores. As sparse attention mechanisms mature and quantization techniques become widespread, the engineering barrier for long-horizon agents is expected to decrease further.