MoE Large Model Long-Horizon Agent Capabilities: Sparse Attention, Thinking Budget, and Engineering Deployment

1. Background: New Architectural Requirements for Long-Horizon Agents

As the Agent paradigm shifts from single-turn question answering to multi-step autonomous execution, models need to continuously maintain long-context states, repeatedly invoke external tools, and reason over and correct intermediate results. These tasks impose challenges on the underlying model across three dimensions:

• Context length: Scenarios such as terminal operations, code repository understanding, and multi-turn retrieval naturally require context windows of hundreds of thousands or even millions of tokens;
• Inference efficiency: If every forward pass performs full attention computation, the compute and memory overhead under long contexts becomes unsustainable;
• Task generalization: Agent tasks span multiple downstream domains including code, office work, retrieval, and design, making a single optimization objective insufficient.

A recently open-sourced large-scale MoE model is specifically designed around these challenges. This article analyzes it from the perspectives of architecture, attention mechanisms, training strategies, benchmark performance, and deployment practices, helping developers understand the capability boundaries and engineering considerations of such models.

2. Overall Architecture: MoE Design with 2.8T Total Parameters and 104B Activated Parameters

The model adopts a Mixture-of-Experts (MoE) architecture with a total parameter count of 2.8T and approximately 104B parameters activated per forward pass. This design follows the core MoE principle: use total parameters to carry knowledge capacity, and use activated parameters to control per-pass compute cost.

For developers, several practical implications deserve attention:

1. Memory and weight format: The officially released weights are in MXFP4 quantized format. MXFP4 is a microscaling floating-point format that significantly compresses weight size while maintaining numerical stability, making large models easier to deploy across multiple GPUs or even multiple GPUs on a single machine.
2. Inference framework compatibility: The model supports mainstream inference engines such as vLLM, SGLang, and TokenSpeed. When selecting a framework, verify its level of support for MoE routing and the quantization format to avoid falling back to inefficient paths due to missing operators.
3. API compatibility: Both OpenAI-compatible and Anthropic-compatible APIs are provided, facilitating low-cost migration for existing Agent frameworks.

It is worth emphasizing that MoE’s “large total parameters, small activated parameters” does not automatically equal efficiency—routing strategy, expert load balancing, and communication overhead all significantly affect actual throughput. It is recommended to run stress tests based on your own request distribution before deployment.

3. Attention Mechanism: A Hybrid Approach of Gated DSA and Delta Attention

The core bottleneck for long contexts lies in attention computation. The model makes two key changes at the attention layer:

3.1 From Gated MLA to Gated DSA

Compared to the previous generation, the new version replaces 24 layers of Gated MLA with Gated DSA (Dynamic Sparse Attention). Its core mechanism is: each attention computation selects only 2,048 historical tokens to participate, rather than performing dense attention over the full context.

The significance of this design:

• Computational complexity shifts from growing quadratically with context length to being related to “number of selected tokens × current length,” yielding clear gains under long contexts;
• The sparse selection is “dynamic,” meaning the model can decide which historical segments to attend to based on the current Query content, rather than using a fixed sliding window or fixed stride;
• Combined with a gating mechanism, it can further suppress interference from irrelevant historical information on the current representation.

3.2 Combination with Kimi Delta Attention

Beyond the sparse layers, the model retains 69 layers of Kimi Delta Attention. Delta Attention-type mechanisms typically compress historical information by maintaining an incremental state, making them suitable for capturing stable long-range dependencies. The two combine to form a hybrid structure of “sparse precise retrieval + incremental state compression”:

Component	Layers	Role
Gated DSA	24 layers	Dynamically selects key historical tokens for precise modeling
Kimi Delta Attention	69 layers	Incremental state compression to maintain long-range dependencies

For Agent scenarios, this combination is particularly well-suited to the “accumulate while executing” task pattern: tool call results, historical reasoning, and error information need to be retained over the long term, but not every step requires reviewing the full history.

4. Training Strategy: Joint Reinforcement Learning and Quantization

4.1 Joint Reinforcement Learning Covering Hundreds of Downstream Domains

The model underwent large-scale joint reinforcement learning post-training on top of the base model, covering hundreds of downstream domains. “Joint” here means it is not fine-tuned for a single task, but rather optimizes reward signals from multiple task categories—code, retrieval, office work, design—in a unified manner.

The direct value for Agent capabilities is reflected in:

• Multi-step task consistency: The model is less prone to “losing track of goals mid-execution” or “format drift” during long-horizon execution;
• Tool call robustness: When facing abnormal or empty tool returns, it exhibits a certain tendency toward self-correction;
• Cross-domain transfer: Planning capabilities learned from terminal operations can partially transfer to scenarios such as document processing.

4.2 MXFP4 Weight Quantization

Introducing MXFP4 weight quantization during the training phase means quantization is not a post-deployment “patch” but is coordinated with the training process. This typically reduces accuracy loss from quantization, but also requires the inference side to have corresponding dequantization operator support.

5. Capability Coverage: From Code Agents to Knowledge Work

5.1 Code Development and 3D Creation

Targeting long-horizon code agents, software engineering, and 3D design tasks, official evaluations cover terminal operations, code repair, and CAD generation. The common characteristic of these tasks is large state space and long feedback delays: a single code fix may involve multi-file reads and writes, test execution, error localization, and secondary modifications.

In practice, it is advisable to decompose such tasks into explicit step state machines, combined with the model’s thinking budget tiers to control per-step reasoning depth.

5.2 Agents and Knowledge Work

Supports web retrieval, financial research, office document processing, and tool calling. The inference side provides three thinking budget tiers—low, high, and max—which can be dynamically set based on task complexity:

• low: Suitable for deterministic tasks such as format conversion and simple extraction, with latency priority;
• high: Suitable for multi-hop retrieval and medium-complexity analysis;
• max: Suitable for complex planning and long-horizon code repair; official benchmarks use this tier by default.

The thinking budget is essentially a trade-off between “reasoning token overhead” and “task success rate.” In production environments, it is recommended to configure different tiers by task type rather than applying a uniform setting globally.

6. Official Benchmark Interpretation

Some officially published results are as follows (default max thinking tier; some evaluations incorporate Agent frameworks such as intelligent coding tools):

• Terminal-Bench 4.0: 65.7
• SWE Marathon v1.1: 51.7
• DeepSWE v1.1: 75.5
• BrowseComp: 92.1
• BenchCAD (with tools): 92.7
• OfficeQA Pro v2: 64.4

Several points should be noted when interpreting these results:

1. Evaluation condition differences: With tools vs. without tools, and whether an Agent framework is used, make results not directly comparable across settings;
2. Task difficulty differences: Long-horizon marathon-type evaluations (such as SWE Marathon) generally score lower than single-point tasks, which is normal;
3. Reproduction barriers: Thinking tier, tool configuration, and history message retention strategy all affect reproduction results.

7. Deployment and Integration Practice Points

7.1 History Retention for Multi-Turn Tool Calls

The official documentation explicitly states: multi-turn tool calls require retaining the complete thinking and tool call messages. This is a critical detail for Agent integration. Illustrative pseudocode is as follows:

messages = []
while not done:
    resp = model.chat(messages, thinking_budget="max")
    messages.append(resp.assistant_message)  # includes thinking and tool calls
    if resp.tool_call:
        result = execute_tool(resp.tool_call)
        messages.append({"role": "tool", "content": result})
    else:
        done = True

If historical thinking content is trimmed between turns, the model loses the basis for its decisions, leading to repeated calls or logical breaks.

7.2 Licensing and Compliance

The model weights use a dedicated license, not Apache 2.0 or MIT. Enterprises should carefully read the license terms before commercial use to confirm the scope of use, distribution conditions, and attribution requirements.

7.3 Inference Framework Selection

Supports frameworks such as vLLM, SGLang, and TokenSpeed. When selecting, consider:

• Whether the MoE structure and MXFP4 are natively supported;
• KV Cache management strategy under long contexts;
• Prefix caching hit rate in multi-turn conversations.

8. Summary

The open-sourcing of such large-scale MoE models signals that the underlying capabilities for long-horizon Agents are rapidly maturing. The key technical points can be summarized as: sparse attention controls long-context cost, hybrid attention maintains long-range dependencies, joint reinforcement learning improves multi-domain generalization, and quantization plus inference framework compatibility lowers deployment barriers. For developers, the real challenge is not calling an API, but rather designing task state machines, configuring thinking budgets, retaining complete history messages, and—under license compliance—steadily translating model capabilities into business value.