Large-Scale Sparse Attention Open-Source Models: Architecture Evolution and Long-Horizon Agent Practices

1. Background: New Infrastructure Requirements from Long-Horizon Agents

As agent technology moves from experimental prototypes to production deployment, developers face a growing core tension: the longer the task chain, the higher the demands on model context capacity, inference efficiency, and tool-calling stability. A typical long-horizon coding agent may need to execute dozens of consecutive terminal commands, read multiple files, fix compilation errors, and generate design diagrams. The interaction history for such a process can easily exceed the hundred-thousand to million-token range.

Traditional dense attention architectures expose two bottlenecks in this setting. First, attention computation grows quadratically with sequence length, causing long-context inference costs to rise sharply. Second, having the full history participate in computation dilutes key information, which in turn degrades subsequent decision quality. As a result, combining sparse attention with Mixture-of-Experts (MoE) architectures has become a mainstream technical approach for handling long-horizon tasks.

A recent open-source model release adopts a 2.8T total-parameter, 104B activated-parameter MoE architecture, supports a 1M-token context with native image-text input, and ships quantized weights. This article systematically breaks down the technical approach across four dimensions: architecture design, training methods, benchmark performance, and deployment practices.

2. Architecture Analysis: Co-Design of Gated Sparse Attention and Delta Attention

2.1 MoE Backbone: Decoupling Total and Activated Parameters

The model uses a Mixture-of-Experts architecture with 2.8T total parameters, but only about 104B parameters are activated per forward pass. The core value of this design is trading a smaller inference cost for a larger knowledge capacity. For long-horizon agent scenarios, this means the model can maintain response speed while accommodating broader domain knowledge—from terminal command semantics to computer-aided design conventions.

The key engineering challenge of MoE architectures is load balancing in expert routing. With a poor routing strategy, a small number of experts can be over-activated, leading to training instability and inference hotspots. A common industry practice is to introduce auxiliary loss functions and capacity factor constraints to keep the number of tokens processed by each expert relatively uniform.

2.2 From Gated Multi-Head Attention to Gated Sparse Attention

Compared with the previous generation, the new model makes two important substitutions in the attention layers:

• Replacing 24 layers of Gated Multi-head Latent Attention (Gated MLA) with Gated Sparse Attention (Gated DSA)
• Combining it with 69 layers of Kimi Delta Attention

The core mechanism of sparse attention is that each attention computation selects only 2,048 historical tokens to participate, rather than letting all historical tokens participate. This design yields two direct benefits:

1. Computational complexity drops from quadratic to approximately linear, making inference over a 1M-token context feasible;
2. Historical information is filtered before participating in computation, reducing noise interference and helping the model focus on the context segments most relevant to the current task.

A simplified pseudocode sketch of the selection logic looks like this:

def gated_sparse_attention(query, history_tokens, top_k=2048):
    # Compute relevance scores between the query and historical tokens
    relevance_scores = compute_relevance(query, history_tokens)
    # Select the top_k tokens with the highest relevance
    selected_tokens = top_k_select(history_tokens, relevance_scores, top_k)
    # Run attention only over the selected tokens
    attention_output = scaled_dot_product_attention(query, selected_tokens)
    return attention_output

Notably, the sparse selection strategy and delta attention are complementary: the former is responsible for quickly locating key information in a long history, while the latter preserves fine-grained modeling capability within local windows.

3. Training Methods: Joint Reinforcement Learning and Quantization Awareness

3.1 Joint Reinforcement Learning Across Hundreds of Downstream Domains

In the post-training stage, the model adopts a joint reinforcement learning strategy covering hundreds of downstream domains. Unlike traditional single-task fine-tuning, joint reinforcement learning aims to let the model optimize multiple objectives simultaneously: accuracy in code repair, robustness in terminal operations, information integration in web retrieval, and conformance to standards in office document processing.

The difficulty of this training approach lies in reward signal design and conflict resolution. Reward scales differ considerably across tasks, and direct summation may cause one task category to dominate the gradient direction. Industry practice typically relies on reward normalization and task sampling weight adjustment to achieve balance.

3.2 Multiple Thinking Budgets: low, high, max

To accommodate tasks of varying complexity, the model offers three thinking budgets:

Tier	Use Case	Characteristics
low	Simple Q&A, format conversion	Fast response, low compute cost
high	Routine code generation, document processing	Balanced quality and latency
max	Long-horizon agents, complex reasoning	Default evaluation tier, quality first

Developers can dynamically set the thinking budget based on task type. For example, in terminal operation tasks with fairly fixed command sequences, the low tier can be selected to reduce latency; in multi-step software engineering tasks, the max tier is recommended to ensure reasoning depth.

3.3 MXFP4 Weight Quantization

After training, the model weights are quantized to the MXFP4 format. MXFP4 is a 4-bit floating-point quantization scheme that, compared with traditional INT8 quantization, further compresses model size while maintaining numerical stability. This is particularly important for scenarios that need to deploy a 104B activated-parameter model under limited GPU memory.

The accuracy loss from quantization must be compensated through quantization-aware training or post-training calibration. The model still achieves strong results across multiple benchmarks after quantization, indicating that its quantization pipeline has been thoroughly tuned.

4. Benchmark Performance: Interpreting Multi-Dimensional Evaluations

The officially published evaluation results cover terminal operations, code repair, web retrieval, CAD generation, and office document processing. The specific figures are as follows (max thinking tier by default, with some evaluations using an intelligent coding tool framework):

• Terminal operation benchmark: 65.7
• Software engineering marathon benchmark: 51.7
• Deep software engineering benchmark: 75.5
• Web retrieval benchmark: 92.1
• CAD generation benchmark (with tools): 92.7
• Office document QA benchmark: 64.4

It is important to emphasize that these results reflect performance under specific evaluation frameworks and tool integration conditions. Real-world production results also depend on prompt design, tool interface stability, history message management strategy, and other factors. Developers should not equate benchmark scores directly with usability in business scenarios; instead, they should validate against their own tasks.

Looking at the distribution, web retrieval and CAD generation score relatively high, suggesting the model has an advantage in tasks that require integrating information with external tools. The software engineering marathon benchmark scores relatively lower, indicating that long-horizon coding tasks remain difficult and may require more refined context management and tool orchestration strategies.

5. Deployment Practices: Compatible Interfaces and Multi-Turn Tool Calling

5.1 Inference Framework Support

The open weights support mainstream inference frameworks, including vLLM, SGLang, and TokenSpeed. These frameworks support optimization techniques such as continuous batching and paged attention, which can effectively improve throughput. During deployment, note the following:

• Confirm whether the framework version supports the model’s sparse attention operators;
• Choose an appropriate tensor parallelism or pipeline parallelism strategy based on GPU memory capacity;
• Use the corresponding dequantization configuration when loading quantized weights.

5.2 Compatible API Design

The model provides API interfaces compatible with OpenAI and Anthropic styles. This means existing applications built on these interface specifications can migrate at relatively low cost. A typical call looks like this:

# Pseudocode sketch; refer to the actual documentation for specific interfaces
response = client.chat.completions.create(
    model="large-moe-model",
    messages=[
        {"role": "system", "content": "You are a long-horizon coding agent"},
        {"role": "user", "content": "Fix the compilation errors in the current directory"}
    ],
    extra_body={"thinking_budget": "max"}
)

5.3 History Management for Multi-Turn Tool Calling

In agent scenarios, the model needs to call tools over multiple turns. In this case, the complete thinking messages and tool-call messages must be preserved; otherwise the model loses its decision context, leading to repeated calls or broken logic. Recommended practices:

1. Maintain a session history list and append each message in chronological order;
2. After a tool returns a result, append the result as a new message and preserve the original call ID;
3. When history length approaches the context limit, use a sliding window or summarization compression strategy, but ensure the most recent tool-call chain remains complete.

5.4 License Notes

The model weights use a dedicated license rather than a permissive license such as Apache 2.0 or MIT. Enterprise users should carefully read the license terms before commercial use and confirm whether their usage scenarios comply with the requirements. This is an easily overlooked but critical compliance step in deploying open-source models.

6. Summary and Outlook

The combination of large-scale sparse attention and MoE architectures offers long-horizon agents a technical path that balances capacity and efficiency. By filtering key history through gated sparse attention, preserving local modeling through delta attention, covering multi-domain tasks through joint reinforcement learning, and lowering deployment barriers through quantization, this approach demonstrates engineering feasibility across multiple dimensions.

For developers, worthwhile next steps include the interpretability and tunability of sparse selection strategies, context compression algorithms for multi-turn tool calling, and how to achieve domain adaptation at low cost in vertical domains. With open weights and compatible interfaces maturing, the barrier to building long-horizon agents is dropping significantly—but the real challenge remains translating model capability into business value reliably.