Open-Source Sparse Attention LLM Released: 2.8T MoE Architecture for Long-Horizon Agents

Release Overview and Core Specifications

On November 4, a large language model designed for long-horizon agent tasks was released with open weights. Built on its predecessor’s architecture and trained via large-scale joint reinforcement learning, the model uses a Mixture-of-Experts (MoE) design with 2.8T total parameters and 104B active parameters. It supports a 1 million-token context window and native multimodal input (text and images).

These specifications signal several key design choices. First, the combination of 2.8T total parameters with 104B active parameters indicates a sparse activation strategy that preserves large knowledge capacity while keeping single-inference compute costs manageable—a common trade-off in modern large-scale MoE systems. Second, the 1M-token context window targets long-horizon scenarios such as cross-file code refactoring, multi-round agentic tool-use workflows, and coherent processing of long documents. Third, native image-text input means the model processes visual information directly, rather than relying on an external vision encoder.

The weights are released under the model’s dedicated license, not under permissive licenses like Apache 2.0 or MIT. This matters for enterprise users: before integrating the model into commercial products, review the license terms carefully—especially usage scope, redistribution conditions, and derivative work requirements.

Architecture Evolution: From Gated MLA to Gated DSA

This version introduces structural changes to the attention mechanism. Compared to the previous release, 24 layers of Gated MLA (Gated Multi-head Latent Attention) have been replaced with Gated DSA (Gated Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention.

The core idea of DSA is dynamic sparsification: for each attention computation, only 2,048 historical tokens are selected for participation, rather than performing dense attention over the full context. This yields direct benefits in long-context scenarios—dense attention grows quadratically as sequence length reaches hundreds of thousands or millions of tokens, while sparse attention significantly reduces computational complexity.

A simplified illustration of how it works:

# Pseudo-code: DSA dynamic sparse attention selection logic
def gated_dsa_attention(query, history_tokens, top_k=2048):
    # 1. Compute relevance scores between query and historical tokens
    scores = relevance_score(query, history_tokens)
    # 2. Dynamically select the top_k most relevant historical tokens
    selected = top_k_select(scores, k=top_k)
    # 3. Perform attention only on the selected tokens
    attn_output = sparse_attention(query, selected)
    # 4. Gating mechanism modulates information flow
    return gate(query) * attn_output

It is worth emphasizing that the engineering challenge of sparse attention lies not in “selecting” but in “selecting accurately and quickly.” If the selection strategy itself is too expensive, the compute saved on attention is offset by the selection cost. Gated DSA combines a gating mechanism with dynamic selection to balance efficiency and effectiveness.

Training Strategy: Joint Reinforcement Learning and Quantization

On the training side, this version employs joint reinforcement learning across hundreds of downstream domains. The significance of this approach: traditional pipelines often perform general pretraining followed by task-specific fine-tuning, which can fragment capabilities across tasks. Joint reinforcement learning allows the model to optimize across multiple domains simultaneously, fostering transferable general problem-solving abilities—especially in agentic tasks requiring multi-step reasoning and tool use.

The weights use MXFP4 quantization. MXFP4 is a 4-bit floating-point format that, compared to integer formats like INT4, better represents weights with large dynamic ranges. This helps compress model size while preserving inference quality. For deployers, this means the model can be loaded with relatively limited VRAM, lowering the hardware barrier for inference services.

Capability Coverage: Three Task Directions

The official evaluation covers three main directions, each mapped to specific task types and metrics.

Code Development and 3D Creation

This direction targets long-horizon coding agents, software engineering, and 3D design tasks. Evaluations cover terminal operations, code repair, and CAD generation, with support for tool-augmented multi-step tasks.

Specific results: Terminal-Bench 4.0 scores 65.7, SWE Marathon v1.1 scores 51.7, DeepSWE v1.1 scores 75.5, and BenchCAD (with tools) scores 92.7. These metrics correspond to terminal-environment operation ability, completion of long-horizon software engineering tasks, code repair and generation quality, and tool-augmented CAD generation, respectively. The BenchCAD result near 93 suggests strong tool-collaboration capability in structured 3D design tasks.

Agents and Knowledge Work

This direction supports web search, financial research, office document processing, and tool invocation. In corresponding evaluations, BrowseComp scores 92.1 and OfficeQA Pro v2 scores 64.4. BrowseComp tests browsing and information retrieval scenarios; a score of 92.1 indicates high completion quality in open-domain information acquisition.

For reasoning, the model offers three thinking budgets: low, high, and max. This design is practical for deployment: simple tasks can use the low budget to reduce latency and cost, while complex reasoning tasks can switch to max for higher accuracy. Developers can set the budget dynamically based on task complexity, rather than paying the same compute cost for every request.

Evaluation Conditions

It should be noted that the above scores default to the max thinking budget, and some evaluations used agent frameworks such as intelligent coding tools. This means the scores reflect the combined capability of “model + tool framework,” not the model running in isolation. When assessing reproducibility, consider your own technical stack accordingly.

Deployment and Integration Considerations

Weights are released in MXFP4 format and support mainstream inference frameworks including vLLM, SGLang, and TokenSpeed, with OpenAI/Anthropic-compatible APIs. This lowers integration costs—if your existing system already supports these compatible interfaces, the migration effort is minimal.

In multi-turn tool-call scenarios, you must retain the full history of reasoning and tool-call messages. This point is easy to overlook: some frameworks only keep the final reply and discard intermediate reasoning steps and tool-call records, causing the model to lose context in subsequent turns—resulting in “forgetting what tools were called and what results were returned.” The correct approach is to pass the complete message chain, including reasoning fragments and tool responses.

# Example of retaining multi-turn tool-call messages
messages = [
    {"role": "user", "content": "Query a dataset and generate a summary"},
    {"role": "assistant", "content": "I need to call the search tool", 
     "tool_calls": [{"name": "search", "arguments": {...}}]},
    {"role": "tool", "content": "Search results..."},
    {"role": "assistant", "content": "Based on the results, here is the summary..."}
]
# The next request must carry the full messages list, not just the last entry

Regarding the license: again, a dedicated license is not equivalent to a permissive open-source license. Confirm compliance before commercial use.

Practical Recommendations and Summary

For teams considering adopting this model, evaluate the following dimensions:

Scenario fit. Long-horizon coding agents, multi-step tool invocation, and long-document processing are the model’s strengths. If your tasks are mainly short-text classification or simple Q&A, this scale of model may be unnecessary.

Thinking budget configuration. Use the low/high/max tiers to allocate calls by task complexity, balancing latency, cost, and accuracy.

Message chain completeness. For multi-turn tool calls, always retain the full history. This is a critical engineering detail for maintaining coherent agent behavior.

License review. The dedicated license terms require confirmation by both legal and engineering teams to avoid downstream risks.

Overall, this release addresses long-context computational efficiency through the Gated DSA and Kimi Delta Attention combination; improves cross-domain capability through joint reinforcement learning; and lowers deployment barriers through MXFP4 quantization and compatible APIs. For developers building long-horizon agent systems, this is a technical option worth evaluating.