Why Long-Horizon Agents Have Become a Key Challenge

Over the past year, competition among large language models has gradually shifted from “single-turn answer quality” toward “multi-step task completion.” Whether it is automated operations in terminal environments, cross-file code repair, or 3D model generation with CAD tools, these workloads place engineering demands on models far beyond traditional inference: the model must maintain goal consistency across dozens or even hundreds of steps, call external tools correctly, and accurately retrieve key information from long contexts.

This trend is commonly summarized in the industry as “long-horizon agents.” The core challenge is not a lack of single-point capability, but rather attention decay over long contexts, state management across tool-call chains, and generalization across multi-domain tasks. Recently, an open-source model officially released its 2.8T-parameter version, offering a systematic engineering answer to these problems.

Architecture Upgrades: From Mixture-of-Experts to Sparse Attention

The MoE Foundation

This version adopts a Mixture-of-Experts (MoE) architecture with 2.8T total parameters and 104B activated parameters. The core idea behind MoE is that a routing mechanism activates only a subset of expert networks on each forward pass, keeping the compute cost of a single inference within acceptable bounds while retaining massive parameter capacity. With 104B activated parameters, the model operates at trillion-parameter scale overall, yet the parameters actually involved in computation are comparable to a mid-sized dense model—balancing capacity and efficiency.

The model supports a 1M-token context window and native image-text input. A million-token context is an infrastructure-level capability for long-horizon agents: it allows the model to load large code repositories, complete project documentation, or multi-turn tool-call histories in one pass, without relying on frequent summarization and compression.

Structural Changes with Dynamic Sparse Attention

Compared with the previous generation, the new version makes a key substitution in the attention mechanism: the original 24 layers of Gated Multi-head Latent Attention (Gated MLA) are replaced with Gated Dynamic Sparse Attention (Gated DSA), used in combination with 69 layers of Kimi Delta Attention.

The core mechanism of DSA (Dynamic Sparse Attention) is that, at each attention computation, it dynamically selects the 2,048 most relevant tokens from the history to participate, rather than attending to all historical tokens. This design directly addresses two pain points in long-context scenarios:

• Compute complexity: The cost of full attention grows quadratically with sequence length, which is unsustainable at million-token contexts. Sparse attention caps the number of tokens involved in each computation within a fixed budget, keeping inference costs controllable.
• Attention focus: In ultra-long sequences, the historical information truly relevant to the current generation step is often only a tiny fraction. Dynamic selection effectively lets the model actively “retrieve” the most relevant historical segments, instead of passively letting attention weights dilute across the entire sequence.

The gating mechanism further allows the model to adaptively trade off between sparse and dense paths, mitigating the information loss that sparsification can introduce.

Training Strategy: Joint Reinforcement Learning and Quantization

Joint RL Across Hundreds of Downstream Domains

This version was post-trained with large-scale joint reinforcement learning on top of the previous-generation model. “Joint” means the training signals come not from a single task or domain, but span hundreds of downstream domains—including terminal operations, code repair, CAD generation, web search, financial research, and office document processing.

The value of this strategy lies in the fact that long-horizon agent deployment scenarios are inherently cross-domain, and the model must switch flexibly between different tool sets and task paradigms. By optimizing across multiple domains simultaneously, joint RL encourages the model to learn more generalizable task decomposition and tool-calling strategies, rather than overfitting to the specific patterns of one task type.

MXFP4 Weight Quantization

MXFP4 weight quantization was adopted during training. MXFP4 is a 4-bit floating-point quantization format whose advantage is that it maintains good numerical stability at an extremely low bit width, compressing model storage and memory footprint while minimizing precision loss. For a model at the 2.8T-parameter scale, quantization is a critical link in deployment feasibility.

Inference Budgets: The Design Logic of Three Thinking Modes

The version provides three thinking budget levels at inference time: low, high, and max. The engineering significance of this design:

• low: For latency-sensitive simple tasks, such as single-turn Q&A or lightweight tool calls; the model reduces internal reasoning steps and returns results quickly.
• high: For moderately complex multi-step tasks, balancing reasoning depth and response time.
• max: For long-horizon agent tasks such as terminal operations, code repair, and CAD generation; the model is allowed ample internal reasoning and planning.

Official evaluation data uses the max level by default. This means that in long-horizon task scenarios, the model tends to trade more internal computation for higher task completion rates. Developers can flexibly set the thinking level based on their business trade-offs between latency and success rate.

Capability Coverage: From Code Development to Knowledge Work

Code Development and 3D Creation

This version is specifically optimized for long-horizon code agents, software engineering, and 3D design tasks. Official evaluations cover three scenarios—terminal operations, code repair, and CAD generation—and support multi-step tasks combined with tools.

Take code repair as an example. A typical workflow might be: read the error message → locate the relevant source files → analyze dependencies → generate a fix patch → run tests to verify → iterate based on test results. A mistake at any step in this chain can lead to final failure, placing high demands on the model’s context retention and tool-calling accuracy.

Agents and Knowledge Work

For knowledge work scenarios, this version supports web search, financial research, office document processing, and tool calling. Combined with the three thinking budgets, the model can run at max for deep reasoning in research-type tasks, and at high to balance efficiency in document processing tasks.

Official Benchmark Results at a Glance

Official results across different tasks are as follows (default max thinking level; some evaluations use industry-standard agentic coding tools as the agent framework):

• Terminal-Bench 4.0: 65.7
• SWE Marathon v1.1: 51.7
• DeepSWE v1.1: 75.5
• BrowseComp: 92.1
• BenchCAD (with tools): 92.7
• OfficeQA Pro v2: 64.4

These results span terminal operations, software engineering, web search, CAD generation, and office Q&A, providing reference baselines for evaluating the model’s overall capability in long-horizon agent scenarios.

Deployment and Integration Practices

Open Weights and Inference Framework Support

This version releases MXFP4 weights and supports mainstream inference frameworks such as vLLM, SGLang, and TokenSpeed, along with OpenAI/Anthropic-compatible APIs. This means developers can integrate the model relatively smoothly into existing engineering stacks built on compatible APIs, reducing migration costs.

Notes on Multi-Turn Tool Calling

The official guidance specifically notes that multi-turn tool calling requires preserving the complete history of thinking and tool-call messages. This is a key engineering detail in long-horizon agent deployment—if historical thinking content and tool-call records are truncated or dropped during multi-turn interactions, the model loses its ability to track task progress, causing subsequent steps to drift from the goal. Therefore, when building agent systems, you must ensure the complete pass-through of conversation history and the tool-call chain.

License Notes

The weights of this version are released under a custom license, not permissive open-source licenses such as Apache 2.0 or MIT. Developers should read the license terms carefully and confirm the compliance boundaries before commercial use or redistribution.

Conclusion

The release of the 2.8T-parameter version marks a systematic advance by large-scale open-source models in the long-horizon agent direction. Its technical route can be summarized as: a MoE architecture to carry massive parameter capacity, sparse attention to control long-context compute costs, joint reinforcement learning to improve cross-domain task generalization, and multi-tier inference budgets to match different latency and success-rate requirements. For developers building long-horizon agent applications, this version offers an open-source foundation worth evaluating—but they should also pay attention to its custom license terms and the engineering constraints around multi-turn tool calling.