Open-Source LLM Releases New Long-Horizon Agent Version: Architecture, Capabilities, and Benchmark Analysis

1. Background: What Long-Horizon Agents Demand from Models

Over the past year, the application focus of large language models has shifted rapidly from single-turn Q&A to long-horizon agents. A long-horizon task requires the model to repeatedly cycle through “observe—reason—call a tool—observe again” over dozens or even hundreds of steps, while maintaining memory of the task goal and the ability to plan throughout the process.

This type of scenario imposes three requirements that were less prominent in earlier use cases:

1. Very long context: Tool call history, file contents, and retrieval results accumulate quickly. The context window must start at the million-token level; otherwise, tasks get truncated midway.
2. Sparse attention: As context grows longer, attention computation scales quadratically. Sparse attention mechanisms are needed to preserve key information while controlling computational cost.
3. Controllable reasoning budget: Different tasks have very different needs for reasoning depth. Simple tool calls do not require long reasoning chains, while complex code repair demands thorough planning. The model needs adjustable thinking budget tiers.

Recently, an open-source LLM released its new-generation long-horizon agent version and made the model weights publicly available. This article provides a systematic overview across five dimensions: architecture, training, capabilities, benchmarks, and deployment.

---

2. Architecture Upgrade: From Dense Attention to Hybrid Sparse Attention

2.1 Overall Parameter Structure

This version adopts a Mixture of Experts (MoE) architecture with approximately 2.8T total parameters and roughly 104B active parameters per forward pass. This “large total, small activation” design allows the model to maintain a large knowledge capacity while keeping per-inference compute overhead within an acceptable range.

For context length, this version supports approximately 1 million tokens of input and natively supports mixed image-text input. This means it can directly process screenshots, design mockups, and interface images without requiring a separate external vision encoder module.

2.2 Replacing the Attention Layers

Compared with the previous generation, the core architectural upgrade lies in restructuring the attention layers:

• Previous approach: 24 layers used Gated Multi-head Latent Attention (Gated MLA).
• New approach: These 24 layers are replaced with Gated Dynamic Sparse Attention (Gated DSA), combined with 69 Kimi Delta Attention layers.

The working mechanism of DSA can be summarized as follows: during attention computation at each layer, the model does not attend to all historical tokens. Instead, it dynamically selects approximately 2,048 historical tokens for computation. The selected tokens are typically those with the highest semantic relevance to the current query, such as task goal statements, key tool return results, and recent error messages.

The value of this design lies in several aspects:

• Lower computational complexity: Attention cost shifts from quadratic growth with sequence length to near-linear growth, making million-token contexts feasible from an engineering standpoint.
• Higher information density: Sparse selection naturally acts as an information filter, reducing the interference of noise in long contexts.
• Complementary behavior with incremental attention: Kimi Delta Attention handles incremental updates from recent information, while DSA retrieves key segments from distant history. Together they form a “near-range + long-range” collaboration.

2.3 Quantization and Inference Optimization

The weights use MXFP4 quantization format. MXFP4 is a microscaling floating-point format that shares exponents across blocks, significantly reducing memory usage while preserving numerical precision as much as possible. For MoE architectures, quantization effectively relieves the memory pressure caused by the extremely large parameter count, making single-node multi-GPU deployment more feasible.

---

3. Training Strategy: Joint Reinforcement Learning Across Hundreds of Domains

This version is not pre-trained from scratch. Instead, it builds on the previous generation through large-scale joint reinforcement learning post-training. Its key characteristics include:

1. Broad domain coverage: The reinforcement learning reward signals span hundreds of downstream domains, including terminal operations, code repair, software engineering, retrieval-based Q&A, office document processing, financial research, and 3D design. This multi-domain joint approach aims to prevent the model from overfitting to a single task and to preserve general-purpose agent capabilities.
2. Diverse task formats: Training tasks include not only single-step Q&A but also compound tasks requiring multiple rounds of tool calls. The model must learn to maintain state throughout long-horizon interactions.
3. Reward design: The training typically combines outcome rewards (whether the final task was completed) with process rewards (whether intermediate steps were reasonable), mitigating the credit assignment problem in long-horizon tasks.

It is worth noting that the official benchmark notes mention that some evaluations were run with an agent framework. This means the reported capabilities reflect a “model + framework” combination rather than the raw model alone.

---

4. Capability Dimensions: Three Typical Task Categories

4.1 Code Development and 3D Creation

This targets long-horizon coding agents, software engineering, and 3D design tasks. Official evaluations cover terminal operations, code repair, and CAD generation, all of which require the model to complete multi-step tasks with tools. For example, in code repair, the model needs to: read the error message → locate relevant files → understand the context → generate a patch → run verification → iterate based on the results.

4.2 Agentic and Knowledge Work

The model supports web retrieval, financial research, office document processing, and tool calling. The difficulty in these tasks lies in the heterogeneity of information sources: web pages are semi-structured text, documents are long-form text, and retrieval results may conflict with each other. The model needs the ability to evaluate and integrate information.

4.3 Three Thinking Budget Tiers

This version provides three thinking budget tiers: low, high, and max. These determine the upper limit of reasoning tokens the model can allocate before generating the final answer:

• low: Suitable for simple tool calls and format conversion tasks. Fast response and low cost.
• high: Suitable for medium-complexity reasoning, such as multi-step retrieval and summarization.
• max: Suitable for complex code repair and long-horizon planning. Longer reasoning chains, but higher latency and cost.

Developers can dynamically select the tier based on task type, balancing effectiveness and cost.

---

5. Official Benchmark Results

The officially published benchmark results are as follows (using the max thinking budget by default):

Benchmark	Score
Terminal-Bench 4.0	65.7
SWE Marathon v1.1	51.7
DeepSWE v1.1	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro v2	64.4

Several observations can be drawn from the score distribution:

• Strong performance on retrieval tasks: BrowseComp reaches 92.1, indicating strong capability in locating and integrating information in long contexts. This is directly related to the sparse retrieval mechanism of DSA.
• Excellent CAD generation: BenchCAD with tools scores 92.7, showing maturity in structured geometric generation combined with tool usage.
• Software engineering tasks still have room for improvement: SWE Marathon at 51.7 reflects that very long-horizon coding tasks—potentially involving dozens of files and multiple rounds of verification—remain a shared challenge for all current models.

It is important to note that benchmark scores are affected by multiple factors, including the agent framework, prompts, tool sets, and retry strategies. Direct cross-model comparisons should be made with caution.

---

6. Deployment and Usage Notes

6.1 Inference Engine Support

The weights are released in MXFP4 format and are supported by mainstream inference engines such as vLLM, SGLang, and TokenSpeed. OpenAI-compatible and Anthropic-compatible APIs are provided. This means existing applications built on either API style can be migrated at relatively low cost.

6.2 Multi-Round Tool Calling Considerations

The official guidance specifically notes that in multi-round tool calling scenarios, the complete history of reasoning content and tool call messages must be preserved. The reasons are:

• Sparse attention relies on historical tokens for retrieval. If history is truncated, the model may lose track of context.
• Tool call messages contain parameters and return results, which are necessary for subsequent reasoning.
• If reasoning content is discarded, the model may repeat completed reasoning in the next round, wasting resources.

A simplified message structure example:

[
  {"role": "system", "content": "You are an agent that can call tools"},
  {"role": "user", "content": "Fix the build error in the project"},
  {"role": "assistant", "content": "I will first read the build log", "tool_calls": [...]},
  {"role": "tool", "content": "<build log content>"},
  {"role": "assistant", "content": "The issue is in the config file", "tool_calls": [...]}
]

The key point is that the assistant reasoning content and tool return content from each round should be fully preserved in the context.

6.3 License Notice

The weights for this version are released under a custom license, not a permissive license such as Apache 2.0 or MIT. Enterprises should carefully review the license terms before commercial use, including allowed use cases, redistribution conditions, and compliance requirements.

---

7. Summary and Selection Guidance

Overall, the core upgrades in this version focus on three areas: hybrid sparse attention enabling long-context capability, joint reinforcement learning across hundreds of domains enabling general-purpose agent capability, and three thinking budget tiers enabling cost controllability.

For developers and enterprise users, the following considerations may help when evaluating this type of model:

1. Clarify task length: If tasks typically complete within 10 steps, the long-context advantage is less significant. If tasks involve many files and multiple rounds of tool calls, this type of long-horizon model delivers more value.
2. Evaluate cost structure: The max tier produces the best results but incurs the highest cost. A tiered routing approach in production—sending simple tasks to the low tier—is recommended.
3. Validate framework compatibility: Benchmark scores depend on the agent framework. End-to-end validation within your own business framework is necessary before deployment.
4. Pay attention to license compliance: A non-permissive license means legal review is required before commercial use.

Long-horizon agents are still evolving rapidly. Architectural sparsification and multi-domain reinforcement learning are becoming mainstream technical directions in this space. Worth watching in the future: whether sparse attention selection strategies become more learned, whether thinking budgets can be allocated dynamically and automatically, and whether long-horizon task success rates can move from “usable” to “reliable.”