Understanding Long-Horizon Agent Capabilities in an Open-Weight Sparse-Attention MoE Model

1. Background: Why Long-Horizon Agents Require a New Model Architecture

Over the past year, agents have evolved rapidly from single-turn Q&A to multi-step task execution. Whether performing repository-level code fixes, orchestrating terminal commands, or handling cross-page web retrieval and office document processing, task chains often span dozens or even hundreds of steps. These scenarios impose three hard requirements on the underlying model:

1. Very long context: Historical reasoning, tool call results, and intermediate artifacts must be retained throughout the entire task lifecycle. Otherwise, the agent is prone to “forgetting” and repeating work.
2. Stable long-horizon reasoning: As the number of steps grows, error accumulation becomes more pronounced. The model must maintain decision consistency without significantly increasing inference cost.
3. Adjustable compute budgets: Different tasks have widely varying sensitivity to latency and quality. Simple Q&A does not require maximum reasoning effort, while complex engineering tasks demand deeper inference.

Recently, an open-weight MoE model based on a sparse-attention architecture was released specifically to address these requirements. This article reviews its technical characteristics across four dimensions: architecture, training, evaluation, and deployment.

2. Core Specifications: Sparse MoE and Million-Token Context

The model uses a Mixture-of-Experts (MoE) architecture with 2.8T total parameters and approximately 104B activated parameters per forward pass. The significance of this design is that the large capacity enabled by sparse activation improves knowledge density and generalization while keeping inference cost manageable.

Key specifications can be summarized as follows:

• Total parameters / Activated parameters: 2.8T / 104B
• Context length: Supports 1M tokens
• Input modality: Native image-text input
• Weight quantization: MXFP4
• Inference budgets: low, high, and max reasoning modes

The value of a 1M-token context for agents goes beyond simply fitting more content. In multi-turn tool-calling scenarios, fully preserving historical reasoning and tool call messages is a prerequisite for maintaining a correct task state machine. If context is truncated, the model may forget operations it has already performed, parameters it has confirmed, or paths it has already ruled out, leading to duplicate calls or state conflicts. Long context is therefore a functional requirement here, not merely a capacity metric.

3. Architecture Evolution: From Gated MLA to Gated DSA

This version makes a clear change in its attention mechanism: the previous 24 layers of Gated MLA are replaced with Gated DSA, combined with 69 layers of Kimi Delta Attention. The core behavior of DSA is that each attention computation selects only 2048 historical tokens to attend to.

The engineering implications of this design are worth unpacking:

• Sparse selection reduces computation: In a million-token context, full attention over all historical tokens would quickly drive compute and memory costs out of control. By selecting only the 2048 tokens most relevant to the current query at each step, attention cost is significantly compressed.
• Gating preserves stability: The gated structure helps regulate information flow across layers, preventing sparse selection from causing information fragmentation.
• Synergy with Delta Attention: Delta Attention handles state updates and incremental information, while DSA performs efficient retrieval over history. Together they form an attention division of labor based on “efficient retrieval + stable updates.”

For developers, this means the model is more likely to maintain a balance of “remembers, retrieves, and computes efficiently” in long-horizon tasks.

4. Training Strategy: Large-Scale Joint Reinforcement Learning

This model is not simply the product of language model pretraining. Instead, it builds on a base version with joint reinforcement learning post-training covering hundreds of downstream domains. The goal of this strategy is to improve the model across a diverse task distribution simultaneously, rather than overfitting to a single benchmark.

The value of joint reinforcement learning lies in two areas:

1. Cross-domain transfer: Tasks such as code repair, terminal operations, retrieval Q&A, and office document processing differ on the surface, but all rely on planning, tool calling, and result verification capabilities. Joint training helps consolidate these general abilities.
2. Long-horizon consistency: Reinforcement learning signals can cover the overall success or failure of multi-step tasks, not just single-step output quality, thereby optimizing the model’s decision-making behavior over long chains.

Combined with MXFP4 weight quantization, the model reduces memory usage and deployment barriers while maintaining high accuracy, making it possible for the open-source community to run it on general-purpose hardware.

5. Capability Focus: Code Engineering, Agents, and Knowledge Work

5.1 Code Development and 3D Creation

The model is explicitly oriented toward long-horizon code agents, software engineering, and 3D design tasks. Official evaluations cover terminal operations, code repair, and CAD generation, with support for completing multi-step tasks through tool use.

A typical workflow can be abstracted as follows:

1. Read the task description and repository state
2. Locate relevant files and dependencies
3. Generate a modification plan and execute it through tools
4. Run tests or validation commands
5. Iterate on fixes based on feedback
6. Output the final changes and explanation

In this flow, the model must simultaneously possess code understanding, tool invocation, and error recovery capabilities. The difficulty of multi-step tasks lies not in single-point generation, but in state tracking and rollback decisions across steps.

5.2 Agents and Knowledge Work

For agent and knowledge work scenarios, the model supports web retrieval, financial research, office document processing, and tool invocation. The inference side provides three reasoning budgets—low, high, and max—that can be set dynamically based on task complexity.

The practical value of this design lies in:

• low: Suitable for simple Q&A, format conversion, and other low-latency scenarios.
• high: Suitable for tasks requiring moderate reasoning depth but with cost sensitivity.
• max: Suitable for complex engineering, long-horizon planning, and high-precision tasks.

Developers can combine reasoning budgets with task routing. For example, in an agent framework, retrieval-oriented subtasks can use a lower budget, while planning and repair subtasks can use a higher budget, striking a balance between overall cost and quality.

6. What the Official Evaluation Data Shows

The officially published evaluation results are as follows (default max reasoning mode, with some evaluations using an agent framework):

• Terminal-Bench 4.0: 65.7
• SWE Marathon v1.1: 51.7
• DeepSWE v1.1: 75.5
• BrowseComp: 92.1
• BenchCAD (with tools): 92.7
• OfficeQA Pro v2: 64.4

These numbers should be understood as reference values under specific evaluation conditions, not as universal capability guarantees. When evaluating model options, readers should note:

1. Whether tools are used, and which agent framework is used, can significantly affect results.
2. Task distributions vary widely across benchmarks; a high score on one benchmark does not imply comprehensive leadership.
3. Real-world performance also depends on prompt design, tool quality, and context management strategy.

7. Deployment and Integration Considerations

The model is released with MXFP4 weights, supports mainstream inference frameworks, and provides OpenAI/Anthropic-compatible APIs. Key engineering considerations include:

• Multi-turn tool calls must retain the complete history: This includes historical reasoning and tool call messages. Any lossy truncation can break task state.
• The license is not a permissive license: The weights use a custom license, not Apache 2.0 or MIT. Review the terms carefully before commercial use.
• Reasoning budgets should work with the agent framework: The framework should support dynamically setting budgets per subtask to avoid cost inflation from a global max setting.
• Validate quantized weight precision: MXFP4 performs well on most tasks, but targeted regression testing is recommended for numerically sensitive tasks.

8. Summary

The core approach of this model can be summarized as: using sparse MoE for large capacity, sparse attention for long context, joint reinforcement learning to refine long-horizon agent capabilities, and adjustable reasoning budgets to balance cost and quality. For teams building code agents, retrieval Q&A systems, or office automation tools, it provides an open-source option worth evaluating. The key to successful deployment lies not in any single benchmark score, but in the overall engineering capability around context management, tool orchestration, and budget scheduling.