Open-Weight LLM Released: 2.8T Sparse MoE Architecture for Long-Horizon Agent Tasks

1. Background: Demands of Long-Horizon Agent Tasks

As scenarios like intelligent programming, automated office work, and deep research mature, the industry’s demand for large language models is shifting from “single-turn Q&A” to “long-horizon task execution.” Long-horizon Agent tasks typically exhibit the following characteristics:

• Multi-step dependencies: A task may require dozens or even hundreds of steps, where the output of previous steps determines subsequent actions.
• Intensive tool calling: Frequent invocation of external tools such as terminals, retrieval systems, file processors, and drawing utilities.
• Extreme context length: Complete conversation history, tool return results, and code file contents continuously accumulate.
• High fault tolerance requirements: Deviating from the goal at any intermediate step can cause the entire task to fail.

These characteristics impose three hard requirements on the underlying model: a sufficiently large context window, efficient attention computation, and controllable reasoning depth. A recently open-weighted large model has been systematically designed around these three aspects at both the architecture and training levels.

2. Architecture Analysis: Sparse MoE and Hybrid Attention Mechanism

2.1 Overall Parameter Configuration

The model employs a sparse Mixture-of-Experts (MoE) architecture with a total parameter count of 2.8T, activating approximately 104B parameters during a single forward pass. The core objective of this “large total, small activation” design is to control the computational overhead of a single inference pass while maintaining the model’s knowledge capacity. For enterprise deployment, this means achieving expressive power close to that of an ultra-large dense model under a relatively manageable hardware budget.

The model supports a 1-million-token context window and natively supports multimodal (image-text) input. The significance of a million-level context is that complete code repositories, lengthy technical documentation, and multi-turn tool call histories can all be placed into the context at once, reducing information loss caused by truncation.

2.2 Replacement and Combination of Attention Layers

Compared to the previous generation architecture, the new version introduces two key adjustments to the attention mechanism:

1. Replaced 24 layers of Gated Multi-head Latent Attention (Gated MLA) with Gated Dense Sparse Attention (Gated DSA);
2. Combined with 69 layers of Kimi Delta Attention.

The sparse attention works by selecting 2,048 tokens from the historical tokens to participate in the attention computation, rather than performing full attention on all historical tokens. This design directly mitigates the issue where attention computation grows quadratically with sequence length in long-context scenarios.

From an engineering perspective, this hybrid structure of “sparse layers + partial fine-grained attention layers” is a pragmatic compromise: it controls the computational cost of long sequences while avoiding the loss of critical fine-grained dependencies caused by over-sparsification.

2.3 Quantization and Deployment Friendliness

The weights are released in MXFP4 format. MXFP4 is a low-bit floating-point quantization scheme that significantly compresses the weight volume while preserving numerical accuracy as much as possible. For teams looking to deploy in their own environments, this means lower VRAM usage and higher single-GPU throughput potential.

Official support is also provided for mainstream inference frameworks, alongside API interfaces compatible with OpenAI and Anthropic styles. For existing applications developed based on these two types of interfaces, migration costs are relatively low.

3. Training Methodology: Joint Reinforcement Learning and Thinking Budgets

3.1 Joint Reinforcement Learning Across Hundreds of Downstream Domains

The model was post-trained via large-scale joint reinforcement learning (RL) on top of the previous generation base model. The training covered hundreds of downstream domains rather than targeting a single task for directed optimization. The benefit of this approach is that the model possesses strong generalization capabilities across heterogeneous tasks such as coding, retrieval, office work, and design, without sacrificing performance in other scenarios due to overfitting a specific task category.

3.2 Three-Tier Thinking Budgets

During the inference phase, three thinking budgets are provided: low, high, and max, which can be dynamically configured based on task complexity:

• low: Suitable for lightweight tasks like simple Q&A and format conversion; offers fast response and low cost.
• high: Suitable for medium-complexity analysis and generation tasks.
• max: Suitable for heavy tasks like long-horizon Agents, complex code repair, and multi-step reasoning.

From a system design perspective, the thinking budget essentially provides an adjustable knob balancing “latency, cost, and quality.” In production environments, routing can be based on request types: simple requests go to the low tier, and complex requests go to the high tier, thereby optimizing overall resource utilization.

4. Capability Coverage: From Code Development to Knowledge Work

4.1 Code Development and 3D Creation

The model has been specifically optimized for long-horizon code Agents, software engineering, and 3D design tasks. Official evaluations cover three scenarios: terminal operations, code repair, and CAD generation, supporting multi-step tasks completed via tool integration.

A typical long-horizon code task flow can be abstracted as follows:

1. Read requirement descriptions and existing codebase
2. Locate relevant files and functions
3. Generate patches and call terminal to execute tests
4. Iterate and fix based on test outputs
5. Repeat the above process until passed

This flow demands high capabilities in context retention, tool call format stability, and error backtracking from the model. The combination of a million-level context and sparse attention is designed precisely for such scenarios.

4.2 Agents and Knowledge Work

In the direction of Agents and knowledge work, the model supports web retrieval, financial research, office document processing, and tool calling. The commonality among these tasks is the need for the model to actively plan retrieval paths, integrate multi-source information, and output conclusions in a structured manner.

During multi-turn tool calling, the official guidance specifically notes the need to retain the complete thinking history and tool call messages. This is critical in engineering implementation: if the intermediate process is truncated, the model may lose its previous reasoning chain, leading to repeated calls or logical breaks in subsequent steps.

5. Overview of Official Evaluation Data

The table below shows the official results of the model across various tasks. The default setting uses the max thinking budget, and some evaluations utilized Agent frameworks like intelligent programming tools:

Evaluation Metric	Score
Terminal-Bench 4.0	65.7
SWE Marathon v1.1	51.7
DeepSWE v1.1	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro v2	64.4

It should be noted that these data are published officially, and factors such as evaluation conditions, prompt templates, and tool frameworks will affect the final results. During actual model selection, it is recommended to conduct targeted verification based on your own business scenarios rather than drawing conclusions solely from a single benchmark.

6. Deployment and Usage Recommendations

6.1 Inference Framework Selection

The weights are released in MXFP4 format and support loading by mainstream inference frameworks. During deployment, pay attention to:

• Whether the VRAM capacity meets the loading requirements of the quantized weights;
• Whether the inference framework supports this sparse attention structure;
• The KV cache management strategy in long-context scenarios.

6.2 Interface Compatibility

APIs compatible with OpenAI and Anthropic styles are provided to facilitate quick integration of existing applications. For internal enterprise platforms, unified encapsulation can be performed at the gateway layer to mask the interface differences of different backend models, exposing a consistent calling method to upper-layer applications.

6.3 Considerations for Multi-Turn Tool Calling

In multi-turn Agent scenarios, it is recommended to:

1. Fully retain historical messages: Including thinking content and tool call records;
2. Control single-turn context growth: Summarize or truncate overly long tool return results;
3. Set thinking budgets reasonably: Use the low tier for simple steps and the high tier for key decision-making steps;
4. Implement failure retry and timeout control: A single-step failure in a long-horizon task should not cause the overall system to crash.

6.4 License Information

The model weights are released under a proprietary license, rather than permissive protocols like Apache 2.0 or MIT. Enterprise users should carefully read the license terms before commercial use to confirm whether their usage scenarios comply with the requirements.

7. Conclusion

The core philosophy of this open-weight large model can be summarized in three points: using sparse MoE to control inference costs, using hybrid attention to support million-level context, and using joint reinforcement learning to improve cross-domain generalization. For teams building applications in areas like long-horizon Agents, intelligent programming, and deep research, it provides an open-source option worth evaluating.

During actual implementation, it is recommended to conduct small-scale validations on typical business tasks first, focusing on three key metrics: long-context stability, tool call format accuracy, and multi-turn task completion rate, before deciding whether to incorporate it into the production system.