Trillion-Parameter MoE Model Goes Open Source: A Technical Architecture Analysis of Long-Horizon Agent Capabilities

1. Background: New Architectural Requirements for Long-Horizon Agent Tasks

As agent technology evolves from simple question-answering toward multi-step, long-chain tasks, developers are placing higher demands on underlying models in terms of context capacity, reasoning depth, and tool-calling capabilities. Traditional dense architecture models face quadratic growth in attention computation when handling million-token contexts, making inference cost and latency difficult to control. Meanwhile, scenarios such as code development, software engineering, and 3D design require models to maintain state consistency across dozens or even hundreds of operations, posing new challenges for memory mechanisms and reasoning budget management.

Recently, an open-source model based on a Mixture-of-Experts (MoE) architecture officially released its weights. Its design of 2.8T total parameters, 104B activated parameters, a 1-million-token context window, and native image-text input capabilities offers a reference technical path for addressing these challenges. This article analyzes the model across four dimensions: architecture design, training strategy, inference mechanisms, and engineering deployment.

2. Architecture Design: A Hybrid Approach Combining Sparse Attention and Linear Attention

2.1 The Basic Logic of MoE Architecture

The model adopts an MoE architecture with a total parameter scale of 2.8T, but only 104B parameters are activated per forward pass. This design maintains model capacity while keeping the computation per inference run at a level comparable to tens-of-billions-scale dense models. The core of MoE lies in the dynamic routing of tokens to experts via a gating network: different tokens are assigned to different expert sub-networks, achieving a balance between parameter efficiency and computational efficiency.

For long-horizon agent tasks, the advantage of MoE architecture is not only reflected in inference cost, but also in the fact that its multi-expert structure can implicitly learn specialized representations for different task domains—for example, code repair, terminal operations, and document processing may activate different expert combinations.

2.2 From Gated MLA to Gated DSA: Evolution of Attention Mechanisms

Compared to the previous generation, the new model makes a key replacement in its attention mechanism: replacing 24 layers of Gated MLA (Multi-head Latent Attention) with Gated DSA (Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention (a linear attention variant).

The core mechanism of DSA is: during each attention computation, dynamically select the 2,048 most relevant tokens from historical tokens to participate in the computation, rather than performing full attention over all historical tokens. This design reduces the attention computation complexity in long-context scenarios from O(n²) to approximately O(n×k), where k is the number of selected tokens (2,048 in this case).

Schematically, the DSA selection process can be understood as:

# Pseudocode illustration
def gated_dsa_attention(query, history_tokens, top_k=2048):
    # Compute relevance scores between query and history tokens
    relevance_scores = compute_relevance(query, history_tokens)
    # Select top-k most relevant tokens
    selected_tokens = top_k_select(history_tokens, relevance_scores, k=top_k)
    # Perform attention only on selected tokens
    output = attention(query, selected_tokens)
    return output

This sparsification strategy is particularly critical in million-token contexts: full attention computation grows quadratically with sequence length, while DSA controls it within a linearly acceptable range.

2.3 The Complementary Role of Linear Attention Layers

The 69 layers of Kimi Delta Attention, as a linear attention mechanism, have computational complexity that scales linearly with sequence length. This complements DSA’s sparse selection: linear attention layers are responsible for capturing global, continuous context dependencies, while DSA layers handle precise sparse retrieval at key positions. This hybrid architecture balances efficiency and precision in long-horizon tasks.

3. Training Strategy: Joint Reinforcement Learning and Quantization

3.1 Large-Scale Joint Reinforcement Learning

The model undergoes large-scale joint reinforcement learning post-training based on the previous generation, with training covering hundreds of downstream domains. The core challenge of joint reinforcement learning lies in the fact that reward signals differ dramatically across tasks, and balancing heterogeneous tasks such as code repair, terminal operations, CAD generation, and web retrieval within a single training framework is difficult.

One feasible approach is multi-task reward normalization with dynamic weight adjustment: standardizing reward signals for each domain and dynamically adjusting the sampling ratio of each task based on training progress, preventing the model from overfitting to one domain while forgetting other capabilities.

3.2 MXFP4 Weight Quantization

Model weights are quantized using the MXFP4 format. MXFP4 is a 4-bit floating-point quantization scheme. Compared to traditional INT4 quantization, the floating-point format has advantages in representing weights with large dynamic ranges. For trillion-parameter models, quantization not only reduces storage requirements but also decreases memory bandwidth pressure during inference, making deployment feasible under reasonable hardware configurations.

4. Inference Mechanisms: Multi-Tier Thinking Budgets and Tool Calling

4.1 Three-Tier Thinking Budget

The model offers three thinking budget tiers: low, high, and max. Developers can flexibly configure these based on task complexity. The engineering significance of this design is:

• Low tier: Suitable for simple Q&A, short text generation, and other low-latency scenarios;
• High tier: Suitable for medium-complexity code completion and document processing;
• Max tier: Suitable for long-horizon agent tasks, multi-step reasoning, complex code repair, and similar scenarios.

The essence of the thinking budget is controlling the number of internal reasoning tokens the model generates before producing a final answer. A higher budget gives the model more “thinking space” to plan steps and verify intermediate results, but latency and cost increase accordingly.

4.2 State Management for Multi-Turn Tool Calling

In agent scenarios, the model needs to call external tools across multiple turns (such as code executors, retrieval interfaces, CAD engines, etc.). A key engineering detail is that multi-turn tool calling must preserve the complete history of thinking and tool-calling messages. This means that when constructing conversation history, it is not sufficient to retain only the final answer; the reasoning process, tool-calling requests, and tool return results from each turn must be fully preserved so the model can correctly understand the context state in subsequent turns.

5. Capability Evaluation and Scenario Analysis

The official evaluation covers multiple dimensions. Below are the results under the default max thinking tier:

Evaluation Dimension	Benchmark	Score
Terminal Operations	Terminal-Bench 4.0	65.7
Code Repair	SWE Marathon v1.1	51.7
Software Engineering	DeepSWE v1.1	75.5
Web Retrieval	BrowseComp	92.1
CAD Generation	BenchCAD (with tools)	92.7
Office Documents	OfficeQA Pro v2	64.4

From the evaluation design, it is clear that the model’s capabilities cover a complete chain from low-level terminal operations to high-level knowledge work. Some evaluations used agent frameworks such as intelligent programming tools, indicating that its tool-calling interface has been adapted to mainstream agent orchestration solutions.

6. Engineering Deployment: Deployment and Interface Compatibility

6.1 Inference Framework Support

The model releases MXFP4 weights and supports deployment on mainstream inference frameworks. Developers can choose an appropriate inference backend based on their existing infrastructure and leverage quantized weights to reduce memory usage.

6.2 API Compatibility

The model provides OpenAI/Anthropic-compatible APIs, meaning existing applications built on these interface specifications can migrate at relatively low cost. For enterprise users, this compatibility lowers the integration barrier and facilitates replacing the underlying model in existing agent systems.

6.3 License Notes

The weights use a proprietary license, not permissive open-source licenses such as Apache 2.0 or MIT. Enterprises should carefully review the license terms before commercial use to confirm the scope of use and compliance requirements.

7. Summary and Outlook

The open-sourcing of a trillion-parameter MoE model provides a high-capacity, customizable underlying option for long-horizon agent tasks. Its technical highlights can be summarized in three points:

1. A hybrid architecture combining sparse attention and linear attention, achieving a balance between efficiency and precision in million-token contexts;
2. Joint reinforcement learning covering hundreds of domains, enabling a single model to possess cross-task generalization capabilities;
3. Multi-tier thinking budgets and complete tool-calling state management, providing flexible inference control mechanisms for agent engineering.

For developers, the next areas worth attention are: how to design appropriate thinking budget strategies for specific business scenarios, how to construct high-quality tool-calling histories, and how to maintain task precision under quantized weights. As long-horizon agent application scenarios continue to expand, architectural innovation and engineering optimization of underlying models will remain key directions for continuous evolution.