Large-Scale Sparse MoE Architecture and Long-Horizon Agent Development Practices

Large-Scale Sparse MoE Architecture and Long-Horizon Agent Development Practices

In recent years, Large Language Models (LLMs) have achieved significant breakthroughs in natural language understanding and generation. However, as application scenarios expand into complex software engineering, automated terminal operations, and multi-step task planning, traditional dense models face immense challenges regarding computational costs and long-context inference efficiency. In this context, large-scale sparse models utilizing the Mixture-of-Experts (MoE) architecture have gradually become an industry consensus. This article explores the architecture design, training strategies, and engineering deployment practices of an open-source MoE model with 2.8T total parameters and 104B activated parameters, specifically focusing on its application in long-horizon Agent tasks.

1. Architectural Requirements and MoE Advantages for Long-Horizon Agents

The core of a long-horizon Agent lies in the model’s ability to maintain instruction focus within an extended context window (e.g., at the 1 million Token level), continuously performing multi-turn tool calls, state tracking, and environmental feedback processing. This rigorously tests not only the model’s foundational cognitive capabilities but also its memory mechanisms and reasoning stability.

Adopting an MoE architecture with 2.8T total parameters and approximately 104B activated parameters per inference effectively balances model capacity and computational latency. When processing massive image-text inputs and ultra-long contexts, the sparse activation mechanism ensures that the model only invokes the expert subnetworks most relevant to the current Token. This significantly raises the upper limit for solving complex tasks without linearly increasing the computational burden.

2. Foundational Architecture Evolution: Deep Optimization of Sparse Attention Mechanisms

In ultra-long context scenarios, the computational and memory overhead of standard Multi-Head Attention (MHA) and its variant, Multi-head Latent Attention (MLA), grow quadratically. To break through this bottleneck, the new generation of models has undergone deep sparse reconstruction in its attention mechanisms.

1. Application of Gated DSA (Dilated Sparse Attention)

Traditional attention mechanisms often suffer from “attention dissipation” when facing million-level Tokens. This model replaces the previous 24 layers of Gated MLA with Gated Sparse Attention. The core idea of DSA is to introduce a dynamic routing mechanism. When computing the attention for each query Token, the model no longer traverses all historical Tokens; instead, it uses a lightweight gating network to dynamically select the 2,048 most critical historical Tokens to participate in the attention computation. This “hard sparsity” strategy not only significantly reduces computational complexity from O(N²) but also effectively reduces background noise interference in long-horizon tasks by forcing the model to focus on high-value historical information.

2. Collaborative Computation with Kimi Delta Attention

In addition to Gated DSA, the model integrates the Delta Attention mechanism in the other 69 network layers. Delta Attention further compresses the VRAM footprint of the KV cache through low-rank approximation or local window shifting of the attention matrix. The layered collaboration between DSA and Delta Attention enables the model to maintain a native input capacity for million-level Tokens while achieving extremely high throughput performance, providing solid foundational support for multi-turn tool calls.

3. Multi-Stage Joint Reinforcement Learning and Quantized Training

The model’s superior capabilities stem not only from architectural innovations but also from data and strategy design during the post-training phase.

1. Joint Reinforcement Learning Across Hundreds of Domains

After pre-training, the model employs a large-scale joint reinforcement learning post-training strategy. Unlike simple instruction fine-tuning, joint reinforcement learning spans hundreds of downstream application domains, including but not limited to static code analysis, terminal command execution feedback, 3D structure generation, and financial data extraction. By constructing multi-dimensional reward models, the model continuously optimizes its decision boundaries within the “Think-Act-Observe” loop. This strategy significantly improves the model’s fault tolerance and logical rigor in complex tool calling scenarios.

2. Inference Thinking Budget Mechanism

To accommodate application scenarios with varying latency sensitivities, the model introduces a three-tier thinking budget at the inference level: low, high, and max.

• Low: Suitable for simple intent recognition or single tool calls, returning results quickly.
• High: Balances deep reasoning and response time, suitable for routine multi-step code repairs or document retrieval.
• Max: Fully unleashes the model’s Chain of Thought potential, used to solve highly difficult software engineering marathon tasks or complex 3D CAD generation.

Developers can dynamically adjust the budget based on the specific business flow context, enabling fine-grained control over computational costs.

3. MXFP4 Weight Quantization

To reduce the storage and transmission pressure brought by the massive parameter count, both training and inference natively support MXFP4 (Microscaling FP4) weight quantization. By blocking weights and representing them as 4-bit floating-point numbers combined with fine-grained scaling factors, MXFP4 compresses the model size to an extremely small proportion while maximally preserving the gradient precision distribution. This avoids the catastrophic forgetting issues that traditional INT4 quantization often suffers in long-horizon tasks.

4. Core Application Scenarios and Capability Breakdown

Leveraging the aforementioned architectural and training advantages, the model has demonstrated strong engineering practical value in multiple complex Agent benchmark evaluations.

1. Code Development and Automated Engineering

In the software engineering domain, the model not only supports routine code completion but also possesses the foundational capability to act as a long-horizon code Agent.

• Terminal Operations: The model can understand terminal output streams and autonomously plan repair paths based on error messages, achieving a high score of 65.7 in Terminal-style evaluations.
• Code Repair and Marathon Tasks: Facing complex cross-file codebase modifications, the model can use the Max thinking budget alongside code retrieval tools to gradually locate bug root causes and generate patches. In high-difficulty evaluations like DeepSWE, the accuracy rate exceeded 75.5%.
• 3D Design and CAD Generation: The model breaks through text-only limitations, supporting native image-text input and spatial structure understanding. Combined with professional tool APIs, it can generate CAD instructions based on sketches or natural language descriptions, scoring as high as 92.7% in the tooled BenchCAD evaluation.

2. Knowledge Work and Multimodal Document Processing

In enterprise knowledge management scenarios, the model supports real-time web retrieval and deep parsing of complex office documents (including charts and layout information). It scored 64.4 in the OfficeQA Pro v2 evaluation, proving its ability to extract key business logic from unstructured data. Additionally, it scored 92.1 in BrowseComp, indicating it can serve as an efficient information retrieval Agent, autonomously completing cross-webpage information aggregation and summarization.

5. Engineering Deployment and API Integration Best Practices

For enterprise-level deployment, the model provides comprehensive backend compatibility solutions and interface specifications.

1. Inference Backend and API Compatibility

The quantized weights can be seamlessly deployed on industry-standard high-performance inference engines (such as frameworks supporting vLLM, SGLang specifications, and customized acceleration engines). At the service level, it natively provides interfaces compatible with mainstream LLM API specifications. Enterprise developers can directly reuse existing OpenAI/Anthropic SDK standards for business migration without extensively modifying underlying call logic.

2. Historical Message Management for Multi-Turn Tool Calls

In long-horizon Agent development, a critical technical detail is the maintenance of multi-turn tool call messages. This model requires retaining the complete “thinking process” and “tool call messages” across multi-turn conversations. Below is a standard pseudo-code example for constructing a multi-turn tool call context:

# Construct multi-turn Agent historical messages
messages = [
    {"role": "user", "content": "Please retrieve recent financial report data and extract core profit metrics."},
    # The model's thinking and tool call requests must be retained
    {
        "role": "assistant",
        "content": "I need to call the financial retrieval tool to get the latest financial report.",
        "tool_calls": [
            {
                "id": "call_001",
                "type": "function",
                "function": {"name": "search_financial_report", "arguments": "{\"year\": 2023}"}
            }
        ]
    },
    # The tool return results must be retained
    {
        "role": "tool",
        "tool_call_id": "call_001",
        "content": "{\"net_profit\": 5.2 billion, \"revenue\": 20.1 billion}"
    }
]
# In the next request, pass the complete messages list to the model
# The model generates the final analysis result based on the full historical context
response = inference_client.chat.completions.create(
    model="moe-pro-2.8t",
    messages=messages,
    thinking_budget="max" # Set thinking budget
)

Discarding the Chain of Thought or tool return results will cause the model to hallucinate or repeatedly call the same tool in long-horizon tasks. This is a trap that developers must avoid when building complex Agent workflows.

3. Open Source License Notes

It should be noted that this model adopts its specific commercial open-source license, rather than the standard Apache 2.0 or MIT licenses. Before integrating it into commercial products, enterprises must carefully review the specific terms regarding commercial authorization and open-source requirements for derivative works.

6. Conclusion and Outlook

From a technical perspective, the 2.8T parameter MoE model has successfully found an optimal balance between million-Token context and computational efficiency by introducing Gated Sparse Attention (DSA) and Delta Attention mechanisms. Combined with joint reinforcement learning across hundreds of domains and MXFP4 quantization technology, the LLM is no longer merely a “conversational machine,” but an “autonomous agent” capable of deep involvement in code engineering, 3D creation, and complex data analysis.

For developers and enterprises, when utilizing such high-performance LLMs to build long-horizon Agents, they should focus on the adaptability of inference engines, multi-turn context storage solutions (leveraging object storage or dedicated logging services to record ultra-long histories), and dynamic routing strategies for thinking budgets. As sparse architectures and reinforcement learning technologies continue to evolve, we expect to see more industrial-grade Agents that can autonomously take over complete software lifecycles and complex data pipelines.