Ultra-Large-Scale MoE Architecture and Long-Horizon Agent Technology: Evolution and Engineering Practice

In recent years, as large language models have demonstrated remarkable potential on complex tasks, industry demand has shifted from simple text dialogue to the “long-horizon agent” stage—where models must autonomously plan, invoke tools, and execute multi-step operations. An open-source large model project recently released its latest generation of weights, with 2.8T total parameters and 104B activated parameters. The model not only adopts a sparse MoE (Mixture of Experts) design but also achieves deep optimization in the underlying attention mechanism, reinforcement learning post-training, and engineering deployment. This article sets aside specific product attributes and, from a general technical perspective, analyzes the core technical evolution and engineering practices of models at this scale for long-horizon agent tasks.

1. Underlying Architecture Design for Long-Horizon Agents

Balancing Ultra-Large Parameters with Sparse Activation

When handling long-horizon agent tasks—such as complex terminal operations, repository-level code fixes, or 3D design—models require extremely high knowledge capacity to understand multi-domain instructions and context. However, the computational cost of dense large models grows exponentially with parameter count. The MoE architecture adopted by this model guarantees the upper bound of knowledge storage through 2.8T total parameters while activating only 104B parameters during inference, effectively controlling per-forward-pass computational latency and memory footprint. This “large capacity, low overhead” design is the foundation for completing complex agent workflows within acceptable timeframes.

Native Multimodality and Long-Context Support

Long-horizon agent execution is often accompanied by multi-source heterogeneous data input. The model supports native image-text input and a context window of up to 1 million tokens. Native image-text capability means the model does not need to rely on additional visual encoders when analyzing UI screenshots or parsing charts, reducing system complexity. The 1M-token context window allows the model to receive, within a single session, the complete historical logs of an entire code repository and long structured data returned by multiple rounds of tool calls (such as API responses and database query results), avoiding the “task amnesia” phenomenon caused by context truncation in traditional models.

2. Engineering Breakthroughs in Attention Mechanisms and Thinking Budgets

Deep Evolution of Sparse Attention Mechanisms

In ultra-long-context scenarios, the computational complexity of standard attention (O(N^2)) becomes a severe bottleneck. The new generation model makes a key replacement at the underlying architecture level: replacing Gated MLA in part of the network layers (e.g., 24 layers) with Gated DSA (Delta Sparse Attention), combined with 69 layers of Kimi Delta Attention. The core idea of DSA is to dynamically select the most informative historical tokens for computation at each step (for example, selecting 2,048 tokens each time). This mechanism substantially reduces memory read/write pressure, enabling the model to capture long-range dependencies while maintaining high inference throughput when processing millions of tokens.

Dynamic Scheduling of Tiered Thinking Budgets

Agent tasks of different complexity have vastly different requirements for inference depth. Simple text summarization does not require complex reasoning, whereas resolving a high-difficulty bug demands exhaustive logical chain deduction. The new generation model provides three thinking budget tiers at the inference interface level: “Low,” “High,” and “Max.” When building applications, developers can dynamically allocate compute based on task type—for example, using Low for routing and dispatch decisions, High for multi-step planning, and Max for complex code refactoring. This not only optimizes resource utilization but also reduces API call costs for enterprise applications.

3. Reinforcement Learning Post-Training and Complex Tool Calling

Joint Reinforcement Learning for Improved Task Generalization

Traditional supervised fine-tuning (SFT) tends to trap models in a shallow pattern of “imitating tool-call formats,” lacking the ability to dynamically adjust strategies in real environments. In the later stage of training, this model introduces a joint reinforcement learning (RL) mechanism covering hundreds of downstream domains. Through trial-and-error and reward optimization in real code execution environments, terminal sandboxes, and web retrieval environments, the model learns not only “how to call tools” but also “when to call them and how to degrade gracefully after a call fails.” This post-training strategy is key to the model’s ability to handle high-barrier knowledge work such as software engineering and financial research.

Session Management for Multi-Step Tool Calling

In multi-turn tool-calling scenarios, developers must properly manage message history. According to engineering best practices, multi-turn tool calls must preserve the complete “historical reasoning process” and “tool call messages,” because when executing step N, the model often needs to review the reasoning logic and intermediate artifacts from step N-3.

Below is a simplified example of multi-turn tool calling compatible with mainstream OpenAI/Anthropic API specifications:

import requests
import json
# Assume an inference engine compatible with mainstream specifications
API_URL = "http://localhost:8000/v1/chat/completions"
# Build a complete context including historical reasoning and tool calls
messages = [
    {"role": "user", "content": "Please check the repository's current commit history and fix the most recent syntax error."},
    # Preserve the model's reasoning process and tool call request
    {"role": "assistant", "content": "I need to run git log first to check the commit history.", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "execute_bash", "arguments": "{\"cmd\": \"git log -5\"}"}}]},
    # Return the tool execution result
    {"role": "tool", "tool_call_id": "call_1", "content": "commit 1234: syntax error in main.py..."},
    # Continue with the next reasoning step
    {"role": "assistant", "content": "The error has been located. Now applying the fix.", "tool_calls": [{"id": "call_2", "type": "function", "function": {"name": "edit_file", "arguments": "{\"file\": \"main.py\", \"changes\": \"...\"}"}}]}
]
payload = {
    "model": "ultra-moe-agent",
    "messages": messages,
    "thinking_budget": "max", # Set the highest thinking budget for complex code fixes
    "tools": [{"type": "function", "function": {"name": "execute_bash", "parameters": {"type": "object", "properties": {"cmd": {"type": "string"}}}}}]
}
response = requests.post(API_URL, json=payload)
print(json.loads(response.text))

4. Deployment Optimization and Enterprise Application Scenarios

Quantization Techniques and High-Performance Inference Frameworks

Direct deployment of a 2.8T-scale model places extremely high demands on GPU memory. The new generation model natively supports MXFP4 weight quantization. Through group-wise quantization and scaling factors, MXFP4 compresses model size to one quarter while preserving the numerical precision of the original model to the greatest extent. Combined with mainstream open-source inference acceleration frameworks (such as vLLM and SGLang), enterprises can achieve high-concurrency deployment of this model on standard GPU clusters. At the cloud infrastructure layer, this typically requires container platforms for elastic scaling, object storage to host model weights, and logging and monitoring systems to track inference latency and tool-call success rates in real time.

Typical Application Scenario Mapping

Based on the technical foundation described above, such ultra-large-scale MoE models show significant potential in the following scenarios:

1. Automated software engineering: Combining terminal operations and code retrieval capabilities to act as an autonomous software engineer, enabling a full closed loop from issue analysis and code localization to patch generation and automated testing.
2. 3D design and CAD generation: Understanding complex three-dimensional spatial design instructions and generating or modifying CAD drawings through tool calls to assist industrial design.
3. Deep knowledge retrieval and office automation: In financial research and legal document review, extracting key information and generating structured reports through multi-step web retrieval and document parsing.

5. Open-Source License Compliance Notes

It is worth noting that although such models open their weights for community research and commercial use, they typically adopt a proprietary license rather than the traditional Apache 2.0 or MIT license. Before integrating such models into commercial products, developers must carefully review the license terms, especially restrictions on commercial use, open-source requirements for derivative models, and usage-scenario limitations, to ensure compliance of technology applications.

Conclusion

From the deployment of 2.8T-parameter MoE architectures to the introduction of DSA sparse attention mechanisms, and further to the stimulation of agent capabilities through joint reinforcement learning, large models are rapidly evolving toward “intelligent agents with long-horizon planning and execution capabilities.” For developers and enterprises, understanding these underlying technical principles, mastering engineering conventions for multi-turn tool calling, and deploying efficiently with cloud-native infrastructure will be key to unlocking the real value of large models in complex business scenarios.