Ultra-Large MoE Models for Long-Horizon Agents: Architecture Evolution and an Implementation Guide

As large language models evolve deeper into agentic applications, they need more than strong foundational understanding: they must maintain stable long-range reasoning across massive context windows and execute tool calls with high accuracy. A recently open-sourced ultra-large mixture-of-experts (MoE) model—with 2.8 trillion total parameters and 104 billion activated parameters—introduces significant innovations in both architecture design and post-training strategy, particularly in sparse attention and joint reinforcement learning. These advances offer a new solution for long-horizon agent scenarios such as large-scale code engineering and complex knowledge work.

This article sets aside product-specific details and takes a deep look at the core technical elements of this class of ultra-large MoE agent models—from the underlying architecture and inference scheduling strategies to engineering integration guidelines.

1. Architectural Innovation: From Dense to Sparse Attention (DSA)

When processing ultra-long contexts on the order of one million tokens, the attention computation of a conventional Transformer architecture scales explosively—the root cause of GPU memory blowups and rising inference latency. The new generation of models addresses this by fundamentally restructuring the attention layers.

1.1 Hybrid orchestration of Gated DSA and Delta Attention

Whereas earlier versions used Gated MLA (Multi-head Latent Attention) throughout, the new architecture replaces it with Gated DSA (Gated Sparse Attention) in 24 layers, combined with 69 layers of Kimi Delta Attention. The essence of this hybrid design is “allocating compute only where it is needed”:

• Sparse token selection: In each DSA layer, the model dynamically selects only the 2,048 most relevant historical tokens for attention computation. This breaks the redundancy of global attention: when working through a long codebase or a large set of retrieved documents, the model can “focus on the key clues,” much like a human, instead of being distracted by irrelevant context.
• Improved compute efficiency: A gating mechanism controls the attention distribution, substantially reducing KV cache memory consumption during inference while increasing decoding throughput in long-text scenarios.

1.2 Sparse expert activation in the MoE design

The model pairs 2.8T total parameters with only 104B activated parameters. In each forward pass, roughly 3.7% of the expert networks fire. This design preserves enormous knowledge capacity while keeping actual inference compute at the tens-of-billions level—an effective balance between model capability and deployment cost.

2. Building Long-Horizon Agent Capabilities: Post-Training and Joint Reinforcement Learning

Once a model has solid foundational ability, the core challenge of post-training is adapting it to complex, multi-step agent tasks such as terminal operations, software engineering fixes, and CAD drawing generation.

2.1 Joint reinforcement learning across hundreds of domains

Traditional SFT (supervised fine-tuning) tends to lock models into “imitating trajectories,” leaving them with limited ability to generalize and explore in unseen environments. In the post-training stage, this model introduces a joint reinforcement learning (RL) mechanism covering hundreds of downstream domains.

• Multi-task objective alignment: By defining reward functions across agent scenarios such as code repair, web research, and financial research, the model learns to plan steps dynamically in complex environments, call external tools, and self-correct when errors occur.
• Long-horizon planning: RL training enables the model to maintain goal consistency across dozens of dialogue turns—critical for software engineering tasks that require multi-round debugging.

2.2 Native multimodal input

Long-horizon agents must handle more than plain text; they also need to interpret UI screenshots and chart data. The model supports native image-and-text input, directly encoding visual information and aligning it with text tokens. This provides natural visual grounding for tasks such as terminal operations (recognizing UI elements) and 3D design (understanding CAD sketches).

3. Inference Scheduling: Multi-Level Thinking Budgets

Real-world workloads differ enormously in complexity. A simple document summary should not consume the same inference compute as an intricate code refactoring. The model therefore exposes three thinking-budget tiers—Low, High, and Max—at the inference API layer:

• Low: Suited for quick Q&A and simple information extraction. The model responds immediately, minimizing first-token latency.
• High: Suited for typical web research, office document processing, and multi-turn conversations. The model performs moderate internal reasoning before answering.
• Max: Designed for high-difficulty code refactoring, deep financial analysis, and multi-step tool calling. The model generates a deep chain of thought and thoroughly weighs possible approaches before taking action.

Developers can configure the thinking budget dynamically through an API parameter, striking the optimal balance between system response latency and task success rate.

4. Engineering Practices and a Deployment Guide

For enterprises and developers, integrating a model of this scale into existing business workflows requires attention to quantized deployment, API compatibility, and tool-calling protocols.

4.1 MXFP4 weight quantization and inference framework support

Given the memory pressure of 2.8T parameters, the model natively ships MXFP4 (4-bit floating-point) weights. Compared with traditional INT4 quantization, MXFP4 better preserves extreme values and weight distributions, reducing quantization precision loss. On the serving side, the model supports mainstream high-performance inference frameworks such as vLLM, SGLang, and TokenSpeed, and can run efficiently on multi-GPU clusters using tensor parallelism.

4.2 API compatibility and tool-calling best practices

The model provides interfaces compatible with mainstream API protocols (such as the OpenAI and Anthropic formats), which greatly reduces migration costs for existing systems. When handling multi-turn tool calls, however, developers must pay special attention to message-history retention.

During multi-step agent execution, the model continuously observes tool results and adjusts its next move. The following example illustrates a standard multi-turn tool-calling flow:

import requests
# Assume an endpoint compatible with the OpenAI protocol
api_url = "https://api.your-inference-platform.com/v1/chat/completions"
# Key point: multi-turn tool calls must preserve the full history of
# reasoning and tool-call messages
messages = [
    {"role": "user", "content": "Analyze the code structure of this repository and find potential dependency conflicts."},
    # Turn 1: model reasoning and tool-call request
    {"role": "assistant", "content": "Analyzing...", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "run_terminal_cmd", "arguments": "{\"cmd\": \"tree -L 2\"}"}}]},
    # Turn 1: tool result
    {"role": "tool", "tool_call_id": "call_1", "content": "Directory structure..."},
    # Turn 2: model continues reasoning based on the history
    {"role": "assistant", "content": "Based on the directory structure, I will inspect the dependency file.", "tool_calls": [{"id": "call_2", "type": "function", "function": {"name": "run_terminal_cmd", "arguments": "{\"cmd\": \"cat package.json\"}"}}]},
    # Turn 2: tool result
    {"role": "tool", "tool_call_id": "call_2", "content": "Dependency list..."}
]
# Set the Max thinking budget for complex code tasks
payload = {
    "model": "your-large-moe-model-id",
    "messages": messages,
    "thinking_budget": "max",
    "tools": [{"type": "function", "function": {"name": "run_terminal_cmd", "description": "Execute a command in the terminal"}}]
}
response = requests.post(api_url, json=payload)

Key caveat: When assembling multi-turn conversation context, never truncate or filter out earlier tool_calls messages and their corresponding tool role messages. The deep reasoning of a long-horizon agent model depends heavily on having the complete execution trajectory as context.

5. Evaluation Dimensions and Benchmark Results

According to the officially released evaluation data, the model performs strongly on several highly challenging agent benchmarks (using the Max thinking tier by default):

• Terminal and software engineering: 65.7 on Terminal-Bench 4.0, 51.7 on SWE Marathon v1.1, and 75.5 on DeepSWE v1.1. These results indicate that the model reaches an engineering-grade usability standard for understanding complex codebases, making cross-file changes, and executing terminal commands.
• Knowledge work and retrieval: 92.1 on BrowseComp and 64.4 on OfficeQA Pro v2, showing that its long-context advantages in extracting information from vast numbers of web pages and understanding complex document formats translate effectively into real tasks.
• Vertical-domain creation: 92.7 on the tool-augmented BenchCAD benchmark, demonstrating high execution precision in vertical fields such as 3D design that demand strong spatial imagination and code control.

Conclusion

By combining sparse attention (DSA) with large-scale joint reinforcement learning, this 2.8T-parameter MoE model breaks through the bottlenecks that have limited long-horizon agents: computational efficiency and planning over complex tasks. For developers, successful integration comes down to two priorities: using the multi-level thinking budgets to tune latency, and rigorously preserving the integrity of multi-turn tool-calling context in the engineering implementation.

For enterprise-scale deployments, cloud providers such as Baidu AI Cloud offer high-performance container platforms, object storage, and monitoring and alerting services that can further improve the deployment efficiency and operational stability of ultra-large MoE models—accelerating the adoption of long-horizon agents across vertical business scenarios.