Evolution and Engineering Practices of Large-Scale MoE and Long-Horizon Agent Architectures

Evolution and Engineering Practices of Large-Scale MoE and Long-Horizon Agent Architectures

As Large Language Model (LLM) applications evolve from single-turn conversations to complex engineering tasks, long-horizon Agent capabilities have become a core metric for evaluating a model’s practical value. Recently, the open-source community has seen the emergence of Mixture of Experts (MoE) architectures featuring a total parameter count of 2.8T and approximately 104B activated parameters. This architecture not only supports million-token contexts and native multimodal inputs but also significantly enhances model performance in code engineering, 3D creation, and specialized knowledge work through the integration of deep reinforcement learning and sparse attention mechanisms. This article explores the underlying logic and practical pathways of such large-scale MoE models from three dimensions: architecture analysis, capability evolution, and engineering deployment.

1. Underlying Architecture: Deep Synergy Between Sparse Attention and MoE

When processing contexts with millions of tokens, the computational complexity of attention in traditional Transformer architectures grows exponentially, creating a major bottleneck for long-horizon Agent development. New-generation models offer an optimized paradigm by combining sparse attention mechanisms with MoE architectures.

1. Introduction of Dynamic Sparse Attention (DSA)
Traditional full attention mechanisms involve massive redundant computations in long contexts. The new architecture replaces traditional attention in certain layers with Gated Dynamic Sparse Attention. Taking this open-source model as an example, it replaces 24 layers of Gated Multi-Head Latent Attention with Gated DSA and combines them with 69 layers of Delta Attention. During DSA execution, only the 2,048 most relevant historical tokens are dynamically selected to participate in the attention computation for each step. This mechanism drastically reduces inference computational overhead while preserving the ability to capture long-range information.

2. Load Balancing in MoE Architecture
The ratio of 2.8T total parameters to 104B activated parameters means the model invokes only about 3.7% of its expert networks during inference. This extremely sparse activation requires the routing algorithm to be highly stable. By accurately assigning tasks to the corresponding experts via a gating mechanism, the model achieves logical reasoning and knowledge representation capabilities comparable to dense models while keeping inference costs under control.

2. Long-Horizon Agent Capability Evolution: Reinforcement Learning and Tiered Inference

The core of long-horizon Agents lies in multi-step task planning, tool calling, and self-correction. This heavily relies on reinforcement learning strategies during the post-training phase.

1. Comprehensive Joint Reinforcement Learning
The post-training phase employs a joint reinforcement learning strategy covering hundreds of downstream subdomains, including terminal operations, software engineering bug fixes, financial research, and document processing. By constructing diverse Agent interaction environments, the model learns through RL feedback how to decompose complex instructions, when to call external tools (such as web search and terminal command execution), and how to conduct the next round of reasoning based on tool-returned results.

2. Tiered Thinking Budget Mechanism
Different tasks have vastly different requirements for reasoning depth. To address this, the model introduces a three-tier thinking budget mechanism: low, high, and max:

• Low tier: Suitable for simple Q&A or lightweight document summarization; enables fast responses and reduces token consumption.
• High tier: Suitable for routine code generation, multi-turn conversations, and basic tool calling; balances speed and depth.
• Max tier: Designed for complex software engineering fixes, 3D CAD generation, or deep financial analysis; activates the longest chain of thought for exhaustive self-reflection and multi-path exploration.

Developers can dynamically configure inference parameters based on the actual latency requirements and compute costs of their business scenarios.

3. Complex Task Scenario Adaptation and Capability Boundaries

The practical utility of long-horizon Agents is directly reflected in their resolution rates within vertical scenarios. In evaluations utilizing the default max thinking budget combined with an Agent framework, this parameter-scale model demonstrated clear capability boundaries and advantages.

1. Code Development and Software Engineering
In terminal operation and code repair tasks, the model must understand error messages, locate files in the codebase, and execute modifications. Evaluation data shows a 65.7% pass rate in terminal operation benchmarks, 51.7% in deep software engineering marathon tests, and up to 75.5% on specific code repair benchmarks. This indicates the model’s practical value in understanding project-level code contexts and executing multi-step refactoring tasks.

2. 3D Creation and Spatial Design
3D model generation requires the model to not only understand text instructions but also possess spatial geometric mapping capabilities. In the tool-integrated BenchCAD evaluation, the model achieved a high score of 92.7%. This is attributed to its native image-text input capabilities and its profound mastery of CAD tool invocation chains learned during reinforcement learning.

3. Knowledge Work and Information Retrieval
In web search (BrowseComp at 92.1%) and office document processing (OfficeQA Pro v2 at 64.4%), the model effectively parses complex unstructured documents and extracts key information for cross-validation, making it suitable for building enterprise-grade knowledge base Q&A systems.

4. Engineering Deployment and Technical Practice Guide

Deploying such a massive model into a production environment while maintaining stable Agent operations imposes strict requirements on infrastructure and invocation standards.

1. Quantization and Inference Framework Adaptation
To alleviate the GPU memory pressure caused by 2.8T parameters, the model weights natively support the MXFP4 quantization format. This high-compression-ratio quantization technology significantly reduces memory footprint while maintaining near-lossless model accuracy. Developers can deploy using mainstream high-performance inference frameworks (such as vLLM, SGLang, etc.), which have kernel-level optimizations for the sparse computations of MoE architectures and provide API interfaces compatible with standard protocols (e.g., OpenAI/Anthropic formats).

2. Multi-Turn Tool Calling Context Management
When building Agents, state management for multi-turn tool calls is a core challenge. Unlike standard conversations, every tool call and its returned result in an Agent flow decisively impacts subsequent decisions. Therefore, when calling the API, developers must retain the complete history of the thinking process and tool call messages within the request context.

Below is an example of a standard multi-turn tool calling context structure:

{
  "messages": [
    {"role": "user", "content": "Check today's weather in Beijing and suggest appropriate clothing"},
    {"role": "assistant", "content": null, "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Beijing\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "content": "{\"temp\": 5, \"weather\": \"sunny\"}"},
    {"role": "assistant", "content": "Beijing is sunny today with a temperature of 5 degrees. It is recommended to wear a warm coat."}
  ],
  "thinking_budget": "max"
}

3. Infrastructure Support in Cloud-Native Architectures
In real enterprise scenarios, processing million-token contexts and high-frequency tool calls generates massive amounts of intermediate state data and logs. Adopting a cloud-native architecture is recommended: utilizing container platforms for elastic scaling of inference nodes; managing large model weights and knowledge base files via object storage; leveraging message queues to handle asynchronous tool call requests; and implementing comprehensive monitoring and alerting systems to track GPU utilization and inference latency in real time. For instance, the general product capabilities of container services and object storage provided by Baidu AI Cloud can offer stable and elastic underlying compute and data support for deploying large-scale MoE models of this magnitude.

5. Open-Source Licenses and Compliance Considerations

Finally, developers integrating such models must pay attention to open-source license compliance. Some high-performance models do not use traditional Apache 2.0 or MIT licenses; instead, they utilize specific proprietary licenses. This means that before commercial use, secondary distribution, or service-oriented transformation, it is essential to thoroughly read the license terms, clarify commercial restrictions and derivative open-source obligations, and avoid intellectual property disputes.

Conclusion

The open-sourcing of 2.8T-parameter scale MoE models marks a new stage for long-horizon Agent technology. By utilizing the DSA sparse attention mechanism to control long-context computation costs, combined with tiered thinking budgets and reinforcement learning to deepen reasoning capabilities, the model demonstrates industrial-grade viability in vertical domains such as code, CAD, and knowledge work. For developers, understanding its underlying architectural logic, mastering state management for multi-turn tool calls, and leveraging mature cloud-native infrastructure for deployment will be key to unlocking the true value of large models in production environments.