Deep Dive into a 2.8T Open-Source MoE Model: Long-Horizon Agent Architecture and Engineering Practices

As large model parameter scales grow exponentially, the Mixture-of-Experts (MoE) architecture has become the industry consensus for balancing computational costs with model performance. Recently, an open-source community released an ultra-large MoE model with a total parameter count of 2.8T and only 104B activated parameters during a single inference pass. This model not only supports a million-token context window and native multimodal inputs but is also deeply optimized for “long-horizon Agent” tasks at the underlying architecture and post-training stages. Stripping away specific product attributes, this article analyzes the technical route of such models across four dimensions: core technical architecture, long-horizon task capabilities, underlying algorithm optimizations, and enterprise engineering practices.

1. Ultra-Large MoE Architecture and Long-Horizon Agent Foundations

When building long-horizon Agents, models must possess continuous context memory, multi-turn tool calling, and complex logical reasoning capabilities. The 2.8T parameter MoE model utilizes a sparse activation mechanism to maintain a massive knowledge capacity while keeping the computational complexity of a single inference pass at the 104B parameter level, significantly reducing inference latency.

1.1 Native Multimodality and Million-Token Context

Long-horizon Agent interaction scenarios typically involve web parsing, code repository reading, and long document analysis. This model natively supports a 1M token context length and can directly process mixed image-text inputs. This means that during multi-turn interactions, the Agent can carry the complete system prompt, historical conversation logs, and even the structural information of an entire code repository for inference, avoiding the context forgetting issue caused by traditional sliding window truncation.

1.2 Tiered Thinking Budget Mechanism

To address Agent tasks of varying complexities, the model introduces a three-tier thinking budget mechanism: Low, High, and Max. This is a resource-aware inference strategy:

• Low tier: Suitable for simple intent recognition, daily conversations, or rapid tool calls, achieving low-latency responses.
• High tier: Suitable for multi-step code repairs, document information extraction, and other tasks requiring moderate logical chains.
• Max tier: Designed for deep software engineering, complex financial research, or terminal OS-level long-horizon tasks. It activates the longest reasoning chain, sacrificing some latency for high accuracy.

2. Long-Horizon Agent Task Scenarios and Technical Analysis

The core of a long-horizon Agent lies in the closed loop of “task decomposition - tool calling - result feedback - state update.” This model demonstrates significant long-horizon task support capabilities across three core domains: code development, 3D creation, and knowledge work.

2.1 Code Development and 3D Creation (Software Engineering-Level Agents)

In software engineering scenarios, an Agent must not only generate code but also possess terminal operation, code repair, and CAD generation capabilities. Take terminal operations as an example: the Agent needs to understand system error messages, autonomously plan repair steps, invoke Shell commands or file editing tools, and adjust its strategy based on system feedback after execution. In 3D design tasks, combined with CAD generation tools, the model must understand spatial geometry descriptions and translate them into parametric modeling instructions. This requires the model to have robust structured data output capabilities and strict tool protocol adherence.

2.2 Knowledge Work and Toolchain Integration

In knowledge work scenarios such as financial research and office document processing, the model must combine web search with local document parsing tools. The difficulty for long-horizon Agents in such tasks is that the data returned after tool calls is often extremely lengthy (e.g., text converted from a dozens-page financial report PDF). Leveraging the 1M token context, the model can temporarily store the complete results returned by the tool within the context window and then extract key information through the attention mechanism to form the final analysis report.

3. Underlying Algorithm Breakthroughs: Sparse Attention and Joint RL

Supporting efficient inference for 1M tokens simply by stacking compute power is impractical. The model features significant innovations in its attention mechanism and post-training stage.

3.1 Hybrid Architecture of Gated DSA and Delta Attention

Traditional Transformer attention mechanisms cause compute and memory consumption to grow quadratically under long contexts. This model applies a heterogeneous design to the attention layers:

• The original 24 layers of Gated MLA (Multi-Head Latent Attention) were replaced with Gated DSA (Delta Sparse Attention). The DSA mechanism uses a dynamic routing algorithm to select only the 2,048 most relevant historical tokens to participate in the attention calculation for each step. This sparse connection drastically reduces computational overhead while preserving the ability to capture long-range dependencies.
• Combined with a 69-layer Kimi Delta Attention architecture, the model achieves information aggregation of different granularities at different layers, forming a hybrid attention network that separates high and low-frequency features.

3.2 Large-Scale Joint Reinforcement Learning Post-Training

The leap in the model’s capabilities is inseparable from reinforcement learning (RL) during the post-training stage. The model employs a joint RL strategy covering hundreds of downstream domains. This means that during the RL phase, the model is not solely optimized for code or conversation; instead, a single training batch mixes various tasks such as tool calling, logical reasoning, and multimodal understanding. This joint RL mechanism prompts the model to learn cross-domain general problem-solving paradigms, greatly enhancing its zero-shot generalization capabilities.

4. Enterprise Deployment and Engineering Best Practices

For developers and enterprises, the key challenge is how to efficiently deploy ultra-large MoE models and integrate them into existing business workflows.

4.1 MXFP4 Low-Bit Quantization and Inference Acceleration

Given the massive 2.8T parameter size, the model offers an MXFP4 weight quantization version. MXFP4 is a microscaling floating-point format that, compared to traditional FP16 or INT8, heavily compresses the model size and reduces memory bandwidth pressure with minimal precision loss. At the deployment level, the quantized weights natively support mainstream inference frameworks (such as vLLM, SGLang, etc.) and provide API interfaces compatible with OpenAI/Anthropic standards.

# Example: Calling the Max thinking budget via an OpenAI-compatible API
import openai
client = openai.Client(
    base_url="http://localhost:8000/v1",
    api_key="your_custom_key"
)
response = client.chat.completions.create(
    model="2.8t-moe-model",
    messages=[
        {"role": "system", "content": "You are a professional code repair Agent."},
        {"role": "user", "content": "Help me analyze and fix the following error: ..."}
    ],
    # Specify the thinking budget as Max to activate long-horizon reasoning capabilities
    extra_body={"thinking_budget": "max"} 
)
print(response.choices[0].message.content)

4.2 Context Management for Multi-Turn Tool Calling

In complex Agent frameworks (such as coding agents similar to Claude Code), multi-turn tool calls can easily lead to context window overflow or state loss. Best practices require: in the API request, the complete history of the thinking process and tool call messages must be preserved. This means you should not pass only the final result, but input the complete chain of tool_calls and tool_response as context. Combined with the sparse characteristics of Gated DSA, the model can efficiently extract key states from these lengthy historical records, ensuring the coherence of long-horizon tasks.

4.3 Scheduling and Storage in Cloud-Native Environments

In an enterprise-grade production environment, if integrating with infrastructure like Baidu AI Cloud, it is recommended to use a container platform for elastic compute scheduling, paired with object storage services to store massive model weight files and evaluation datasets. Given the expert parallelism characteristics of MoE models, high-performance networks and GPU clusters on the cloud platform can be utilized for optimized deployment. Simultaneously, integrating logging services and a monitoring/alerting system enables full-chain tracing of long-horizon Agent task execution times, tool call success rates, and API latency.

5. Conclusion and Outlook

The 2.8T total parameter, 104B activated MoE architecture model represents a significant trend in the current development of long-horizon Agent technology: solving the long-context computational bottleneck through heterogeneous sparse attention (Gated DSA + Delta Attention), shaping cross-domain generalization capabilities via large-scale joint reinforcement learning, and reducing enterprise adoption barriers using MXFP4 quantization.

For developers, mastering the scheduling strategy of multi-tiered thinking budgets, standardizing context management for multi-turn tool calls, and optimizing deployment with cloud-native architectures will be critical to building the next generation of efficient, intelligent long-horizon Agents. As the open-source ecosystem continues to flourish, such models will unleash immense potential in high-value scenarios like automated software engineering and complex data analysis.