Deep Dive into the 2.8T Sparse MoE Model: Architecture and Long-Term Agent Practices

Deep Dive into the 2.8T Sparse MoE Model: Architecture and Long-Term Agent Practices

In early November, the open-source community witnessed a major update: a new generation large language model was officially open-sourced. The model features a total parameter count of 2.8T with a sparse Mixture of Experts (MoE) architecture, activating approximately 104B parameters. It natively supports a 1-million token ultra-long context length and multimodal inputs (text and image). Notably, through large-scale joint reinforcement learning post-training, the model focuses its capabilities on “long-term Agent” tasks, providing a new foundational infrastructure for complex software engineering, 3D design, and enterprise-level knowledge work.

This article will provide an in-depth analysis of the model’s technical characteristics from four dimensions: technical architecture evolution, core application scenarios, benchmark performance, and engineering deployment practices, offering a reference for developers looking to implement this technology.

1. Core Technical Architecture and Training Innovations

When supporting million-level contexts and long-term Agent tasks, large models face significant challenges, primarily the exponential explosion in attention computation complexity and the stability of multi-step reasoning. This model introduces substantial innovations in its underlying architecture and training paradigm:

1. Hybrid Sparse Attention Mechanism
In traditional Transformer architectures, attention computation complexity grows quadratically with sequence length. This model deeply modifies the foundation by replacing 24 layers of Gated MLA (Multi-Head Latent Attention) with Gated DSA (Delta Sparse Attention), combined with 69 layers of a common linear attention variant (Delta Attention).
The core advantage of this hybrid architecture is “dynamic sparse selection.” In the DSA layers, the model selects only the 2,048 most relevant historical tokens to participate in attention computation. This mechanism significantly reduces VRAM usage and computational latency in long-context scenarios. It also effectively filters out redundant interference in long-term tasks, ensuring the model maintains precise information extraction capabilities even at a depth of one million tokens.

2. Joint Reinforcement Learning and Quantization Strategy
The model’s post-training phase employs a joint reinforcement learning strategy covering hundreds of downstream domains. This means the model does not simply fit into a single vertical domain; instead, it enhances the generalization of tool calling, logical reasoning, and instruction following through cross-domain task composition. Furthermore, the model weights utilize the MXFP4 quantization format, a cutting-edge quantization scheme that significantly reduces VRAM usage while maintaining model accuracy. This makes it possible to deploy a 2.8T-scale model on consumer-grade or enterprise-grade mid-sized compute clusters.

2. Agent Capability Evolution for Long-Term Tasks

A “long-term Agent” refers to an intelligent agent capable of autonomously decomposing goals, spanning multiple steps, continuously calling external tools, and processing intermediate feedback. The model demonstrates specific design considerations for long-term tasks in the following two core scenarios:

1. Software Development and 3D Creation Engineering
In software engineering, developers often require models with global codebase understanding capabilities and closed-loop abilities for terminal execution feedback. This model supports long-term code Agent tasks, combining terminal operations, code repairs, and other tools to complete multi-step development workflows. For example, upon receiving an instruction to “fix a legacy bug in a specific module,” the model can autonomously search the codebase, locate the anomaly, generate a patch, invoke the terminal to run test cases, and perform secondary fixes based on error messages.
Additionally, the model showcases robust tool synergy in 3D design, with official evaluations covering CAD generation tasks. It can understand complex spatial geometry instructions and generate or modify 3D models by calling underlying CAD engine APIs, significantly shortening the cycle for industrial design and digital twin construction.

2. Knowledge Work and Multi-Dimensional Tool Synergy
In daily enterprise operations, processing massive amounts of unstructured data is a major pain point. The model natively supports tasks such as web search, financial research, and office document processing (e.g., complex table parsing, long report extraction). To accommodate varying business needs regarding cost and depth, the model offers three tiers of “thinking budget” during inference: Low, High, and Max.

• Low tier: Suitable for simple Q&A and single tool calls, prioritizing low latency.
• High tier: Suitable for routine knowledge retrieval and document summarization.
• Max tier: Unlocks deep reasoning capabilities for handling complex long-term tasks that require multi-step planning and multi-tool synergy. This tiered mechanism grants enterprises more flexible compute scheduling strategies.

3. Benchmark Performance and Technical Analysis

According to officially released data (using the Max thinking tier by default, with some evaluations executed on mainstream industry Agent frameworks), the model has achieved significant breakthroughs in several high-difficulty benchmarks:

• Terminal-Bench 4.0: 65.7: Demonstrates the model’s command generation and error fixing capabilities in a terminal environment.
• SWE Marathon v1.1: 51.7 / DeepSWE v1.1: 75.5: These software engineering marathon tests require the model to perform multiple rounds of modifications and debugging in ultra-long codebases. The significant score improvements validate the stability of its long-term code Agent.
• BrowseComp: 92.1: Web search and information extraction capabilities reach an extremely high level, able to precisely locate required information within complex DOM trees.
• BenchCAD (with tools): 92.7: Excellent performance in CAD generation with tool calls validates its execution capability for 3D design instructions.
• OfficeQA Pro v2: 64.4: Demonstrates strong long-text understanding and logical deduction capabilities when facing complex nested office documents.

These metrics indicate that current open-source models are already capable of supporting practical industrial-grade production on the technical track of “tool calling + long-term reasoning.”

4. Engineering Deployment and API Integration Best Practices

For developers and enterprises, model implementation relies not only on capabilities but also on engineering usability. The model provides MXFP4 weight quantization and natively supports mainstream high-performance inference frameworks like vLLM and SGLang. For cloud deployment, developers can leverage GPU container instances and object storage services provided by Baidu AI Cloud to achieve rapid model weight loading and build highly available inference services.

Regarding API compatibility, the model offers interfaces highly compatible with OpenAI and Anthropic. Developers can seamlessly integrate it using standard SDKs:

import openai
# OpenAI-compatible API call example
client = openai.OpenAI(
    base_url="https://your-inference-endpoint/v1",
    api_key="your-api-key"
)
response = client.chat.completions.create(
    model="new-moe-2.8t",
    messages=[
        {"role": "user", "content": "Analyze this financial report and call the search tool to compare profit margins within the same industry"}
    ],
    # Thinking budget can be configured based on task complexity
    extra_body={"thinking_budget": "max"} 
)
print(response.choices[0].message.content)

Key Integration Note: Multi-Turn Tool Calling State Management
When building long-term Agents, context management for multi-turn tool calling is a core challenge. Developers must note that in every interaction request with this model, the complete historical thinking process and tool calling messages must be retained. This means that not only the User and Assistant conversations need to be passed, but the model’s internal chain of thought and the results returned by tool executions must also be fully appended to the context in chronological order. If historical call records are truncated, the model may lose its judgment on the current state, leading to infinite loops or incorrect instruction outputs. Given that the context can rapidly expand, it is recommended to introduce message queues and high-performance caching components (such as the message queue and Redis services provided by Baidu AI Cloud) into the system architecture to ensure context consistency for long-term tasks.

5. Compliance and Licensing

Finally, it is important to note that this model does not use the common Apache 2.0 or MIT licenses found in the open-source community. Instead, it operates under a proprietary license agreement. Before commercially integrating it into their product lines, enterprises must have their legal and technical teams carefully review the scope of constraints in the licensing terms to ensure their application scenarios comply with the open-source agreement.

Conclusion

The new generation open-source MoE model, featuring 2.8T total parameters, million-token context support, and native multimodal capabilities, pushes open-source large model Agent capabilities to new heights through DSA sparse attention and joint reinforcement learning. Whether for complex software engineering closed loops or deep enterprise knowledge processing, this model provides robust foundational capabilities. For developers, understanding its thinking budget mechanism and multi-turn context management standards will be key to efficiently building next-generation long-term intelligent agent applications.