Open-Source 2.8T-Parameter MoE Model: Sparse Attention Architecture and Long-Horizon Agent Engineering

In recent years, large-scale Mixture-of-Experts (MoE) models have shown remarkable potential for handling complex tasks. Recently, the weights of a MoE large model with 2.8 trillion total parameters and 104 billion activated parameters were officially released to the public. Beyond supporting a 1-million-token context window and native text-and-image input, the model introduces an innovative sparse attention mechanism and large-scale joint reinforcement learning post-training, which together significantly improve the execution stability of long-horizon agents. This article analyzes the model from four perspectives: technical architecture, core capabilities, application scenarios, and engineering deployment.

1. Core Architecture and Sparse Attention Design

At such a massive parameter scale, balancing computational efficiency with long-context modeling capability is a central engineering challenge. The model incorporates targeted architectural optimizations, most notably a substantial overhaul of the attention mechanism.

1.1 Gated Dynamic Sparse Attention

The model replaces the conventional gated multi-head latent attention in the bottom 24 layers with gated dynamic sparse attention (DSA). Under DSA, each attention computation no longer traverses the entire context. Instead, it dynamically selects the 2,048 most relevant historical tokens to participate in the calculation. This substantially reduces the time complexity of long-sequence computation, allowing the model to maintain high inference efficiency even when processing million-token contexts.

1.2 Hybrid Attention Layer Architecture

In addition to the 24 DSA layers, the remaining 69 layers adopt a specific Delta Attention mechanism. The design philosophy behind this hybrid, layered architecture is that the lower layers use the sparse mechanism to quickly filter out irrelevant information, while the higher layers rely on the specialized attention mechanism to maintain global semantic coherence—preserving memory depth while keeping feature decay under control.

2. Large-Scale Joint Reinforcement Learning Post-Training

Much of the model’s capability stems from its post-training stage. Rather than single-task fine-tuning, the model employs a joint reinforcement learning strategy covering hundreds of downstream domains.

2.1 Cross-Domain Capability Alignment

By constructing reward models across hundreds of vertical domains—including code engineering, terminal operations, financial research, and office document processing—the training process aligns the model’s capabilities across multiple dimensions. This joint training effectively prevents overfitting to specific tasks and equips the model with strong generalization.

2.2 Compute Efficiency Optimization

At both the training and inference levels, the model fully adopts MXFP4 weight quantization. This technique significantly reduces VRAM footprint without a noticeable loss in model precision, giving developers a practical path to deploying ultra-large models under limited hardware resources.

3. Thinking Budget and Task Orchestration for Long-Horizon Agents

As the core foundation for long-horizon agents, the model offers great flexibility in controlling its reasoning logic, focusing on two key problems in complex tasks: “thinking depth” and “tool invocation.”

3.1 Three-Tier Thinking Budget

The model provides three thinking budget configurations: Low, High, and Max. Developers can set the level dynamically based on the latency requirements and complexity of a given business scenario.

# Pseudocode example: specifying the thinking budget via the API
response = client.chat.completions.create(
    model="open-source-moe-2.8t",
    messages=[{"role": "user", "content": "Analyze new energy vehicle sales trends over the past three years and generate a report"}],
    extra_body={
        "thinking_budget": "max"  # Use the max tier for complex reasoning tasks
    }
)

For simple document Q&A, the Low tier is sufficient to deliver fast responses. For code architecture design and multi-step business workflow orchestration, the Max tier is recommended to ensure the model has sufficient time for deliberation.

3.2 Multi-Turn Tool Calling and State Retention

During long-horizon agent execution, the model supports completing multi-step tasks in combination with external tools. To keep the context intact, developers must retain the complete history of the reasoning process and tool-call messages across multiple conversation turns. In practice, this means every result returned by a tool must be fed back into the model as context, so the model can adjust its next action based on previous outcomes.

4. Core Application Scenarios and Capabilities

Built on the architecture and training strategy described above, the model demonstrates clear advantages in the following complex workflows:

4.1 Code Development and 3D Creation

Beyond conventional code generation and bug fixing, the model extends its capabilities to terminal operations and 3D design tasks. In software engineering scenarios, it can autonomously analyze error logs, invoke terminal commands to run tests, and iteratively fix code based on test feedback. In 3D creation, it can parse text-and-image instructions and generate complex three-dimensional structural designs by calling CAD tool interfaces.

4.2 Knowledge Work and Automated Research

In knowledge-intensive scenarios such as financial research and document processing, the model can autonomously perform web searches, information extraction, and structured analysis. With its million-token context window, it can ingest dozens of financial reports or lengthy research papers in a single pass, then use tool calls to automatically draw data charts and write analytical reports.

5. Deployment Practices and Engineering Guide

For enterprise deployment, the open release of the model weights greatly simplifies private, on-premise adoption. Still, teams should keep several key points in mind during implementation:

5.1 Inference Framework Compatibility

The model weights natively support mainstream high-performance inference frameworks, such as frameworks with PagedAttention support and high-concurrency inference engines. For deployment, it is advisable to integrate the model with an enterprise container platform and an elastic GPU resource pool to enable dynamic compute scheduling. On cloud platforms such as Baidu AI Cloud, the underlying compute infrastructure can further improve cluster network communication efficiency and optimize cross-node inference latency for the MoE architecture.

5.2 API Compatibility and Message Construction

The model offers an API compatible with mainstream LLM API specifications. When building agent applications, developers should pay close attention to how the message list is constructed: the history of multi-turn tool calls must include the complete reasoning content and the tool_calls records. Otherwise, truncated context can break the agent’s execution logic.

5.3 License Compliance

It is important to note that the model weights are released under a dedicated license—not a permissive open-source license such as Apache 2.0 or MIT. Before integrating the model into commercial product lines, an enterprise must have its legal team rigorously review the license terms to ensure the intended use case falls within the authorized scope.

Conclusion

From the sparse architecture of the 2.8T MoE model to the flexible scheduling enabled by the three-tier thinking budget, this newly open-sourced model provides a highly valuable engineering reference for applying very large models to long-horizon agent scenarios. In practice, developers should focus on memory optimization strategies for the DSA attention mechanism and on state management for multi-turn tool calls, and—taking into account the characteristics of their own compute platforms—build intelligent agent applications that are highly reliable and low in latency.