2.8T MoE Model Open-Sourced: Sparse Attention and Long-Horizon Agent Capabilities Explained

The open-source community has recently welcomed an ultra-large-scale mixture-of-experts (MoE) model with 2.8 trillion total parameters and 104 billion active parameters. Beyond its sheer size, the model introduces substantial innovations in both architecture and post-training, and natively supports 1M-token contexts and multimodal input. These advances provide a strong foundation for demanding workloads such as long-horizon agent development, complex software engineering, and 3D design. This article takes a deep look at the model’s technical architecture, agent capabilities, and real-world deployment.

1. Core Architectural Innovation: Deep Integration of Sparse Attention and MoE

When processing contexts of a million tokens or more, a conventional Transformer’s computational cost and memory footprint grow explosively. During post-training, the model’s underlying attention mechanism was significantly reworked, introducing a dynamic sparse attention mechanism to replace part of the traditional multi-head latent attention (MLA).

Specifically, the model replaces its 24 Gated MLA layers with Gated DSA, deeply combined with 69 layers of Delta Attention. The core of DSA is “dynamic selection”: at each attention computation, instead of traversing the entire token history, the model precisely selects the 2,048 most relevant historical tokens to participate in the computation. This sparsity drastically reduces the compute cost of long-sequence inference, making effective understanding of 1M tokens feasible within limited GPU memory.

In addition, the model is released with MXFP4 weight quantization. MXFP4 is a microscaling floating-point format that keeps model accuracy essentially intact while significantly reducing weight storage size and memory bandwidth pressure, laying the groundwork for efficient inference deployment on-premises or in the cloud.

2. Long-Horizon Agent Capabilities: From Knowledge Retrieval to Engineering Practice

The essence of a long-horizon agent is the model’s ability to stay consistent with its goal across multi-turn interactions and multi-step tasks while calling external tools accurately. Through joint reinforcement learning post-training that covers hundreds of downstream domains, the open-source model markedly improves its tool calling and task decomposition abilities.

1. Dynamic reasoning budget allocation

To balance inference efficiency against task accuracy, the model offers three thinking budget levels — low, high, and max — that developers can flexibly configure per business scenario:

• low: suited to simple intent recognition, everyday Q&A, and other high-frequency, low-latency scenarios; reduces the generation of internal chain-of-thought.
• high: suited to routine web search, financial data analysis, and office document processing; provides moderate logical reasoning.
• max: designed for long-horizon coding agents, terminal operations, and complex software engineering; allows deep self-reflection and multi-path exploration.

2. Tackling complex engineering tasks

According to the officially published evaluation results, the model performs strongly on Terminal-Bench 4.0 (65.7), SWE Marathon v1.1 (51.7), and DeepSWE v1.1 (75.5). This indicates that it can not only understand complex codebase structures but also execute multi-step tasks such as code fixes and automated testing in combination with terminal tools. In 3D creation, a score of 92.7 on BenchCAD (with tools) demonstrates its significant potential for spatial geometry understanding and generation with professional CAD tools.

3. Deployment and API Integration Guide

For developers and enterprises, putting the model into production is the key to closing the technical loop. The open-source release ships with comprehensive ecosystem support: it is compatible with mainstream inference frameworks such as vLLM, SGLang, and TokenSpeed, and exposes OpenAI- and Anthropic-compatible API interfaces, greatly reducing business migration costs.

When integrating a long-horizon agent, pay special attention to context management across multi-turn tool calls. Because agent tasks typically follow a “think – call a tool – observe the result – think again” loop, the model requires each request to carry the complete history of thinking and tool-call messages. Below is a standard pseudocode-style example of multi-turn tool calling:

import requests
import json
# Assume an OpenAI-compatible API service
api_url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
# Build the multi-turn message body; the full history of thinking and tool calls must be included
messages = [
    {"role": "user", "content": "Look up today's weather in Beijing and decide whether it is suitable for outdoor running."},
    # The model's first-round thinking and tool call request
    {"role": "assistant", "content": "Thinking: I need to call the weather query tool to get today's weather in Beijing.", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Beijing\"}"}}]},
    # The tool's returned result
    {"role": "tool", "tool_call_id": "call_1", "content": "{\"temp\": 22, \"weather\": \"sunny\", \"aqi\": 45}"},
    # The model's final output based on the tool result
    # The model continues generating here...
]
payload = {
    "model": "open-source-moe-2.8t",
    "messages": messages,
    "thinking_budget": "max",  # Set the highest thinking budget to handle complex logic
    "stream": False
}
response = requests.post(api_url, headers=headers, json=payload)
print(response.json())

4. Cloud-Native Architecture Considerations for Enterprise Deployment

Bringing a 2.8T-parameter MoE model into enterprise production makes compute scheduling and high-availability architecture the central challenges. By leveraging mature cloud-native infrastructure (Baidu AI Cloud being one example), developers can build an efficient model-serving architecture:

1. Containerized deployment and GPU scheduling: Use a container platform (for example, Baidu AI Cloud CCE) to manage vLLM or SGLang inference nodes, applying GPU memory affinity scheduling policies so that the model’s different expert networks load efficiently. Combined with MXFP4 quantization, this can substantially reduce the number of GPUs required per node and lower inference costs.
2. Storage and bandwidth optimization: Even after quantization, the weight files of a 2.8T-parameter model remain enormous. Centralized management and version control of weights through object storage (such as BOS), combined with high-speed intranet distribution, accelerates the cold start of inference nodes.
3. Monitoring and logging: For multi-step agent workflows, deploy comprehensive logging services. Recording the inputs and latency of every tool call, together with a monitoring and alerting platform, allows developers to precisely locate latency bottlenecks or infinite logic loops when a long-horizon agent executes complex tasks.

5. Summary and Compliance Notes

The open-sourcing of this 2.8T MoE model marks a solid step forward for the open-source community in tackling two industry-hard problems: long-horizon agents and million-token contexts. Its innovative Gated DSA attention mechanism, flexible three-tier thinking budgets, and reinforcement-learning-based post-training strategy provide a new technical foundation for building sophisticated agents.

Developers should note in particular that the released weights use a custom license, not the industry-standard Apache 2.0 or MIT. Before any enterprise-grade commercial use, carefully review and verify the compliance requirements in the license terms to ensure your business model conforms to the license. With compliance addressed and an efficient cloud-native deployment in place, the model holds strong potential to unlock productivity in high-value domains such as software engineering, financial research, and 3D design.