Deep Dive into the Open-Weight 2.8T MoE Architecture: Long-Horizon Agents and Sparse Attention

Deep Dive into the Open-Weight 2.8T MoE Architecture: Long-Horizon Agents and Sparse Attention

As artificial intelligence evolves toward executing complex tasks, long-horizon agents have become a critical breakthrough in the industry. Recently, an ultra-large-scale Mixture-of-Experts (MoE) model with a total parameter count of 2.8T has officially had its weights open-sourced. Beyond achieving industry-leading scale, the model provides robust underlying technical support for long-horizon code generation, software engineering, and automated knowledge work through native multimodal input, a 1-million-token context window, and deeply optimized sparse attention mechanisms. This article will analyze the model’s technical characteristics and practical guidelines from the dimensions of architecture design, core capabilities, post-training strategies, and engineering deployment.

1. Core Architecture Design: Efficient Collaboration Between Sparse MoE and Long Context

The model adopts a typical sparse Mixture-of-Experts architecture. While its total parameter count reaches 2.8T, only 104B parameters are activated during a single inference pass. This sparse activation mechanism ensures a vast knowledge capacity while significantly reducing the computational overhead of each forward pass, striking a balance between compute efficiency and model capability.

When processing ultra-long contexts of up to a million tokens, traditional attention mechanisms incur quadratically growing computational and memory consumption. To break through this bottleneck, the model implements deep architectural modifications to the attention mechanism:

1. Dynamic Sparse Attention (DSA): The model replaces 24 layers of gated Multi-Layer Perceptron (MLP) attention from its original architecture with gated DSA. The DSA mechanism breaks the inherent paradigm of global attention by dynamically selecting only the 2,040 most relevant historical tokens to participate in the attention computation for each pass. This relevance-based routing sparse strategy drastically reduces redundant computations during long-sequence processing.
2. Delta Attention Fusion: In addition to the 24 DSA layers, the model deeply integrates 69 layers of Delta attention. The combination of the two allows the model to accurately capture distant key information via DSA while maintaining local context coherence via Delta attention when processing ultra-long texts, thereby sustaining stable comprehension and reasoning capabilities in long-horizon tasks.

2. Long-Horizon Agent Capabilities: Full-Scenario Coverage from Code Engineering to 3D Creation

The model’s design goals are explicitly aimed at Agent tasks requiring multi-step, long-term planning. In practical applications, it can integrate with external tools to complete complex instruction chains.

1. Complex Code Development and Software Engineering
In the code domain, the model demonstrates strong execution capabilities for long-horizon code agents and software engineering tasks. It supports not only routine code generation but also excels in high-difficulty tasks like terminal operations and bug fixing. The model can comprehend the context of an entire software engineering project and autonomously conduct multi-round debugging and refactoring based on feedback from the code execution environment. Official benchmark data shows it scores 65.7 in terminal operation tests, 51.7 in the software engineering marathon, and reaches a high of 75.5 in deep software engineering tests.

2. 3D Creation and CAD Generation
In 3D design, the model supports deep integration with professional design tools. It can comprehend complex spatial geometry descriptions and translate them into CAD generation instructions. In tool-integrated benchmarks, the model achieved an excellent score of 92.7, indicating that it not only possesses spatial imagination for 3D structures but can also accurately map them to the APIs or instruction sets of specific engineering software.

3. Knowledge Work and Multimodal Processing
Beyond code and design, the model natively supports image-text input, enabling it to handle knowledge-intensive tasks such as web retrieval, financial research, and office document parsing. Scoring 64.4 in office document QA tests validates its ability to extract key information from complex unstructured documents and perform logical deductions.

3. Three-Tier Thinking Budgets: Flexible Reasoning Resource Scheduling

To accommodate Agent tasks of varying complexity, the model offers a three-tier thinking budget mechanism during inference: “low,” “high,” and “max”. This design mirrors the different depths of thought humans apply to problems:

• Low budget: Suited for immediate response tasks like simple Q&A or entity recognition, delivering quick conclusions with minimal reasoning steps to reduce latency.
• High budget: Applicable to moderately complex tasks like routine code writing or document summarization, balancing speed and depth.
• Max budget: Designed for long-horizon tasks involving multi-step tool calls, complex software architecture design, or deep financial report generation. In this mode, the model expands a detailed internal chain-of-thought to thoroughly plan task steps. The high benchmark scores mentioned above were all achieved under the “max” thinking setting.

Developers can dynamically configure the thinking budget based on the actual time-sensitivity and logical depth requirements of their business scenarios, achieving optimal control over inference compute costs.

4. Joint RL Post-Training and MXFP4 Quantized Deployment

1. Joint Reinforcement Learning Across Hundreds of Domains
The model’s powerful tool-calling and task-planning capabilities are not solely derived from pre-training but underwent large-scale joint reinforcement learning (RL) post-training. The training process covered hundreds of downstream vertical domains. By establishing multi-dimensional reward functions, the model learned through massive attempts how to more accurately invoke external tools, correct execution paths based on error messages, and maintain goal alignment without drift during long-horizon tasks.

2. MXFP4 Weight Quantization and Open-Source Ecosystem Compatibility
To lower the deployment barrier for ultra-large models, the open weights utilize MXFP4 (Microscaling FP4) quantization technology. MXFP4 compresses weights into a 4-bit floating-point format, drastically reducing memory footprint while maintaining numerical precision.
At the deployment level, the quantized weights are fully compatible with mainstream high-performance inference frameworks such as vLLM, SGLang, and TokenSpeed. Simultaneously, the model natively provides API interfaces compatible with mainstream LLM serving standards, allowing developers to seamlessly integrate it into existing business workflows.

It is worth noting that these weights are released under a proprietary license rather than fully open-source protocols like Apache 2.0 or MIT. Enterprises must carefully review the authorization terms before integrating them into commercial products.

5. Agent Engineering Practices: Message Management for Multi-Turn Tool Calls

In practical long-horizon Agent development, multi-turn tool calling is an unavoidable phase. The model establishes clear engineering requirements: multi-turn tool calls must retain the complete history of thinking and tool call messages.

This differs from standard conversational truncation management. In long-horizon tasks, the model relies on previous “thinking processes” and “tool return results” to decide its next action. If developers truncate the historical thinking chain or tool call records to save context window space, the model may lose task context, leading to infinite loops or hallucinations.

Below is an engineering example framework based on a standard API for multi-turn tool calling, demonstrating how to correctly maintain historical messages:

import requests
import json
# Assuming the API endpoint follows the OpenAI-compatible standard
api_url = "https://api.your-inference-platform.com/v1/chat/completions"
headers = {"Authorization": "Bearer YOUR_API_KEY", "Content-Type": "application/json"}
# Initialize the conversation context with system instructions and initial user requirements
conversation_history = [
    {"role": "system", "content": "You are a senior code agent. Analyze step-by-step and call tools to fix the codebase."},
    {"role": "user", "content": "Please check the login module of the current repository and fix potential SQL injection issues."}
]
def execute_agent_task(history):
    # First round of model inference, setting max thinking budget for complex tasks
    payload = {
        "model": "2.8t-moe-agent",
        "messages": history,
        "thinking_budget": "max",
        "tools": [{"type": "function", "function": {"name": "read_file", "parameters": {}}}]
    }
    response = requests.post(api_url, headers=headers, json=payload).json()
    assistant_message = response["choices"][0]["message"]
    # Key step: Must append the model's thinking process and tool calls completely to the history
    history.append(assistant_message)
    # If the model decides to call a tool
    if assistant_message.get("tool_calls"):
        for tool_call in assistant_message["tool_calls"]:
            # Execute actual tool logic (simulated here)
            tool_result = {"role": "tool", "tool_call_id": tool_call["id"], "content": "Execution result..."}
            # Append the tool return result completely as well
            history.append(tool_result)
        # Recursively execute the next round until the model no longer requests tool calls
        return execute_agent_task(history)
    else:
        # Task completed, return the final result
        return assistant_message["content"]
final_answer = execute_agent_task(conversation_history)
print(final_answer)

In the above logic, the history list completely records system instructions, user inputs, the model’s thinking and tool call requests, and the tools’ return results. This complete context retention mechanism is the core engineering practice ensuring that long-horizon Agents do not deviate from their objectives.

6. Conclusion and Outlook

The open-weight 2.8T MoE model effectively mitigates the computational pressure of long contexts via Dynamic Sparse Attention and empowers the model with long-horizon planning capabilities in software engineering, 3D design, and complex knowledge work scenarios through large-scale reinforcement learning post-training. Combined with MXFP4 quantization and support for mainstream inference frameworks, it provides a solid technical foundation for the deployment of enterprise-grade AI Agents. When building complex applications with this model, developers should focus on the rational allocation of thinking budgets and the complete maintenance of multi-turn tool calling history to fully leverage the model’s advantages in long-horizon tasks.