Open-Source 2.8T-Parameter MoE Model: Long-Horizon Agent Architecture and Inference Optimization

As large language model applications evolve from simple conversational interaction toward complex engineering automation, long-horizon agents place unprecedented demands on a model’s context understanding, multi-step logical reasoning, and tool-calling robustness. An open-source community has recently released the weights of a new-generation, ultra-large-scale mixture-of-experts (MoE) model. The model totals 2.8 trillion parameters with 104 billion activated per forward pass, and supports million-token context windows and native image-text input. Beyond marking a new balance between parameter scale and sparse activation efficiency for open-source models, the release offers highly instructive engineering practices in long-horizon task orchestration and low-level attention optimization.

1. Core Architecture Evolution: Sparse Attention and RL-Based Post-Training

In ultra-long-context scenarios, the quadratic computational complexity of conventional attention becomes the central bottleneck for long-horizon agents. The newly released model introduces deep architectural innovations centered on a comprehensive upgrade of its sparse attention mechanism.

1.1 Hybrid Attention Orchestration

Instead of computing full attention everywhere, the model adopts dynamic sparse attention in its deeper layers. Specifically, the new version replaces the original architecture’s 24 layers of gated multi-head latent attention with gated dynamic sparse attention (DSA), deeply combined with 69 layers of Delta attention. At the DSA layers, the model dynamically selects only the 2,048 most relevant historical tokens for each attention computation. This mechanism effectively gives the model a “memory retrieval” capability: when processing contexts of up to a million tokens, it can precisely locate key historical information, dramatically cutting inference-time memory footprint and compute overhead while preserving long-text awareness.

1.2 Cross-Domain Joint Reinforcement Learning

Beyond the base architecture, the model’s post-training incorporates joint reinforcement learning spanning hundreds of downstream domains. This cross-domain joint training strategy gives the model not only deep competence in individual tasks but also strong generalization in cross-domain tool calling and chaining complex logical steps. Training adopts MXFP4 weight quantization, which significantly reduces storage and memory pressure without loss of model precision, laying the groundwork for efficient inference deployment.

2. A Multi-Dimensional Capability Matrix for Long-Horizon Agents

The essence of a long-horizon agent is the closed loop of “understand the goal, decompose the task, call tools, verify feedback.” The model demonstrates the hard capability to support complex agent workflows across coding, 3D creation, and general knowledge work.

2.1 Software Engineering and 3D Work

For software engineering and 3D design tasks, the model can act as a terminal-operating agent, autonomously performing code fixes, dependency management, and CAD model generation. In practice, a developer describes requirements in natural language, and the model generates an execution plan and completes multi-step tasks using tools such as the terminal command line and a code interpreter. Its strong contextual memory allows it to maintain logical consistency across long-cycle refactoring projects, avoiding modification conflicts caused by “context loss.”

2.2 Knowledge Work and Multi-Step Reasoning

In enterprise office scenarios, the model supports complex web search aggregation, in-depth financial research, and multi-format document processing. To accommodate different latency and depth requirements across business scenarios, the inference stage offers three thinking-budget tiers: low, high, and max.

• Low: suited to simple intent recognition and quick Q&A, delivering low-latency responses.
• High: intended for routine document summarization and single tool calls, balancing compute cost against output quality.
• Max: aimed at complex research tasks requiring deep reasoning, such as writing financial research reports or chained multi-tool calls; the model engages in extended internal reasoning to ensure rigorous conclusions.

3. Benchmark Performance

On widely used industry benchmarks, the model posts strong long-horizon task results (reference scores by dimension):

• Terminal-Bench 4.0: 65.7 — reflects command execution and problem-solving ability in real terminal environments.
• SWE Marathon v1.1: 51.7 and DeepSWE v1.1: 75.5 — measure performance on complex software engineering fixes and deep code maintenance.
• BrowseComp: 92.1 — evaluates complex web browsing, information retrieval, and synthesis.
• BenchCAD (with tools): 92.7 — shows very high accuracy in 3D tool-assisted design generation.
• OfficeQA Pro v2: 64.4 — validates logical analysis in complex office document Q&A.

All evaluations default to the highest thinking tier, and some were completed in combination with commonly used agent frameworks — evidence that the model can serve not only as a standalone reasoning engine but also integrate seamlessly into existing agent ecosystems.

4. Inference Deployment and Integration Practices

For enterprise developers, putting a MoE model of this size into production is a systems engineering effort. The model ships MXFP4 quantized weights and natively supports mainstream high-performance inference frameworks.

4.1 Inference Framework Compatibility and Deployment

The model currently integrates smoothly with leading inference acceleration engines such as vLLM and SGLang. For actual deployment, a distributed cluster supporting tensor parallelism and pipeline parallelism is recommended. Because only 104 billion parameters are activated per forward pass, optimizing the routing and scheduling algorithms can effectively improve GPU utilization and lower the total compute cost per inference.

4.2 API Integration and Multi-Turn Tool-Calling Rules

The model exposes an API compatible with mainstream commercial LLM API standards, making it easy for existing business systems to migrate. However, when implementing multi-turn tool calling in production, developers must pay special attention to one rule: in multi-turn conversations, the full history of reasoning content and tool-call messages must be preserved.

The following is a simplified pseudocode example of context management for multi-turn agent tool calling:

# Pseudocode: building context for multi-turn agent tool calls
def build_agent_context(user_query, model_reasoning, tool_result):
    """
    Assemble the multi-turn conversation history, keeping the reasoning chain intact
    """
    messages = [
        {"role": "user", "content": user_query},
        # The model's reasoning and tool-call requests must be retained
        {
            "role": "assistant", 
            "content": model_reasoning, # The model's internal reasoning or explanation
            "tool_calls": [{"id": "call_01", "name": "search_doc", "args": {"query": "Q3 earnings report"}}]
        },
        # Tool execution results must be passed back as tool messages
        {
            "role": "tool", 
            "tool_call_id": "call_01", 
            "content": json.dumps(tool_result)
        }
    ]
    return messages
# On the next inference call, pass the complete messages list to the inference engine
response = inference_engine.chat(
    model="open_moe_model",
    messages=build_agent_context(...),
    thinking_budget="max" # Set the thinking budget
)

This strict message-retention discipline ensures that, across multiple interaction rounds, the model accurately understands the causal context of prior tool calls, preventing duplicate tool invocations or logical dead loops in long-horizon tasks.

4.3 Cloud-Native Infrastructure Support

At the infrastructure level, deploying a model of this scale typically relies on cloud-native container platforms for elastic scaling. Enterprises can store the massive model weights in general-purpose object storage, accelerate model loading with high-performance file systems, and use logging services together with monitoring and alerting components to track inference latency and GPU utilization in real time. Where in-house compute is insufficient, enterprises can also evaluate public cloud platforms with strong compute foundations — such as Baidu AI Cloud — leveraging their elastic GPU instances and efficient network topologies to further lower the deployment and operations barrier for ultra-large-scale MoE models.

5. Conclusion

From a technology-development perspective, the open-sourcing of this 2.8T-parameter MoE model is not merely a celebration of parameter stacking; it is an engineering exemplar of deeply fusing sparse attention with reinforcement learning. By cleverly combining DSA and Delta attention, it breaks the memory constraints of ultra-long-context computation; through tiered thinking budgets, it achieves a dynamic balance between cost and depth. For developers and enterprises, building long-horizon agents on open models of this class not only removes hard dependence on closed commercial APIs but also opens the way to new agent application paradigms in demanding verticals such as code engineering and research analysis.