画板 1

    Large-Scale MoE Model Open Source: Long-Horizon Agent Architecture and Deployment

    13 hr ago0 views

    This article explores the technical architecture, core capabilities, and deployment practices of a recently open-sourced 2.8T Mixture-of-Experts (MoE) model designed for long-horizon Agent tasks. It covers sparse attention mechanisms, joint reinforcement learning, complex task performance in software engineering and 3D design, and provides a practical guide for industrial-grade MXFP4 quantized deployment using containerized inference engines and OpenAI-compatible APIs.

    As artificial intelligence advances toward executing complex tasks, long-horizon Agent capabilities have become a core focus in the evolution of large language models. Recently, the industry open-sourced a Mixture-of-Experts (MoE) model with a total parameter count of 2.8T and 104B activated parameters. Supporting a 1 million Token context window and native image-text input, the model leverages large-scale joint reinforcement learning post-training and underlying attention mechanism innovations to significantly enhance performance in long-horizon code tasks, software engineering, and 3D design. This article delves into the model’s architectural evolution, core capabilities, and engineering deployment practices.

    1. Underlying Architecture and Attention Mechanism Evolution

    When processing contexts with millions of tokens, traditional attention mechanisms face the challenge of quadratically increasing computational complexity. To balance efficiency and accuracy in long-horizon Agent tasks, this open-source model has been deeply optimized at the underlying architecture level.

    1. Deep Reconstruction of Sparse Attention Mechanism
    The model employs a hybrid strategy in its attention layer layout. Compared to the previous generation, it replaces 24 layers of Gated Multi-Head Implicit Attention with Gated Dynamic Sparse Attention (DSA), combined with 69 layers of Delta Attention. In DSA layers, instead of performing global computation on all historical tokens, the model dynamically selects 2,048 of the most relevant historical tokens to participate in the attention computation. This sparse activation strategy significantly reduces VRAM usage and inference latency for long contexts, enabling the Agent to maintain precise memory of core context during multi-step long-horizon tasks.

    2. Joint Reinforcement Learning and Weight Quantization
    During the post-training phase, the model adopts a joint reinforcement learning strategy covering hundreds of downstream domains. This cross-domain joint training prevents overfitting on single tasks and effectively improves generalization across different instruction sets. Additionally, the model provides an MXFP4 weight quantization version. While ensuring inference quality, it significantly lowers the deployment barrier for large-scale MoE models, making it possible to run ultra-large models on limited hardware resources.

    2. Long-Horizon Agent and Complex Task Scenarios

    The core of a long-horizon Agent lies in multi-step logical reasoning and the deep invocation of toolchains. The model focuses on enhancing capabilities in the following two major areas during training and evaluation:

    1. Code Development and Terminal Operations
    For long-horizon code Agents, software engineering, and 3D design tasks, the model must not only understand static code but also possess the ability to dynamically operate environments.

    • Multi-step Code Repair: The model can autonomously plan code review paths based on error messages, invoke testing tools to validate repair logic, and complete iterative code commits.
    • Terminal Operations and 3D Creation: In official evaluations, the model performed excellently in Terminal-Bench 4.0 (score: 65.7) and the Software Engineering Marathon evaluation (score: 51.7). It achieved a score of 92.7 in the BenchCAD evaluation with tool invocations, proving its instruction-following and spatial modeling capabilities in complex engineering software and CAD generation tasks.

    2. Knowledge Work and Toolchain Collaboration
    In non-code knowledge work, the model supports high-frequency scenarios such as web search, financial research, and office document processing.

    • Three-Tier Thinking Budget Mechanism: To balance inference speed and task depth, the model provides three thinking budget levels during inference: low, high, and max. Developers can dynamically configure this based on task complexity. For example, simple document Q&A can use the low budget for a quick response, while in-depth financial research or code architecture design should enable the max budget, allowing the model to perform sufficient internal reasoning.

    3. Industrial-Grade Quantized Deployment and API Integration Guide

    Deploying a 2.8T parameter MoE model into a production environment requires an efficient inference engine and standardized API design. The open-source model provides MXFP4 quantized weights and is compatible with mainstream large model inference acceleration frameworks.

    1. Inference Framework Adaptation and Containerized Deployment
    The model supports mainstream high-performance inference engines (such as vLLM, SGLang, etc.). Under a cloud-native architecture, deployment combined with a container platform and an elastic GPU resource pool is recommended. Below is an example configuration logic for deploying the MXFP4 quantized model based on a general framework:

    1. # Example configuration for containerized deployment
    2. apiVersion: apps/v1
    3. kind: Deployment
    4. metadata:
    5.   name: moe-llm-serving
    6. spec:
    7.   template:
    8.     spec:
    9.       containers:
    10.       - name: inference-engine
    11.         image: general/vllm-mxfp4:latest
    12.         command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
    13.         args:
    14.           - "--model"
    15.           - "path/to/open-weights-moe-2.8t"
    16.           - "--quantization"
    17.           - "mxfp4"
    18.           - "--tensor-parallel-size"
    19.           - "8"  # Adjust tensor parallelism based on GPU resources
    20.           - "--max-model-len"
    21.           - "1048576" # Enable 1 million Token context
    22.         resources:
    23.           limits:
    24.             nvidia.com/gpu: 8

    2. OpenAI/Anthropic Compatible API and Multi-Turn Tool Calling
    The model provides OpenAI/Anthropic compatible API interfaces, allowing existing business systems to integrate seamlessly. When developing complex Agents, context management for multi-turn tool calling is critical. The system requires preserving the complete “historical thinking” and “tool call messages” during multi-turn interactions to ensure the Agent does not lose state in long-horizon tasks.

    Below is the core code logic for multi-turn tool calling using the Python SDK:

    1. import openai
    3. client = openai.Client(base_url="http://localhost:8000/v1", api_key="demo_key")
    5. # Define search tool
    6. tools = [{
    7.     "type": "function",
    8.     "function": {
    9.         "name": "search_web",
    10.         "description": "Search the internet for the latest information",
    11.         "parameters": {
    12.             "type": "object",
    13.             "properties": {"query": {"type": "string"}},
    14.             "required": ["query"]
    15.         }
    16.     }
    17. }]
    19. # Initialize conversation context (must retain complete history)
    20. messages = [{"role": "user", "content": "Help me analyze recent industry development trends and generate a report outline"}]
    22. # First round of reasoning
    23. response = client.chat.completions.create(
    24.     model="moe-2.8t-agent",
    25.     messages=messages,
    26.     tools=tools,
    27.     extra_body={"thinking_budget": "max"} # Set max thinking budget
    28. )
    30. # Process tool call result and append to context
    31. tool_call = response.choices[0].message.tool_calls[0]
    32. messages.append(response.choices[0].message)
    34. # Simulate tool execution result
    35. messages.append({
    36.     "role": "tool",
    37.     "tool_call_id": tool_call.id,
    38.     "content": "The industry market size is growing steadily, with frequent favorable policies..."
    39. })
    41. # Second round of reasoning (continue generation based on complete context)
    42. final_response = client.chat.completions.create(
    43.     model="moe-2.8t-agent",
    44.     messages=messages,
    45.     tools=tools,
    46.     extra_body={"thinking_budget": "max"}
    47. )
    48. print(final_response.choices[0].message.content)

    4. Conclusion and Cloud Architecture Recommendations

    The open-sourcing of this model provides a highly valuable paradigm for the implementation of long-horizon Agents. From the introduction of Gated Dynamic Sparse Attention to the application of cross-domain joint reinforcement learning, and the flexible scheduling of the three-tier thinking budget, all reflect the trend of large models evolving into “high-complexity task executors.”

    For enterprise-level deployment, the stable service of a 2.8T parameter model requires solid underlying infrastructure support. When deploying in the cloud, it is recommended to fully utilize object storage for efficient distribution of model weights; use message queues and logging services to monitor the stability of multi-turn tool invocation chains; and combine container platforms with monitoring and alerting systems to achieve elastic scaling of inference nodes. Through the deep integration of cloud-native architecture and large model capabilities, developers can focus more on building complex business logic and accelerating the iteration cycle of intelligent applications. Note that this open-source model uses a specific license; please carefully review the relevant authorization terms before commercial use to ensure compliance.

    09040904

    Top articles