Trillion-Parameter MoE Models: QAT and Agent Inference Deployment Practices

Trillion-Parameter MoE Models: QAT and Agent Inference Deployment Practices

Recently, a leading research team fulfilled its open-weight commitment by releasing its latest 2.8-trillion (2.8T) parameter Mixture of Experts (MoE) model to the community. This model ranked first among open-weight models and thirteenth overall in a long-horizon complex agent arena. This release not only represents a leap in model capabilities but also provides the developer community with an excellent engineering sample for researching ultra-large sparse models, quantization-aware training, and complex Agent workflows. This article strips away specific brand attributes to deeply analyze the model’s architecture design, quantization strategy, deployment thresholds, and engineering practices in Agent scenarios from a purely technical perspective.

1. Model Architecture: Deep Fusion of Sparse Attention and Mixture of Experts

As an MoE model designed for long-context and complex reasoning, it has a total parameter count of approximately 2.78 trillion, but only activates about 104B (104 billion) parameters per token during inference. This extreme sparsity is key to maintaining strong expressive power while controlling inference compute.

1. Hierarchical Structure and Attention Mechanism Evolution
The model consists of 93 network layers, interleaving 69 KDA (a specific layer normalization and feed-forward network structure) layers and 24 gated sparse attention layers. Unlike the previous generation, which used full attention, this generation made major architectural adjustments for long-context modeling: replacing the full-attention gated MLP layers with gated layers based on deep sparse attention. This improvement aims to significantly reduce the computational complexity of long-sequence modeling through sparse sampling patterns, enabling the model to natively support an ultra-long context window of up to 1 million tokens while supporting multimodal image-text input.

2. Expert Routing Strategy
For the expert layer design, the model is equipped with 896 independent experts. It uses a Top-K routing mechanism where each token dynamically selects 16 experts for computation, plus 2 additional shared experts. The shared experts capture general features common to all tokens, while the routed experts perform specific calculations for particular patterns. This design effectively improves the model’s generalization and parameter utilization.

3. Multimodal Vision Module
The visual perception part is handled by a lightweight vision encoder with about 401M parameters. This module converts input images into token sequences isomorphic to text, injecting them directly into the main model for joint reasoning, thereby achieving interleaved image-text input comprehension.

2. Low-Bit Quantization Strategy and VRAM Estimation

Open-sourcing ultra-large models inevitably faces storage and transmission challenges. This release employs an aggressive low-bit quantization strategy, significantly compressing the physical size of the weights.

1. Application of Quantization-Aware Training (QAT)
Unlike Post-Training Quantization (PTQ), which only truncates precision, this model uses Quantization-Aware Training (QAT). This means low-precision simulation is introduced during the training phase, and the network learns to resist quantization errors during backpropagation. Weights are stored in MXFP4 format, while activations during forward propagation use MXFP8 format. The MXFP (Microscaling Floating Point) format shares a single scaling factor for a block of data, maintaining a good numerical dynamic range even at extremely low bit widths, making it a cutting-edge technology for ultra-large model quantization.

2. Physical Storage and Parameter Statistics
Thanks to MXFP4 compression, the repository footprint on model hosting platforms is about 1.56TB. Statistics based on the safetensors format show that the actual loadable parameter count is approximately 2.78 trillion. Teams wishing to fine-tune or audit the model need to plan for at least 1.6TB of storage space for weight download and decompression.

3. Local Deployment Thresholds and Inference Framework Adaptation

Although the model weights are open-sourced, running the complete inference process locally still faces a extremely high hardware barrier.

1. Hardware Resource Estimation
Due to the 104B activated parameters and the VRAM consumption of the 1-million-token KV Cache, a traditional single-node 8-GPU setup (such as mainstream 80GB servers) cannot meet deployment requirements. According to industry estimates, a viable hardware configuration requires 8 high-end AI accelerators (such as the latest B300-level GPUs), providing a total VRAM capacity of about 2.3TB per node. Hardware procurement costs alone range from $550,000 to $650,000. This means local deployment is currently primarily suited for large enterprise labs or high-performance computing centers.

2. Inference Engine Adaptation
To lower the deployment barrier for developers, mainstream open-source inference frameworks have quickly followed up with deployment recipes:

• vLLM / SGLang: Both high-performance inference engines support mixed-precision inference with MXFP4/MXFP8, allowing developers to load the model directly via specific launch arguments.
• TokenSpeed and other acceleration frameworks: Deep optimizations have been made for multi-GPU tensor parallelism and KV Cache management, adapting to the model’s sparse routing structure.

Developers can quickly spin up services via standard interfaces of mainstream inference frameworks. For cloud deployments, it is recommended to use container platforms providing high-performance GPU instances and large-capacity shared file storage for elastic scaling.

4. API Integration and Agent Engineering Practices

If constrained by hardware costs, developers can also access the model via the official API. The interface is fully compatible with mainstream OpenAI and Anthropic standards, significantly reducing code migration costs.

1. Standard and Fast Modes
The API provides two model versions: Standard and Fast. The Fast version likely employs more aggressive speculative decoding or operator fusion strategies at the underlying level, with official claims that its inference speed is 2 to 3 times faster than the Standard version.

2. Reasoning Effort Control
The model natively supports a reasoning_effort parameter, with options low, high, and max, defaulting to max. For simple text summarization tasks, it can be set to low to reduce latency and token consumption; for complex code generation or mathematical proofs, keeping it at max fully utilizes its slow-thinking capabilities.

3. Multi-Turn Tool Calling State Management
Multi-turn tool calling is a core capability when building complex Agents. This model has a key engineering requirement when handling multi-turn tool calls: when passing the previous round’s tool execution results back to the model, you must also pass back the previous round’s reasoning_content (reasoning process content) and tool_calls (tool call records).

The following compliant pseudocode demonstrates how to construct the message body for multi-turn tool calls:

messages = [
    {"role": "user", "content": "Check the current weather in Beijing and tell me if I need an umbrella"},
    # Complete structure returned by the model in the first round
    {
        "role": "assistant",
        "reasoning_content": "The user needs to check the weather to decide whether to bring an umbrella. I need to call the weather API to get real-time data for Beijing.",
        "tool_calls": [
            {
                "id": "call_001",
                "type": "function",
                "function": {"name": "get_weather", "arguments": '{"city": "Beijing"}'}
            }
        ]
    },
    # Tool execution result
    {
        "role": "tool",
        "tool_call_id": "call_001",
        "content": '{"temperature": "15C", "condition": "rain"}'
    }
]
# Pass to the model for the next round of inference, retaining the complete context chain
response = client.chat.completions.create(
    model="model-name-pro",
    messages=messages,
    reasoning_effort="high"
)

Preserving the reasoning chain not only maintains the coherence of the model’s thought process but also prevents logical disconnects within the context window, which is crucial in long-horizon Agent tasks.

5. Agent Evaluation Paradigm: From Static Benchmarks to Real-World Sessions

The leap in model capabilities is fully reflected in the Agent Arena rankings. Traditional static benchmarks can no longer accurately measure model performance in real-world complex scenarios. The Arena constructed a more application-relevant evaluation system by collecting 17,500 real long-horizon sessions.

1. Evaluation Signal Mechanism
Instead of relying on simple multiple-choice scores, the Arena extracts multi-dimensional signals from real interactions:

• Task Success Confirmation: Whether the user explicitly confirms task completion at the end of the session.
• Sentiment Feedback: Extracting words of praise and complaint from user conversations.
• Correction Response: The success rate of the model’s subsequent corrective actions when the user points out an error.
• Execution Reliability: Whether the generated shell commands or code snippets executed successfully in the user’s real environment.

2. Performance Breakthrough and Analysis
Based on the “net improvement” metric calculated from the above signals, the model achieved a +5.1% net improvement at maximum reasoning effort, ranking first among open models and thirteenth overall. It topped the open model charts in three sub-domains: Work, Code, and Chat.
More notably is its robustness improvement: in the highly challenging sub-category of Bash command reliability, the model achieved a +7.9% net improvement, ranking first. In contrast, the previous generation model scored -4.5% in this metric, ranking only thirteenth. This indicates a qualitative leap in the new generation model’s accuracy in generating executable code and understanding system environments.

6. Conclusion and Outlook

The open-sourcing of this ultra-large MoE model signals that the open-source community is rapidly approaching closed-source commercial models in scale. Its use of quantization-aware training, deep sparse attention, and a massive expert network demonstrates a viable engineering path for long-context and high-complexity reasoning.

For enterprises and developers, despite the high VRAM threshold for local deployment, the rapid adaptation by mainstream inference frameworks and standardized API services allow developers to integrate these powerful capabilities at a low cost. Particularly in complex Agent workflows, rational configuration of reasoning_effort and proper management of multi-turn tool calling context states can maximize the model’s value in real business scenarios. In the future, as low-bit quantization technology and distributed inference frameworks continue to evolve, the efficient deployment of trillion-parameter models will become much more accessible.