Open Weights for a Trillion-Parameter MoE Model: QAT Practices and Agent Evaluation

1. Background: Ultra-Large MoE Models Going Open-Weight

Recently, an AI research team released the complete weights of its flagship MoE model for public download, fulfilling a previous promise to open-source the weights within two weeks of release. This Mixture-of-Experts model features approximately 2.8 trillion total parameters and activates about 104 billion parameters per token. It supports a million-token context window and multimodal inputs (text and image). For teams looking to deploy, fine-tune, or audit model capabilities independently, this marks a shift from merely calling APIs to actually controlling the model’s internal behavior.

This article systematically outlines the key technical aspects of deploying such ultra-large open-weight models across five dimensions: model architecture, quantization schemes, deployment thresholds, API specifications, and evaluation systems. The goal is to help developers make rational decisions during model selection and implementation.

2. Model Architecture: Sparse Attention and Expert Routing

2.1 Layer Structure Design

According to the public model card, the model consists of 93 layers: 69 layers use a linear attention variant (KDA), and 24 layers use Gated Sparse Attention. Compared to its predecessor, the most significant change is replacing the original full-attention gating layers with sparse attention layers. This aims to significantly reduce computational and memory overhead in long-context scenarios.

This hybrid design—predominantly linear attention with a minority of sparse attention layers—is a mainstream approach for current long-context models. Linear attention processes global information of long sequences with near-constant complexity, while sparse attention performs precise token selection at critical layers, balancing efficiency and recall quality.

2.2 Expert Layer Configuration

The expert layer contains 896 experts, of which 16 are activated per token, alongside 2 shared experts. This combination of fine-grained experts and shared experts is a common strategy in MoE architectures to balance parameter count against inference costs:

• Fine-grained experts: These have smaller individual capacities and large quantities, allowing for more flexible routing and improving model capacity utilization.
• Shared experts: All tokens pass through these experts, which encode general knowledge and reduce redundant learning across specialized experts.

The vision component is a multimodal vision encoder with approximately 401M parameters, responsible for feature extraction from image inputs.

3. Quantization-Aware Training: Storage and Activation for Trillion Parameters

3.1 Precision Choices for Storage and Activation

The model weights are stored in MXFP4 format, while activations use MXFP8. The MX series refers to microscaling data formats recently promoted in the industry. The core concept is to share a single scaling factor within a very small block, maintaining good numerical stability at low bit-widths.

The use of Quantization-Aware Training (QAT) rather than Post-Training Quantization (PTQ) means the model simulated low-precision computation paths during the training phase, incorporating quantization errors into the optimization objective. For a trillion-parameter model, the value of QAT lies in:

1. Significantly reduced memory footprint: Weights stored in 4-bit theoretically reduce weight storage by about 75% compared to FP16.
2. Improved inference throughput: Low-precision matrix operations can be significantly accelerated on supported hardware.
3. Controlled accuracy loss: Compared to direct PTQ, QAT typically retains performance much closer to full precision on downstream tasks.

3.2 Repository Size Reference

The public model repository shows an overall footprint of about 1.56TB, with the weight files accounting for approximately 2.78 trillion parameters. This size imposes practical requirements for downloading, verification, and local storage, making it essential to plan disk capacity and transfer bandwidth in advance.

4. Local Deployment: Hardware Thresholds and Inference Framework Adaptation

4.1 Hardware Threshold Estimation

The barrier to local deployment for this class of model is exceptionally high. Third-party estimates suggest a viable configuration requires 8 high-memory accelerator cards providing roughly 2.3TB of total VRAM, translating to hardware costs between $550,000 and $650,000. It should be noted that these are third-party estimates; the official team has not provided formal hardware recommendations.

When planning deployment, pay attention to the following metrics:

• Total VRAM: Must accommodate weights, KV Cache, and activation peaks.
• Inter-card interconnect bandwidth: MoE expert parallelism is highly sensitive to communication overhead.
• Context length: A million-token context will significantly amplify KV Cache consumption.

4.2 Inference Framework Adaptation

Mainstream inference frameworks have provided deployment solutions, typically involving three stages: weight loading, parallel strategy configuration, and service startup. An illustrative startup configuration structure looks like this:

model: <local_weight_path>
tensor_parallel_size: 8
expert_parallel_size: 8
max_model_len: 1000000
quantization: mxfp4
kv_cache_dtype: auto
enable_prefix_caching: true

In practice, note that the expert parallelism and tensor parallelism strategies for MoE models need to be adjusted according to the specific framework’s scheduling mechanism. Configurations from dense models cannot be directly applied.

5. API Specifications and Multi-Turn Tool Calling

5.1 Interface Compatibility

For teams that prefer not to self-deploy, the model is accessible via API, compatible with both OpenAI and Anthropic interface styles. Additionally, a high-speed version is available, which officially claims to be 2 to 3 times faster than the standard version.

5.2 Reasoning Effort and Tool Calling

The model supports a reasoning_effort parameter with options low, high, and max (default is max). This parameter controls the depth of the model’s reasoning phase, directly impacting cost and latency.

A critical constraint during multi-turn tool calling: the previous round’s reasoning_content and tool_calls must be passed back together. This differs from traditional conversational interfaces because the reasoning content participates in subsequent decision-making. Discarding it will lead to context inconsistency. An illustrative calling structure:

messages.append({
    "role": "assistant",
    "content": response.content,
    "reasoning_content": response.reasoning_content,
    "tool_calls": response.tool_calls
})
messages.append({
    "role": "tool",
    "tool_call_id": tool_call.id,
    "content": tool_result
})

6. Agent Evaluation: Extracting Signals from Real Sessions

6.1 Evaluation Data Sources

Data from a certain evaluation platform is derived from real user agent sessions. The system extracts several types of signals:

• Whether the user explicitly confirms task success.
• Praise or complaints directed at the model.
• Model responses following user corrections.
• Whether shell commands execute successfully.

Based on these signals, the platform calculates a “net improvement” metric to measure the model’s actual contribution in real-world tasks.

6.2 Evaluation Results

In an evaluation published on the day of the weight release, based on over 17,500 real long-horizon sessions, the model (Max tier) achieved a net improvement of +5.1%. It ranked 1st among open-weight models and 13th overall, securing 1st place in the open-weight category across Work, Code, and Chat classifications. The previous generation model had a net improvement of +0.91% and ranked 6th among open-weight models.

In specific sub-categories, this model achieved a Bash reliability of +7.9% among open-weight models, ranking 1st (compared to the predecessor’s −4.5% and 13th place). This indicates a substantial improvement in model stability during long-horizon, multi-step command-line tasks.

7. Selection Recommendations and Practical Key Points

Overall, this class of ultra-large open-weight models is suitable for the following scenarios:

1. Requires deep customization: Intending to fine-tune based on domain-specific data.
2. High data compliance requirements: Must be locally deployed and cannot use public APIs.
3. Requires model behavior auditing: Ability to inspect weights, routing, and activation distributions.

For implementation, it is recommended to:

• Use the API first to validate task feasibility, then evaluate the ROI of local deployment.
• Prioritize real conversational data for evaluation rather than purely synthetic benchmarks.
• Pay attention to the compatibility between quantization formats and inference frameworks to avoid discovering precision or performance issues post-launch.
• Always retain reasoning_content in multi-turn tool calling scenarios to maintain context consistency.

The release of open weights for trillion-parameter MoE models signifies the evolution of the large model ecosystem from “black-box APIs” toward “auditable and customizable” paradigms. For developers, understanding the architecture, quantization schemes, and evaluation logic is the prerequisite to making correct technical decisions.