Trillion-Parameter MoE Open Weights: A Full Breakdown of Architecture, Deployment, and Evaluation

1. Background: Why Open Weights Are a Watershed Moment

As large model competition enters its later stages, “open weights” has evolved from a gesture into a public test of engineering capability. Whether a model can hand its complete weights to developers directly exposes its maturity across three chains: training stability, quantization compression, and inference adaptation—because once weights are open, anyone can reproduce, fine-tune, or audit them on their own hardware.

Recently, a leading model team fulfilled its commitment to “release weights within two weeks of launch,” uploading the complete weights of a trillion-parameter MoE model to a public model repository. The significance of this move is not that “another model was open-sourced,” but that it places three capabilities—ultra-large-scale sparse models + million-token context + multimodal input—on the table in a downloadable, deployable, and verifiable form for the first time. For teams that need self-hosted deployment, domain fine-tuning, or security auditing, this is a genuinely usable technical path, not just a number on an evaluation leaderboard.

This article breaks down the technical essentials of this open-weight release across five dimensions: model structure, quantization scheme, deployment paths, hardware thresholds, and real-session evaluation.

2. Model Structure: Sparse Design Across 93 Layers

2.1 Overall Parameter Distribution

According to the public model card, the model uses a MoE (Mixture of Experts) architecture with the following core structural parameters:

• Total layers: 93
• Attention layer composition: 69 layers use KDA (a linear attention variant), 24 layers use Gated DSA (gated sparse attention)
• Expert scale: 896 experts total, 16 activated per token, plus 2 shared experts
• Activated parameters: approximately 104B per token
• Total parameters: approximately 2.78 trillion (based on weight file statistics)
• Context length: supports 1 million tokens
• Vision encoder: approximately 401M parameter multimodal vision tower

The most noteworthy aspect here is the “hybrid” attention strategy. Compared to the previous generation’s approach of uniformly replacing all full-attention layers with gated MLA (multi-head latent attention), this generation changes full-attention layers to gated sparse attention. The core idea of sparse attention is: rather than having every token attend to all historical tokens, a learnable gating mechanism filters out the truly relevant key-value pairs for computation. In long-context scenarios, this yields dual benefits—it reduces the complexity growth curve of attention computation and decreases KV Cache memory usage.

2.2 Why “Sparse + Linear” Hybrid

Pure linear attention is extremely efficient on ultra-long sequences but tends to suffer information decay on tasks requiring precise retrieval; pure full attention offers good accuracy but costs grow quadratically with length. The combination of 69 linear layers + 24 sparse full-attention layers essentially creates a layered division of labor between “efficiency” and “precise recall”: shallow and middle layers use linear attention to rapidly compress context, while key layers use sparse attention for precise localization.

For developers, this means you cannot simply estimate deployment costs as “layer count × per-layer memory” because the KV Cache structure and computation graph differ across layers. Actual deployment must rely on inference frameworks with dedicated support for hybrid attention patterns.

3. Quantization and Storage: The MXFP4 + MXFP8 Combination

3.1 Separating Storage Format from Activation Precision

A key engineering decision for these weights is the separation of storage and computation precision:

• Weight storage: MXFP4 (4-bit floating point with micro-scaling factors)
• Activation computation: MXFP8 (8-bit floating point)

The core of the MX format family is “block-level shared exponents”—a group of values (typically 32) shares a single scaling factor, maintaining good numerical dynamic range even at extremely low bit widths. Compared to traditional INT4 symmetric quantization, MXFP4 is more stable when handling long-tail weight distributions.

The official documentation states that the model uses quantization-aware training (QAT), which is critical. QAT means the model was already “aware” during training that weights would be compressed to 4 bits and activations to 8 bits, so quantization errors were explicitly optimized during training. This differs fundamentally from “train-then-PTQ post-quantization”: the latter’s accuracy loss tends to concentrate in a few sensitive layers, while QAT can distribute errors more evenly.

3.2 Actual Storage Footprint

The public repository interface shows the weight repository occupies approximately 1.56TB. This number is broadly consistent with the theoretical value of “2.78 trillion parameters × 4 bits” (2.78e12 × 0.5 byte ≈ 1.39TB; adding scaling factors, embedding layers, vision tower, and other non-quantized components, 1.56TB is a reasonable range).

The direct implication for deployment teams is: weight files alone require 1.5TB+ of high-speed storage. If multi-replica inference is needed, storage and loading bandwidth will become the first bottleneck. It is recommended to implement preloading and caching strategies on network storage or local NVMe.

4. Deployment Paths: Three Mainstream Inference Chains

Public information indicates that at least three mainstream inference frameworks have provided deployment recipes for this model, covering both high-throughput serving and low-latency scenarios. While specific commands vary by framework, the deployment logic is highly consistent and can be abstracted into the following steps:

4.1 General Deployment Flow

# Pseudocode: general deployment logic
1. Pull the weight repository (~1.56TB; use a download tool that supports resume)
2. Verify safetensors shard integrity and hashes
3. Select an inference framework and load the corresponding recipe configuration
4. Configure tensor parallelism (TP) and pipeline parallelism (PP)
5. Configure KV Cache precision and capacity limits
6. Enable dedicated kernels for hybrid attention layers
7. Start the service and run health checks

4.2 Key Configuration Notes

• Tensor parallelism (TP): Due to the massive per-layer parameter count, TP must span at least 8 GPUs. Hybrid attention layers have different sensitivity to parallelism strategies—linear attention layers are more TP-friendly, while sparse attention layers require framework support for cross-GPU sparse routing.
• KV Cache management: A 1-million-token context means KV Cache can reach hundreds of GB. Paged Attention and KV Cache quantization must be enabled, or memory will be exhausted before weights are fully loaded.
• Expert parallelism (EP): 896 experts cannot all reside on a single GPU. MoE layers require expert parallelism plus dynamic routing, and the framework must support all-to-all communication optimization, or communication overhead will consume the gains from sparsification.
• Multimodal input: The vision tower has approximately 401M parameters and is typically loaded separately; image-text input requires an additional image preprocessing pipeline.

4.3 Realistic Hardware Threshold Estimates

Regarding local deployment hardware costs, currently available estimates come from third-party analysis: one feasible configuration is a server with 8 high-end data center GPUs and approximately 2.3TB of GPU memory, with total hardware costs in the $550,000 to $650,000 range. It should be emphasized that this is a third-party estimate; no official hardware recommendation has been provided.

From an engineering perspective, this scale is not “accessible to individual developers,” but it is plannable for mid-to-large enterprises, research institutions, and cloud service providers. For those who do not want to build their own infrastructure, API access is a more realistic option—the public API model identifier is compatible with mainstream interface specifications and offers standard and high-speed tiers, with the latter approximately 2 to 3 times faster than the former.

5. Agent Capabilities: How to Read Real-Session Evaluations

5.1 The Distinctive Nature of the Evaluation Data

The agent evaluation leaderboard released alongside this model deserves separate mention for its data source: it does not come from manually constructed test sets, but from real user agent sessions. Evaluators extract multiple signals from sessions:

• Whether the user explicitly confirmed task success
• User praise or complaints about model output
• Response quality after user corrections
• Whether shell commands executed successfully

Based on these, a “net improvement” metric is calculated. The advantage of this approach is proximity to real workloads; the disadvantage is higher signal noise and difficulty controlling variables completely.

5.2 Interpreting Key Results

Based on over 17,500 real long-horizon sessions, the model (at its highest reasoning tier) achieved:

• Net improvement of +5.1%, ranking 1st among open-weight models and 13th overall
• Ranked 1st among open-weight models in all three categories: Work, Code, and Chat
• The previous-generation model had a net improvement of +0.91%, ranking 6th among open-weight models

In the breakdown, Bash reliability reached +7.9%, ranking 1st among open-weight models, while the previous generation scored −4.5% and ranked 13th. This sub-metric deserves particular attention: Bash reliability directly reflects the model’s ability to execute in real command-line environments—whether it can correctly construct commands, handle errors, and recover from failures. The shift from negative to positive indicates substantive improvement in “tool-calling robustness” in this generation.

5.3 API-Side Considerations

For developers using this model via API, two engineering details deserve attention:

1. reasoning_effort parameter: Options are low, high, and max, with max as the default. In latency-sensitive scenarios, explicitly lowering this parameter can significantly reduce inference time, at the cost of some accuracy on complex tasks.
2. Context回传 for multi-turn tool calls: In multi-turn tool-calling scenarios, the previous turn’s reasoning content must be passed back along with tool_calls; otherwise the model loses its intermediate reasoning chain, degrading subsequent decision quality.

# Multi-turn tool call context passing (pseudocode)
messages.append({
    "role": "assistant",
    "content": response.content,
    "reasoning_content": response.reasoning_content,   # must be preserved
    "tool_calls": response.tool_calls                  # must be preserved
})

6. Practical Recommendations for Developers

Based on the above analysis, teams in different roles can adopt different strategies:

• Research/audit teams: Download the complete weights directly and focus on the quantization error distribution after QAT, the actual behavior of hybrid attention layers, and expert load balancing in sparse routing.
• Enterprises with self-hosting capability: Start with small-scale deployment validation (e.g., reduced context length, limited expert parallelism), measure real throughput and memory usage, then decide whether to move to production clusters.
• Application development teams: Prioritize API access, use reasoning_effort tiers to control costs, and strictly ensure multi-turn context integrity in agent scenarios.
• Fine-tuning teams: Note the compatibility of 4-bit stored weights with fine-tuning frameworks; fine-tuning QAT weights typically requires framework support for low-precision training or dequantization first.

The real value of trillion-parameter MoE open weights is not in the parameter count, but in making public the combined engineering solutions for long context, sparse activation, low-precision quantization, and multimodality. For teams planning next-generation inference infrastructure, these weights themselves are a technical reference sample that can be dissected layer by layer.