A Trillion-Scale MoE Model Releases Open Weights: Architecture and Deployment Behind 2.8T Parameters

1. Background: A Promise Delivered and Open Weights

In early November 2024, an AI research organization uploaded the complete weights of its flagship Mixture-of-Experts (MoE) model to a public model hosting platform, fulfilling a prior commitment to open the weights by November 4. The move drew significant attention from the developer community, not only because of the model’s sheer size, but also because it signals a broader trend: ultra-large models are moving beyond pure API services toward an open paradigm where weights can be obtained, audited, and deployed locally.

For teams that need to deploy, fine-tune, or audit model behavior themselves, open weights mean several things:

• Reproducibility: No longer dependent on a black-box API. Outputs can be validated in a self-hosted environment.
• Customizability: Domain-specific fine-tuning and alignment adjustments become possible.
• Auditability: Security teams can perform static and dynamic analysis on the weights.
• Cost control: For sustained high-frequency usage, local inference may be more economical than per-token billing.

This article breaks down the technical essentials of this trillion-scale model across four dimensions: architecture design, quantization strategy, deployment paths, and evaluation methodology.

---

2. Model Architecture: Sparse Activation and Hybrid Attention

2.1 Overall Parameters and Activation Strategy

The model has approximately 2.78 trillion total parameters (based on weight file statistics) and uses a MoE architecture where only about 104B parameters are activated per token. This sparse activation design has become the mainstream approach for ultra-large models: a routing mechanism directs different inputs to only a subset of expert networks, keeping total capacity enormous while limiting per-inference compute to an acceptable range.

2.2 Layer Structure and Attention Mechanism

The model has 93 layers in total, with a hybrid attention design:

• 69 KDA (Kernelized Differential Attention) layers: Used for efficient short-to-medium-range dependency modeling.
• 24 Gated DSA (Gated DeepSeek Sparse Attention) layers: Replace the previous generation’s full-attention Gated MLA layers, specifically optimized for long-context computational efficiency.

The core change here is replacing full attention with sparse attention. At a target context length of 1 million tokens, the O(n²) complexity of full attention creates prohibitive memory and compute overhead. Sparse attention computes only a subset of token-pair relationships, preserving long-range modeling capability while substantially reducing resource consumption. A gating mechanism dynamically adjusts the contribution weight of sparse attention.

2.3 Expert Layer Configuration

• 896 experts, with 16 activated per token.
• An additional 2 shared experts that always participate in computation, carrying general knowledge.

This combination of “fine-grained experts + shared experts” is a common best practice in current MoE design: fine-grained experts improve specialization, while shared experts prevent degradation of general capability caused by routing instability.

2.4 Vision Modality Support

The vision encoder is MoonViT-V2 with 401M parameters, supporting mixed image-text inputs. This enables the model to handle multimodal agent tasks such as screenshot understanding, UI operation, and document parsing.

---

3. Quantization and Storage: The Engineering Trade-off of MXFP4 + MXFP8

3.1 Storage Format

Weights are stored in MXFP4 format, with activations using MXFP8. MXFP is a low-precision floating-point format promoted by the industry in recent years, designed specifically for machine learning workloads. It substantially reduces storage and bandwidth requirements while maintaining numerical stability.

The official documentation states the model uses Quantization-Aware Training (QAT) , meaning the numerical behavior of low-precision inference is simulated during the training phase, making the model weights and activations robust to quantization error. Compared to Post-Training Quantization (PTQ), QAT typically results in smaller accuracy degradation.

3.2 Actual Footprint

The hosting platform interface shows a total repository size of approximately 1.56TB, with weight file statistics indicating roughly 2.78 trillion parameters. For 2.8T parameters stored at 4 bits, the theoretical minimum size is about 1.4TB. Adding quantization scale factors, metadata, and the vision encoder, 1.56TB is a reasonable result.

3.3 Deployment Implications

Low-precision storage directly lowers the GPU memory threshold and loading time, but weights must still be dequantized to compute precision during inference, so actual memory usage will exceed the storage size. This is why local deployment still requires a multi-GPU, high-memory configuration even with MXFP4.

---

4. Deployment Paths: Mainstream Inference Framework Support

Alongside the weight release, three mainstream inference frameworks have provided deployment recipes:

Framework	Support Status	Typical Use Case
vLLM	Recipe available	High-throughput online serving
SGLang	Recipe available	Structured generation and agent scenarios
TokenSpeed	Recipe available	Low-latency inference

These frameworks all support tensor parallelism (TP) and pipeline parallelism (PP) to shard 2.8T parameters across multiple GPUs. Deployment typically requires:

1. Confirming the framework version supports this model architecture (sparse attention + MoE routing).
2. Configuring parallelism degree and memory allocation.
3. Loading quantized weights and calibrating the dequantization strategy.
4. Stress-testing throughput and latency, then tuning batch size and KV cache strategy.

Example launch configuration (generic form):

# Generic multi-GPU deployment example; adjust parameters per framework docs
python -m serve \
  --model /path/to/weights \
  --tensor-parallel-size 8 \
  --quantization mxfp4 \
  --max-model-len 1000000 \
  --enable-chunked-prefill

4.1 Hardware Threshold Estimate

A third-party estimate suggests one viable configuration is 8 high-memory GPUs (approximately 2.3TB total VRAM) , with hardware costs around $550,000 to $650,000. It should be emphasized that this is a third-party estimate; the official team has not provided hardware recommendations. The actual threshold depends on:

• Whether quantized inference is used.
• Target context length.
• Concurrency requirements.
• Whether CPU offload is enabled.

For most teams, API access remains the more realistic starting point.

---

5. API Integration Essentials

For teams that do not self-host, the API exposes two model names:

• Standard version: Compatible with OpenAI and Anthropic interface styles.
• Fast version: Officially stated to be 2–3x faster than the standard version.

Key parameters:

• reasoning_effort: Options are low, high, or max, with max as the default.
• Multi-turn tool calls: The previous turn’s reasoning_content and tool_calls must be passed back together; otherwise, the model loses its reasoning context.

This is particularly important for agent developers: the reasoning chain is not a stateless call, but session state that must be explicitly maintained.

---

6. Evaluation Methodology: Agent Arena Based on Real Sessions

6.1 Data Sources and Signal Design

Agent Arena’s data comes from real user agent sessions, not synthetic benchmarks. It extracts several categories of signals:

• Whether the user explicitly confirmed task success.
• Praise and complaints about the model.
• Response quality after user corrections.
• Whether shell commands executed successfully.

From these, it computes a Net Improvement metric.

6.2 Evaluation Results

Based on more than 17,500 real long-horizon sessions:

• The model (Max setting) achieved a Net Improvement of +5.1%.
• Ranked #1 among open-weight models and #13 overall.
• Ranked #1 among open models in all three categories: Work, Code, and Chat.
• The previous-generation model (Max) scored +0.91%, ranking #6 among open models.

By subcategory, Bash reliability improved by +7.9%, ranking #1 among open models; the previous generation scored −4.5%, ranking #13.

6.3 Methodological Takeaways

The value of this evaluation lies in the fact that it measures “net benefit in real tasks” rather than single-turn QA accuracy. For agent scenarios, this is far closer to production conditions than traditional benchmarks. Developers can borrow the signal design approach to build their own internal evaluation sets:

# Abstract representation of net improvement signals
signals = {
    "explicit_success": bool,
    "praise_or_complaint": int,
    "post_correction_quality": float,
    "shell_exit_code": int
}
net_improvement = weighted_sum(signals)

---

7. Summary and Practical Recommendations

The core takeaways from this open-weight release can be summarized as:

1. Architecture: 93 hybrid attention layers + 896-expert MoE + 401M vision encoder.
2. Quantization: MXFP4 storage / MXFP8 activations, QAT, repository size approximately 1.56TB.
3. Deployment: Mainstream inference frameworks already support it, but local hardware requirements are high.
4. API: Two model names, compatible with mainstream interfaces, with attention to reasoning chain state management.
5. Evaluation: Agent Arena based on real sessions, Net Improvement of +5.1%, #1 among open-weight models.

For developers, a practical progression would be:

• Validation phase: Start with the API to quickly verify task fit.
• Cost evaluation: Only calculate local deployment TCO for high-frequency scenarios.
• Evaluation infrastructure: Build an internal agent evaluation set informed by real-session signals.
• Fine-tuning preparation: Plan data and compute in advance if domain adaptation is required.

The open-weight release of a trillion-scale MoE model is turning “ultra-large models” from a service controlled by a few organizations into infrastructure that more teams can study and customize. The long-term impact of this trend on the agent ecosystem deserves continued attention.