Trillion-Parameter MoE Model Weights: Sparse Attention and Low-Precision Deployment Practices

Open-Sourcing Trillion-Scale MoE Model Weights: A Comprehensive Technical Breakdown from Architecture to Deployment

The release of open weights for large-scale Mixture-of-Experts (MoE) models is transforming the question of “can we run this ourselves?” from a spectator topic into a critical engineering decision. Recently, a model team fulfilled their commitment to open-sourcing the complete weights of an MoE model with approximately 2.8 trillion total parameters, alongside its performance metrics from third-party, real-world conversational evaluations. This article breaks down the model across four dimensions—architecture, quantization and storage, deployment thresholds, and evaluation signals—to help developers assess its applicability and boundaries.

1. Model Architecture: Sparse Attention as the Key Variable for Long Contexts

According to the public model card, the model consists of 93 layers: 69 layers utilize a KDA structure, while 24 layers employ Gated Sparse Attention. Compared to its predecessor, the most significant architectural shift is replacing the previous fully-attentional gated MLA layers with gated sparse attention layers. This change explicitly targets cost control for long-context modeling.

To understand this, consider the fundamental background: standard full attention scales quadratically in computational and memory overhead as sequence length increases. When the context window is pushed to the million-token level, full attention becomes practically unfeasible. Sparse attention restricts the key-value range each token can attend to, compressing the overhead to near-linear scaling. The trade-off is the need for more sophisticated gating designs to maintain information flow. This model adopts a hybrid layering strategy common in the industry—retaining strong global modeling capabilities in the lower layers while using sparsity in the higher layers to boost throughput.

The expert layer is configured with 896 experts, activating 16 per token, alongside 2 shared experts. The shared experts handle general capabilities, preventing capability degradation caused by routing fluctuations. The 16-out-of-896 activation ratio means the actual number of parameters involved in computation for a single token is far less than the total parameter count, which is the economic prerequisite for inferencing trillion-scale models. The vision component is an independent encoder with about 400 million parameters supporting image-text inputs, indicating the model is positioned as a multimodal Agent foundation rather than a pure text model.

2. Quantization Strategy: Trade-offs in MXFP4 Storage and MXFP8 Activations

The weights are stored in MXFP4 format, while activations utilize MXFP8. Official notes confirm that Quantization-Aware Training (QAT) was employed during the training phase. These choices warrant a dedicated discussion.

MXFP4 is a microscaled floating-point format where each value is represented by only 4 bits, but it preserves dynamic range through shared exponent scaling factors. Compared to traditional symmetric INT4 quantization, it offers better fidelity for small values at ultra-low precision, making it suitable for scenarios with massive parameter counts and high tolerance for single-value precision loss. The direct benefit of using FP4 for storage is size reduction: the repository occupies about 1.56TB, while the parameter count measured by safetensors is roughly 2.78 trillion. This translates to less than 0.6 bytes per parameter, aligning with expectations for 4-bit storage plus minor scaling overhead. If FP16 storage were used instead, the repository size for the same parameter count would approach 5.5TB.

Using MXFP8 rather than FP4 for activations is a deliberately conservative choice. Activation values change dynamically during inference, and their distribution drifts with the input. 4-bit precision can easily introduce uncontrollable errors in routing decisions and attention scoring; 8-bit strikes a balance between memory bandwidth and numerical stability. The inclusion of QAT means quantization errors were modeled during training rather than forcefully applied post-training. This is particularly crucial for trillion-scale models—post-training quantization often requires massive amounts of calibration data and struggles to recover precision in ultra-large models.

It is important to note that low-precision storage does not equate to low-precision computation. During actual inference, weights are de-quantized to the compute precision before participating in operations. Therefore, memory footprint and compute requirements must be evaluated separately.

3. Deployment Paths: Multi-Framework Adaptation and Hardware Thresholds

Currently, mainstream inference frameworks provide deployment recipes covering both high-throughput serving and lightweight scheduling scenarios. A typical deployment process can be abstracted into the following steps:

# 1. Pull the weight repository (approx. 1.56TB, parallelized shard download recommended)
# 2. Convert or directly load safetensors shards according to framework requirements
# 3. Configure tensor parallelism and expert parallelism degrees
# 4. Start the service and verify long-context and multimodal input pathways

The critical configuration lies in the parallelism strategy. Even at 4-bit storage, 2.8 trillion parameters require about 1.5TB of VRAM, which cannot fit on a single machine and necessitates cross-node sharding. Expert Parallelism (EP) distributes the 896 experts across different devices, while Tensor Parallelism (TP) shards the weight matrices within a single layer. Both must be matched with interconnect bandwidth—the all-to-all communication in EP puts significant pressure on the network, making high-bandwidth interconnects practically a necessity.

Third-party estimates suggest a feasible configuration comprising 8 high-end accelerators with about 2.3TB of total VRAM, costing in the hundreds of thousands of dollars. This figure is merely a community estimate; the official team has not released hardware recommendations. Actual procurement should be recalculated based on target throughput, context length, and concurrency. For most teams, a more pragmatic approach is to validate business value via managed APIs first, before assessing the feasibility of self-hosting.

On the API side, standard and fast model identifiers are provided, compatible with mainstream conversational interface specifications. Inference intensity offers low, high, and maximum tiers, defaulting to maximum. In multi-turn tool-calling scenarios, there is a common pitfall: the reasoning content and tool call records from the previous turn must be passed back together. Otherwise, the model loses the intermediate reasoning chain, resulting in repeated tool calls or parameter errors. This requires explicit handling when constructing Agent loops.

4. Evaluation Signals: The Significance of Net Improvement in Real-World Sessions

The evaluation data released alongside the weights comes from real user Agent sessions rather than static benchmarks. Its methodology is noteworthy: multiple signals are extracted from sessions—whether the user explicitly confirms task completion, positive and negative feedback, the model’s recovery after user corrections, and command execution success—and synthesized into a “net improvement” metric.

Based on over 17,500 long-horizon sessions, the model achieved a net improvement of approximately 5.1% at the highest inference tier, ranking first among open-weight models and thirteenth overall. The previous generation at the same tier was roughly 0.91%. By category, it leads open-weight models in Work, Code, and Chat. In specific sub-items, command-line environment reliability improved by about 7.9%, whereas the previous generation scored negatively.

Two things must be noted when interpreting this data. First, “net improvement” is an incremental metric relative to a baseline model. A positive value represents an actual improvement in user experience, while a negative one represents a regression. Therefore, the transition from negative to positive is more significant than the absolute value. Second, while real-session evaluations cover long-horizon tasks better than static leaderboards, they are influenced by the user demographic and task distribution, and cannot be directly extrapolated to all business scenarios. Teams should construct their own task sets for comparative evaluations during model selection, paying special attention to tool-calling stability and error-recovery capabilities—these two factors are more decisive for Agent deployment success than single-turn QA quality.

5. Implementation Recommendations

For teams considering adopting this model, the following sequence is recommended:

1. Clarify Context Requirements: If the business does not involve ultra-long documents or long-horizon Agent loops, the benefits of sparse attention will be limited. Prioritize evaluating smaller-scale models instead.
2. API First, Self-Hosting Second: Use managed interfaces to validate the business pipeline and evaluation sets. Once capabilities meet the standard, calculate the cost of self-hosting.
3. Verify Quantization Format Compatibility: Ensure the chosen inference framework has mature support for MXFP4/MXFP8 to avoid loading failures or precision anomalies.
4. Design Agent State Management: Incorporate reasoning content and tool call records into the session state to ensure continuity across multi-turn calls.
5. Establish a Proprietary Evaluation Baseline: Use real business sessions as samples to continuously track incremental metrics like net improvement, rather than relying solely on public leaderboards.

The open-sourcing of trillion-scale MoE model weights essentially shifts the barrier to acquiring “frontier capabilities” from the training side to the deployment side. For teams with the necessary compute and engineering capabilities, this enables fine-tuning, auditing, and private customization. For the majority, a more reasonable approach is to consume these capabilities via managed services, focusing engineering efforts on Agent orchestration and state management—the areas that truly drive differentiation.