Open Weights for a 2.8T Parameter MoE Model: Architecture Analysis and Local Deployment Practices

Background: Open Weights and Industry Impact

In the Large Language Model (LLM) domain, the openness of model weights directly determines whether the technical community and enterprise users can perform deep customization, security audits, and localized deployment. Recently, a leading AI vendor fulfilled its previous commitment by officially uploading the complete weights of its latest generation Mixture-of-Experts (MoE) LLM to mainstream open-source model repositories. This move not only honors its release promise but also provides the global developer community with an opportunity to study the internal mechanisms of a frontier model.

The model has a total parameter scale of 2.8 trillion (2.8T). However, its innovation lies in its sparse activation MoE architecture, where only about 104 billion (104B) parameters are activated per token during forward inference. Natively, the model supports a 1 million token context window and possesses image-text multimodal understanding capabilities. The release of the weight files means that teams with the necessary hardware resources and engineering capabilities can bypass API calls to directly perform private deployment, targeted fine-tuning, or model interpretability research.

Core Technical Architecture: Hybrid Attention and Expert Routing

1. Layer Structure Design: Collaboration between KDA and Gated DSA

The model card indicates that the model contains 93 Transformer layers. Instead of simple repetition, the internal structure adopts an innovative hybrid design: 69 layers are KDA (Key-Value Decomposed Attention) layers, while the remaining 24 are Gated DSA (Gated DeepSeek Sparse Attention) layers.

• KDA Layers: As the primary attention layers, these handle dense contextual information interactions. The core idea is to decompose the Key and Value matrices in attention computation to reduce the parameter count while maintaining high-fidelity information extraction.
• Gated DSA Layers: This is an engineering optimization for long-context scenarios. Compared to KDA layers, Gated DSA introduces a gating mechanism and sparse attention patterns. In ultra-long contexts, not all historical tokens are highly relevant to the current token. Gated DSA uses learnable gating units to dynamically filter the subset of tokens requiring attention computation. This significantly reduces the computational complexity of long contexts from quadratic to linear, drastically saving VRAM usage. This layered hybrid strategy of “dense attention + sparse attention” optimizes inference costs while preserving long-range dependency modeling capabilities.

2. Expert Network and Routing Strategy

In the Feed-Forward Network (FFN) section, the model is configured with a total of 896 Expert modules. During each forward pass, the Router dynamically selects 16 of these experts for computation based on the input token’s features. Additionally, the model retains 2 shared experts to ensure that regardless of which expert a token is routed to, it receives a stable foundational feature representation. This large-scale Expert Parallelism strategy allows the model to maintain immense parameter capacity while sustaining high computational efficiency. The computation per token is equivalent to a dense model with around 100 billion parameters.

Model Storage and Quantization Strategy: MXFP4 and MXFP8

For a 2.8T parameter model, traditional FP16/BF16 storage formats would consume about 5.6TB of VRAM, far exceeding the capacity of a single current server. The weights utilize the industry’s latest MX (Microscaling) floating-point formats for quantized storage.

• Storage Layer (Weights): The weight data is stored in the 4-bit MXFP4 format. By having a group of values (e.g., 32) share a single scaling factor, this format retains a high dynamic range at extremely low bit rates, resulting in significantly lower quantization loss than traditional INT4/INT8 uniform quantization.
• Activation Layer (Computation): During inference, activations use the 8-bit MXFP8 format. It is emphasized that the model employs Quantization-Aware Training (QAT), meaning the precision loss of low-bit deployment was simulated during the training phase. This ensures the model weights maintain near-native performance post-quantization.

Repository information indicates that the total storage requirement for this weight package is approximately 1.56TB. This size makes it possible to load the model on servers equipped with multiple high-VRAM GPUs.

Deployment Scenarios and Hardware Thresholds

1. Inference Framework Adaptation Ecosystem

Following the model’s release, mainstream open-source inference frameworks have completed real-time adaptations. Currently, vLLM, SGLang, and TokenSpeed provide clear deployment recipes. These frameworks have implemented low-level optimizations for MXFP4 weight loading and MXFP8 kernel computations, allowing developers to load weights directly via unified APIs without manually writing specific operators. This indicates that the engineering pipeline from “release” to “usable” is fully established.

2. Hardware Configuration and Cost Estimation

Despite quantization and sparsification technologies, the 2.8T parameter base dictates an extremely high local deployment threshold. According to third-party community estimates, a viable minimum inference configuration requires at least 2.3TB of total GPU VRAM. This means building a server cluster consisting of 8 NVIDIA flagship B300 compute cards (approximately 288GB HBM3e VRAM per card). Hardware procurement costs alone are estimated between $550,000 and $650,000, excluding infrastructure costs such as racks, cooling, and power capacity upgrades. Therefore, fully private deployment of this model is primarily suited for large enterprises or research institutions. For small to medium teams, accessing the model via cloud API remains the mainstream choice.

Note: When accessing via API, the model offers both a Standard and a HighSpeed mode, with the latter claiming a 2x to 3x improvement in response speed. Developers should note that in complex, multi-turn tool-calling scenarios, the reasoning process content and tool call records from the previous turn must be passed into the context to maintain Agent state continuity.

Capability Benchmarks and Real-World Session Evaluation

Regarding the model’s performance in Agentic tasks, an authoritative evaluation platform conducted blind tests based on over 17,500 real-world long-horizon user sessions. The evaluation mechanism was not simple Q&A scoring, but a comprehensive synthesis of five key signals: explicit user markers of task success, qualitative feedback on model output, the model’s correction ability after user feedback, and the actual execution success rate of underlying Shell commands.

In the latest evaluation leaderboard, the model (with inference intensity set to the highest level) achieved a “+5.1%” Net Gain, ranking first among open-weight models and thirteenth in the overall model leaderboard. It secured the top spot among open-weight models in three major vertical categories: software development, daily office tasks, and conversational interaction. For comparison, its predecessor model achieved a Net Gain of +0.91% and ranked sixth, demonstrating a qualitative leap in instruction following and task completion efficiency in this iteration.

In the core Bash tool-calling reliability test, the new model achieved a +7.9% positive improvement (the predecessor model scored -4.5%, meaning the previous version not only failed to complete tasks in some scenarios but also negatively interfered with system states). This data intuitively proves the new model’s significant progress in complex environment operations, script execution, and result verification.

Technical Selection and Conclusion

For technical decision-makers, the availability of these weights provides a “controllable and auditable” path for private deployment. Compared to black-box APIs, holding the weights means complete control over data flow, mitigating the risk of sensitive data leakage. However, this is not a low-cost shortcut. The 2.8T parameter scale imposes extreme requirements on distributed parallelism strategies (especially Expert Parallelism), VRAM management (Offload and Quantization), and underlying communication libraries.

It is recommended that teams prioritize validating the model’s Agent interaction capabilities via API before project initiation. Confirm its Net Gain performance in target scenarios, and then weigh the hardware budget against the value of building a local inference cluster. For most application-layer development enterprises, building applications based on cloud-hosted services and focusing efforts on business logic and workflow orchestration remains the most cost-effective strategy.