Open-Weight 2.8T-Parameter MoE Model: Architecture Analysis and Local Deployment Practices
This article covers the architecture, quantization strategy, local deployment requirements, and license terms of a newly released open-weight MoE model with 2.8T total parameters. It also reviews the model's benchmark results, which are strong in coding and agent tasks but weaker in science and some reasoning tests.
As large language models advance, very large mixture-of-experts (MoE) models have become a key way to get past compute bottlenecks and improve model capability. A leading AI research team recently released the full weights of its latest very large model, as planned. The model has 2.8T total parameters and a file size of about 1.56TB. It is claimed to be among the strongest open-weight models available for download.
The release sets high barriers for architecture, local deployment, and commercial use. This article focuses on the technical side. It examines core architecture, quantization strategy, local inference deployment, and license compliance.
1. Architecture: Combining MoE and Multimodal Design
The model is positioned as a natively multimodal agent model. It was continued from the previous generation and trained further. Its core design uses MoE, a leading approach that balances parameter scale with compute efficiency.
1. Sparse Activation and Expert Routing
The model has 2.8T total parameters, but only 104B are activated for each token during inference. The MoE structure includes 896 expert networks. For each token, a gating network selects the 16 experts that best match the input. The architecture also includes 2 shared experts. These are activated in every computation and capture general features across tasks.
The following simplified pseudocode illustrates the MoE gating and routing logic:
def moe_routing(hidden_states, num_experts=896, top_k=16):# Compute gating scoresgate_logits = torch.matmul(hidden_states, gating_weights)# Select the Top-K expertstop_k_logits, top_k_indices = torch.topk(gate_logits, k=top_k, dim=-1)# Combine with shared expert outputsshared_experts_output = compute_shared_experts(hidden_states)selected_experts_output = compute_selected_experts(hidden_states, top_k_indices)# Merge the final outputfinal_output = combine_outputs(shared_experts_output, selected_experts_output)return final_output
2. Long Context and Sparse Attention
The model has 93 layers and supports a context window of up to 1 million tokens. The compute cost of full attention grows quadratically with context length. To address this, the architecture replaces the original 24 full-attention layers with sparse attention. This substantially reduces memory and compute use for long text without significantly sacrificing model performance, which makes million-token context practical.
3. Native Multimodal Capability
The model includes a dedicated vision encoder for image understanding. Multimodal features are aligned and fused in the lower layers, not attached as an external module. This supports complex agent tasks such as UI operation and chart analysis.
2. Weight Quantization and Distribution Challenges
1. Tradeoffs of Quantization-Aware Training (QAT)
The team released only one quantized version. No BF16 or standard FP8 version is available. The release uses MXFP4 for weights and MXFP8 for activations, and it was produced through quantization-aware training. The model simulated quantization loss during training, which helps it keep performance in low-bit inference. For developers, this single low-bit version greatly reduces the file size, to about 1.56TB. It also limits further fine-tuning and higher-precision inference.
2. Storage and Distribution of Large Files
The 1.56TB of weights consist of 112 files and 96 data shards. Handling this volume places heavy demands on network bandwidth and storage. Downloading such data directly from a public platform is slow and prone to interruption.
Developers often rely on highly available, high-bandwidth cloud infrastructure for this task. For example, Baidu AI Cloud object storage can be used to build a stable repository for large files. Combined with CDN acceleration or dedicated VPC network connections, TB-scale weights can be synchronized quickly and reliably between local clusters and the cloud. This helps keep deployments on schedule.
3. Local Deployment and Inference Hardware Requirements
Sparse activation keeps the compute per inference close to that of a dense 104B model. However, all 2.8T parameters must reside in GPU memory or system memory. This creates a high hardware barrier.
1. Memory Requirements Starting at 8 GPUs
The official documentation states that local deployment requires at least 8 high-end GPUs. With 80GB-class GPUs, eight cards provide 640GB of memory in theory. Model weights, KV cache, CUDA context overhead, and framework memory fragmentation all consume additional memory. Loading 1.56TB of quantized weights therefore leaves a large memory gap. In practice, deployments often use CPU offloading, which moves inactive expert weights to host memory. Others rely on multi-node distributed inference.
2. Inference Framework Support
Running this model smoothly on limited hardware requires an inference engine with deep optimization for MoE routing and low-bit computation. Developers should check framework support for MXFP4 and MXFP8 operators, as well as accelerated implementations of sparse attention.
Enterprises can use Baidu AI Cloud GPU compute and container platform services to build elastically scaling inference clusters. These clusters can allocate resources based on request traffic. This helps maintain response latency while lowering overall compute costs.
4. License Compliance for Open Weights
The most notable part of this release is its license. Major model hosting platforms list a license for the model, but it is not a standard open-source license. It uses custom terms. This is the key difference between “open weights” and “open source.”
1. Permissions and Restrictions
The license resembles a permissive open-source license. It allows users to use, copy, modify, distribute, and fine-tune the model, and to use it commercially, at no cost. Two clauses restrict large-scale commercial use:
- Companies that offer the model as a service (MaaS) to external customers, with cumulative revenue above USD 20 million over 12 months, must sign a separate agreement with the copyright holder before commercial use.
- Commercial products with more than 100 million monthly active users also require additional authorization.
2. Impact on Enterprise Use
This tiered licensing means startups and small and mid-sized developers can integrate the model into their own products without these restrictions. Large internet companies and cloud providers face commercial barriers. Before adopting the model in production, legal and technical teams should assess their business model and revenue forecasts carefully to avoid potential infringement risks.
5. Benchmark Performance and Capability Limits
On benchmarks provided by the research team, the model shows top-tier competitiveness in some areas and clear weaknesses in others.
1. Strong in Coding and Agent Tasks
On a programming benchmark, the model scored 65.7%, slightly below the 66.4% of a leading closed-source model. On a long-horizon code engineering test, it scored 51.7%, which surpasses some top closed-source models. These results suggest clear strengths in complex logical reasoning, code generation, and engineering task execution. The model is a strong foundation for coding assistants and automated R
