2.8T Open-Source MoE Model Deployment: Capacity and Inference Practices
This article examines the deployment considerations for a 2.8T-parameter open-weight MoE model, covering architecture, MXFP4/MXFP8 quantization, weight size and distribution, license terms, and hardware capacity. It explains why eight 80GB GPUs cannot hold the weights and what teams should verify before running inference.
2.8T Open-Source MoE Model Deployment: Capacity and Inference Practices
When a 2.8T-parameter Mixture-of-Experts (MoE) model is released with open weights, most teams first ask whether their existing machines can run it. This article covers the model architecture, quantization scheme, license terms, hardware capacity, weight distribution, and inference configuration. The numbers here fall into two categories: those that can be calculated directly, and those taken from the original release or public materials. Figures that have not been independently verified are noted.
1. Model Architecture Highlights
The model is a native multimodal agent model further trained from a previous-generation base. It includes a vision encoder and supports interleaved text and image input. The MoE structure separates total parameter count from per-token computation. Total parameters determine how much storage the weights occupy, while activated parameters determine the computation cost of each inference step. The original release states that about 104B parameters are activated per token and describes how routed experts and shared experts are combined. These settings should be checked against the configuration files in the official repository. Before deployment, verify the number of experts, the number of experts activated per token, and the number of layers, because they determine how tensor parallelism and expert parallelism are partitioned.
For long context, the original release states that the model supports about 1 million tokens and replaces full attention in 24 layers with sparse attention (DSA). Sparse attention computes attention only among selected positions, reducing computation for long sequences. Two points matter in practice. First, it mainly reduces computational complexity; whether KV cache is reduced at the same time depends on the implementation, so it should not be treated directly as a memory saving. Second, its impact on accuracy and its applicable boundaries need to be tested on your own long-text tasks.
2. Quantization Scheme and Weight Specifications
The official release provides a single version: MXFP4 weights and MXFP8 activations, produced by quantization-aware training (QAT). No BF16 or FP8 version is provided. MXFP4 is a micro-scaling floating-point format in which each element uses 4 bits and shares scaling factors. QAT lets the model simulate quantization error during training, so accuracy is usually better than post-training quantization.
This means there is no ready high-precision fallback. Dequantizing the model to BF16 would require about 2.8T × 2 bytes = 5.6TB for the parameters alone, far larger than the current weight footprint and basically infeasible at this scale. In addition, if an inference framework does not support MXFP4 and MXFP8 compute kernels, it cannot run this version. Before launch, compare quantized performance on your own evaluation sets—for example, one set each for code generation, tool calling, and long-context question answering—and save the results. Do not rely on a single metric.
3. Weight Size and Distribution
The original release lists the weight specification as 112 files, 96 safetensors shards, and a total size of about 1.56TB. These figures have not been independently verified here; they may change with release versions, so use the repository file list as the source of truth.
Based on 1.56TB, downloading and distributing the weights requires sufficient time. On a 1 Gbps dedicated line, the theoretical transfer time is: 1.56 × 10¹² bytes × 8 ÷ 10⁹ bit/s ≈ 12480 seconds, or about 3.5 hours. This is a theoretical value that excludes protocol overhead and retransmission; actual interruptions and retries will make it longer.
It is recommended to place the weights in object storage or NAS, mount them read-only in inference containers, and avoid copying a 1.56TB file to every node. If multiple nodes need access, first create an internal mirror, then let each node pull in parallel.
4. License Terms and Commercial Thresholds
The model uses a custom license. For ordinary developers and small and medium-sized enterprises, the terms are close to an MIT-style permissive license, allowing free use, copying, modification, distribution, and fine-tuning. Large-scale commercial use has two additional conditions, summarized here from the original release:
- Providing model-as-a-service, with cumulative revenue over the past 12 months exceeding $20 million: a separate agreement with the model provider is required before commercial use.
- A commercial product with more than 100 million monthly active users: additional authorization is required.
Use the official license text as the source of truth, and confirm the effective date and the counting scope for revenue and user numbers—for example, whether they are calculated on a consolidated group basis. Legal teams should evaluate the terms against growth expectations for the next year. If the business is close to the thresholds, start commercial discussions early.
5. Hardware Capacity Estimate
This section is central to deployment planning. The conclusion is: eight 80GB GPUs cannot hold the model weights.
The calculation is as follows:
- Weight size is about 1560GB (based on 1.56TB).
- Eight 80GB GPUs provide 8 × 80GB = 640GB of aggregate memory.
- 640GB is less than 1560GB, so the weights alone exceed available capacity.
