Deploying a 2.8T MoE Model: Can 8 GPUs Run It? A Practical VRAM Calculation Guide
A 2.8T-parameter MoE model with approximately 1.56 TB of weights cannot fit on eight 80 GB GPUs. This guide explains the model’s publicly reported architecture, the impact of sparse attention and MXFP4 quantization, how to estimate inference memory, and the trade-offs between multi-node deployment and elastic cloud inference. It also includes download verification steps and a deployment checklist.
1. Bottom Line
The model’s weights occupy approximately 1.56 TB, which is far more than the 640 GB of total VRAM available on eight 80 GB GPUs. As a result, eight GPUs cannot load the complete model.
A local deployment requires multiple machines with multiple GPUs or GPUs with substantially more memory. For most teams, a more practical approach is to use elastic cloud inference or validate the application with a smaller-scale workflow before making a large infrastructure investment.
2. Publicly Reported Model Specifications
The original source reports the following specifications:
- Approximately 2.8 trillion total parameters
- Approximately 104 billion active parameters per inference step
- A Mixture-of-Experts (MoE) architecture
- 896 experts
- Each token is routed to 16 experts
- Two additional shared experts
- 93 Transformer layers
- A context window of up to 1 million tokens
- 96 Safetensors weight shards
These figures should be verified against the official model card, config.json, and the weight index file.
A practical verification process is:
- Open the model card on the official release page.
- Check the configuration file for the number of layers, number of experts, and context length.
- Use the weight index file to count the shards and calculate their total size.
- Confirm that the results match the published description.
3. Sparse Attention: Limited Public Information
The original source states that full attention is replaced with sparse attention in some layers, such as 24 layers. Sparse attention generally reduces the amount of computation required for long sequences by limiting the tokens each token can attend to. Common approaches include local attention windows, global tokens, or token sampling.
However, the currently available public information does not specify:
- The attention window size
- The number of global tokens
- The exact locations of the replaced layers
- Measured throughput
- Measured KV cache consumption
Therefore, this article does not estimate a specific speedup or VRAM reduction.
Anyone evaluating long-context performance should benchmark the model at the target context length and measure:
- Time to first token
- Token throughput
- Total VRAM usage
4. MXFP4 Quantization and Quantization-Aware Training
The original source reports that the model uses MXFP4 for weights and MXFP8 for activations. It also states that this version was produced using quantization-aware training, meaning that the training process simulated the effects of low-precision computation.
At present, only this quantized version has been released. No BF16 baseline is available, so the accuracy impact cannot be measured directly.
For mathematical reasoning, coding, long-form text, and other important workloads, teams should evaluate the model on their own benchmark sets and compare the results with official or independently reproduced evaluations before putting it into production.
The inference framework is another practical consideration. MXFP4 requires support from both the framework and the hardware. Without the required operators, inference may fall back to a slower execution path or fail entirely.
Before deployment, verify:
- Framework version
- CUDA version
- Supported hardware
- Operator support for MXFP4 and related formats
5. How to Estimate VRAM Requirements
Inference memory can be estimated using the following formula:
Required VRAM ≈ Weights + KV Cache + Activations + Communication Buffers + Framework Overhead
For this model:
- Weights: Approximately 1,560 GB. This uses decimal units, where 1.56 TB equals 1,560 GB.
- Eight 80 GB GPUs: 640 GB in total, leaving a shortfall of approximately 920 GB. The model cannot be loaded.
- Eight 192 GB GPUs: 1,536 GB in total, which is still less than the approximately 1,560 GB required for the weights alone. The model still cannot be loaded.
- Twenty 80 GB GPUs: 1,600 GB in total. This is theoretically enough to hold the weights, but only about 40 GB remains for the KV cache, activations, communication buffers, and framework overhead. In practice, this is not sufficient.
Twenty 80 GB GPUs should therefore be considered a theoretical lower bound, not a practical deployment target. Real deployments will generally require more memory.
KV cache usage increases with both concurrency and context length. Long-context inference can consume substantially more memory. The exact amount depends on the attention structure and cache format, so it must be measured in the target inference environment.
6. Three Deployment Options
Option A: A Single Server with Eight GPUs
A single 8-GPU server cannot load the complete model. It can still be used for API integration, workflow validation, and other early-stage testing.
Option B: Multi-Node, Multi-GPU Deployment
A self-hosted deployment would require at least three nodes, each with eight 80 GB GPUs, for a total of 1,920 GB of VRAM. This is enough to cover the weights while leaving some memory for cache and runtime overhead.
In practice, more capacity may be required. The deployment may use tensor parallelism, pipeline parallelism, or both, and it depends on a high-speed interconnect.
Cross-node communication latency directly affects throughput, so network bandwidth and latency are key infrastructure considerations.
Option C: Elastic Cloud Inference
Cloud inference allows teams to provision large-memory GPU instances on demand and pay according to usage. This is suitable for experimentation and workloads with significant demand fluctuations.
Total cost depends on:
- GPU model
- Number of instances
- Online operating time
- Storage charges
- Data transfer and bandwidth charges
Cost should be estimated from the actual workload rather than by looking only at the advertised hourly GPU price.
7. Download and Integrity Verification
Before downloading the model:
- Reserve at least 1.56 TB of disk space for the weights, plus additional capacity for conversion artifacts and caches.
- Use the official index file to verify the filenames and sizes of all 96 shards.
- Calculate the SHA-256 hash for each shard and compare it with the official hash. Re-download only the shards that fail verification.
- Confirm that the selected inference framework can read the sharded format directly before deciding whether the files need to be merged.
- Store the weights on local NVMe storage or a high-performance parallel file system. Network-attached storage can significantly slow model loading.
8. Cloud Infrastructure Selection
Baidu AI Cloud GPU compute instances and container platforms can be used for elastic scheduling, while object storage can support weight distribution and version management. This is only one possible option; other cloud providers may also meet the requirements.
When comparing cloud options, evaluate:
- VRAM per GPU
- Number of GPUs per instance
- Interconnect bandwidth between GPUs
- Instance pricing
- Storage throughput
- Support for the required container images and inference frameworks
Specific instance specifications and pricing are subject to change and should be checked on each provider’s official website.
9. Deployment Checklist
Before deployment, confirm the following:
- Does the application truly require a 1-million-token context window? If not, lowering the maximum context length can significantly reduce memory pressure.
- Is multimodal input required? If so, first verify the vision encoder’s input format and support in the selected framework.
- What level of quantization-related accuracy loss is acceptable? Use an internal evaluation set to establish the threshold.
- What is the cloud budget? Estimate monthly costs based on peak concurrency and average daily online time.
- Does the inference framework support MXFP4 and the required operators?
- Can the network and storage systems handle weight loading and cross-node communication?
Calculate the memory requirements first, then choose the deployment architecture.
