Estimating Minimum VRAM and Hardware Requirements for Local Deployment of Trillion-Parameter MoE Models
Deploying trillion-parameter Mixture of Experts (MoE) models locally requires understanding the critical difference between activated and resident parameters. This article estimates the minimum VRAM needed for a 2.8-trillion-parameter model using 4-bit quantization, factoring in raw weights, non-quantized tensors, runtime workspace, and KV Cache. It outlines hardware strategies including pure GPU clusters, CPU-GPU hybrid setups, and NVMe tiering, concluding that a full local deployment demands over twenty 80GB GPUs. For most developers, downgrading to smaller architecture-equivalent models or renting cloud GPU instances is the most practical path for testing and deployment.
1. A Key Concept: Activated Parameters vs. Resident Parameters in MoE
When developers first see specifications like “2.8 trillion parameters with only 16 experts activated per token,” they might intuitively assume that VRAM requirements depend solely on the activated parameters. While this holds true computationally, it severely underestimates storage requirements during deployment.
Mixture of Experts (MoE) models typically consist of a shared backbone and a large number of expert feed-forward networks. For instance, a model with 2.8 trillion total parameters might contain hundreds or even nearly a thousand expert modules. During inference, a routing mechanism dynamically selects a few experts based on the current token’s representation. Unselected experts do not consume compute power during the forward pass.
However, “not participating in computation” does not mean “not requiring storage.” When an inference service starts, all expert weights must be loaded into high-speed accessible storage. Because routing decisions change dynamically at runtime, it is impossible to predict which experts the next token will hit. If a required expert’s weights are not already in memory or VRAM, a page-fault-like loading process occurs, leading to uncontrollable latency.
Therefore, the first principle of MoE model deployment is:
VRAM capacity must be estimated based on total parameters, while the activation ratio only dictates compute and bandwidth pressure.
This means that even if the activation ratio is just over 1%, a 2.8-trillion-parameter model still requires storage space for all weights.
2. How Large are Raw Weights After 4-bit Quantization?
For a model of this scale, loading it locally in full FP16 or BF16 precision is practically impossible. Storing 2.8 trillion parameters in 16-bit format requires about 5.6 TB just for weights. Even in 8-bit format, it requires roughly 2.8 TB. This volume exceeds the physical memory limits of the vast majority of local servers, let alone VRAM.
Therefore, 4-bit quantization is almost inevitable for local deployment. Among common 4-bit weight formats, MXFP4 is a micro-scaled format with shared scales. It is not simply a standard 4-bit integer; instead, every 32 4-bit elements are accompanied by an 8-bit scale factor.
Based on this specification, the effective storage per parameter is approximately:
4 bit + 8 bit / 32 = 4.25 bit
For 2.8 trillion parameters:
2.8 × 10^12 × 4.25 bit ÷ 8 bit/Byte ≈ 1487.5 GB
So, even with 4-bit weights, the raw weight data alone approaches 1.5 TB. This is the theoretical lower limit, and actual deployment requirements will expand further.
For quick calculations, refer to the conversion table below:
| Precision Format | Effective Bits per Parameter | Weight Size for 2.8T Parameters |
|---|---|---|
| BF16/FP16 | 16 bit | ~5.6 TB |
| INT8/FP8 | 8 bit | ~2.8 TB |
| Standard INT4 | 4 bit | ~1.4 TB |
| MXFP4 (with scale) | ~4.25 bit | ~1.49 TB |
Note that this only accounts for the main weights. The total storage footprint of a complete model includes much more.
3. Why “Raw Weights” Aren’t the Final Answer
Many VRAM estimation tutorials stop at raw weights, but during actual deployment, the following components also consume significant space.
3.1 Non-4-bit Tensors
Not all tensors are suitable for 4-bit quantization. Normalization layer parameters, embedding layers, projection matrices for certain attention modules, routing networks, and the final output layer are often kept in FP16, BF16, or FP8 precision. Although these modules represent a small fraction of the total parameters, their absolute size is substantial.
Take the embedding layer as an example. If the vocabulary size is in the 150,000 to 200,000 range and the hidden dimension is in the thousands, the input embedding matrix alone in FP16 could occupy several GB to tens of GB. If the output layer is not decoupled from the vocabulary, the output projection’s parameters are equally significant.
3.2 Runtime Workspace
Inference is not a static weight-reading process. During the forward pass, space must be reserved for intermediate activations, temporary tensors, and computation-essential buffers (unrelated to gradients). Even with activation checkpointing strategies, the runtime workspace on ultra-large models can still reach tens of GB.
For local deployments, if pipeline parallelism is used, communication buffers between stages must be considered. If tensor parallelism is used, additional temporary space for reductions is required.
3.3 KV Cache
During autoregressive generation, the key-value pairs for each token must be cached. The longer the context and the higher the concurrency, the faster the KV Cache grows. For long-context models, the KV Cache can even exceed the weight size. For example, when the context length reaches tens of thousands of tokens with high request concurrency, the KV Cache consuming hundreds of GB is not an exaggeration.
3.4 Alignment and Metadata
To meet specific operator or hardware alignment requirements, weight matrices usually need to be padded to fixed lengths, resulting in extra space. Additionally, quantization formats require saving scales, zero-points, routing tables, and configuration metadata.
Overall, for a 2.8-trillion-parameter MoE model, even with 4-bit weights, the actual total storage requirement is typically significantly higher than 1.5 TB. Conservatively, you should reserve 1.2 to 1.5 times the raw weight data.
4. Estimating from Existing Large Model Deployment Data
To make the estimation more practical, consider a reference case of a similar scale: a certain 671B-parameter MoE model. Its official deployment configuration uses FP8 precision, with main parameters around 671B and an auxiliary prediction module around 14B, totaling 685B. In FP8, the weights alone require about 685 GB.
If compressed to 4-bit, the main weights would be about 335 GB, but activations, workspace, and KV Cache require an additional 37 GB or more. Ultimately, when running on 8 x 80GB VRAM cards (640 GB total VRAM), it still relies on CPU offloading to meet demands.
Based on this, a 2.8-trillion-parameter model is over 4 times the size of this reference model. Even with the same 4-bit quantization, the raw weight data reaches about 1.5 TB, and the overall storage demand could approach or exceed 2 TB. This is beyond the capacity of a single 8-GPU machine.
If you want to perform inference entirely within GPU VRAM, assuming 80 GB per card, you would need at least:
2000 GB ÷ 80 GB/card ≈ 25 cards
Even calculating based on 1.5 TB raw weights, it requires at least 19 cards, not yet accounting for workspace and KV Cache.
5. Hardware Profiles for Local Deployment
5.1 Pure GPU Solution
Using 80 GB cards as an example, covering the estimated 1.5 TB to 2 TB actual requirement demands at least 20 to 25 compute cards. This scale is no longer a typical “local workstation” but rather a small compute cluster.
Crucially, inter-card interconnect bandwidth directly impacts token generation speed. All-to-All communication across multiple cards is one of the key bottlenecks in MoE inference. If interconnected only via PCIe or standard Ethernet, the communication overhead from expert routing will severely degrade generation speed.
5.2 CPU + GPU Hybrid Solution
A compromise is to use heterogeneous storage: place hot data (backbone weights and frequently used experts) in GPU VRAM, and cold data (low-frequency expert weights) in CPU memory, scheduling on demand via unified memory addressing.
This approach significantly reduces the GPU card requirement. For instance, using 8 to 12 x 80GB VRAM cards paired with over 1.5 TB of system memory could theoretically work. However, the tradeoff is that if a token is routed to an expert not resident in VRAM, weights must be transferred from memory to the GPU. Latency will then jump from milliseconds to sub-second or higher.
This solution is better suited for offline batch inference or latency-insensitive scenarios, not for production services requiring real-time interaction.
5.3 Limitations of High-Speed NVMe Tiering
Theoretically, weights could be stored in an NVMe array and read directly by the GPU. However, in autoregressive generation, each token activates only a few experts, making the read pattern highly random. While NVMe sequential bandwidth is impressive, random read performance drops significantly, and the latency from kernel page caches and driver stacks is much higher than memory channels. Therefore, pure NVMe tiering is largely unfeasible for real-time inference.
6. Downgrading Strategies for Technical Verification
Most developers do not need to run a full 2.8-trillion-parameter model, but rather wish to verify the architecture, quantization scheme, or inference pipeline locally. In such cases, the following downgrade strategies can be applied.
6.1 Use Smaller Models with the Same Architecture
Many MoE architectures release versions of different scales. Choosing a model one to two orders of magnitude smaller for deployment verification can validate key components like the quantization pipeline, routing mechanism, and communication strategy without incurring massive cluster costs.
6.2 Load Only Partial Experts for Single-Step Forward Testing
If the goal is to verify operator correctness, you can artificially fix the routing results and load only the activated expert weights. This way, even if the total parameters are in the trillions, the resident VRAM needed for a single forward pass can be controlled to a few hundred GB. However, this is only for functional verification and does not represent a real generation environment.
6.3 Use Cloud Multi-GPU Instances
If local hardware is insufficient, cloud GPU instances are a more realistic choice. Compared to building a small local GPU cluster, cloud services allow renting high-VRAM instances by the hour, significantly reducing verification costs and operational overhead through object storage, image management, and monitoring. For models requiring dozens of high-VRAM cards to load fully, short-term cloud rentals are virtually the only economically viable option.
7. Cost Estimation: Local Build vs. Cloud Rental
Regarding hardware costs, if calculating based on 25 x 80GB VRAM cards, the cost of the compute cards alone could reach hundreds of thousands of dollars. Factoring in servers, high-speed networking, power, and cooling, the total investment easily exceeds the experimental budgets of ordinary enterprises.
Cloud solutions have a completely different cost structure. Renting high-VRAM instances on demand typically costs a few dollars per card per hour, depending on the region, specifications, and billing strategy. If used only for verification testing, the rental cost for a few hours is usually far lower than building a local cluster.
Note that cloud deployments should also consider:
- Weight upload and storage costs;
- Network performance and bandwidth fees for multi-card instances;
- Whether to retain the instance after inference;
- The need for object storage to save model files;
- The need for monitoring, alerting, and logging services for troubleshooting.
8. Conclusion
The core challenge of locally deploying a 2.8-trillion-parameter MoE model is not “can it compute,” but rather “can it fit.”
With 4-bit micro-scaled formats like MXFP4, raw weights approach 1.5 TB. Adding non-quantized tensors, KV Cache, runtime workspace, and alignment overhead, actual storage demand easily reaches 2 TB or more. This means it requires at least twenty-plus 80GB VRAM cards or a complex CPU-GPU hybrid scheduling scheme.
For the vast majority of developers and enterprises, the reasonable path is:
- Perform local verification using a smaller model with the same architecture;
- Conduct functional testing via fixed routing or partial loading;
- When full deployment is necessary, prioritize cloud multi-GPU instances, renting on demand to control hourly costs.
The sparse activation design of MoE lowers the compute barrier but does not lower the storage barrier. Understanding this is the first step in deployment planning.
