Understanding VRAM Costs for Deploying Trillion-Parameter MoE Models Locally: VRAM and Hardware Thresholds

When a model’s parameter scale reaches the trillion level, local deployment is no longer just a matter of “buying a few GPUs.” It becomes a complex engineering challenge involving storage, bandwidth, compute power, and system architecture. Using a 2.8-trillion-parameter MoE architecture model as an example, this article breaks down the actual VRAM costs and hardware thresholds for local deployment, providing a reusable estimation methodology.

1. Clarifying a Common Misconception: Activation Ratio ≠ VRAM Ratio

When discussing MoE model deployment, many instinctively estimate VRAM usage based on “activated parameters.” This is the most common pitfall.

Take a 2.8-trillion-parameter MoE model as an example: it contains 896 experts, with 16 activated per token, meaning the expert activation ratio is about 1.79%. This small number often leads to the misconception that VRAM requirements can be proportionally compressed. However, the reality is:

• Activation ratio only affects compute consumption: During inference, only the routed experts participate in calculation. FLOPs are calculated based on the activated portion.
• VRAM usage must be calculated based on total parameters: All 896 experts must reside in memory. During inference, the corresponding 16 experts are dynamically called out based on routing.

In other words, the MoE architecture saves computation, not storage. The lower limit of VRAM during deployment is determined by the total parameter scale, not the activated parameter scale.

The model’s backbone architecture employs a linear attention variant with attention residual connections, featuring components like gated multi-head latent attention. These structural designs affect computational efficiency and long-context performance, but they do not change the fundamental constraint that “all experts must be loaded into memory.”

2. How Quantization Formats Affect Actual VRAM Usage

Assume the model introduces quantization-aware training (QAT) from the supervised fine-tuning stage, using the 4-bit MXFP4 format for weights and the 8-bit MXFP8 format for activations. What is the raw data lower limit for 2.8 trillion parameters converted to 4-bit?

Let’s do a basic estimation:

2.8T × 4 bit ÷ 8 = 1400 GB

However, MXFP4 is not pure 4-bit storage. According to the OCP specification, every 32 4-bit elements must be accompanied by an 8-bit scale factor. Therefore, the average bit width per weight element is actually:

4 + 8/32 = 4.25 bit

Recalculating:

2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

This 1487.5 GB only accounts for the weights and does not yet include:

• Non-4-bit tensors (such as layers maintaining higher precision)
• Metadata and routing tables
• Padding overhead from memory alignment
• Runtime workspace
• KV cache

Factoring in these elements, the actual VRAM requirement will be significantly higher than the theoretical lower limit.

3. Cross-Reference: An Estimation Anchor

To better understand this scale, we can use a medium-sized MoE model as a reference.

A certain 671B-parameter MoE model in FP8 format requires about 685 GB for main parameters plus roughly 14B parameters for multi-token prediction. Even with a 4-bit weight quantization scheme, the weights alone take about 335 GB. Adding roughly 37 GB for activations totals about 400 GB, which requires 8 accelerator cards with 80 GB of VRAM each (640 GB total) to run.

The 2.8-trillion-parameter model is over four times the size of the former. Proportionally, even with the same aggressive 4-bit quantization, the weight lower limit approaches 1.5 TB, and the actual deployment VRAM requirement will be even higher.

This leads to a clear conclusion: The lower limit for a full-precision deployment far exceeds the capacity of a single 8-GPU machine.

4. Official Hardware Recommendation: Supernodes

Faced with a model of this scale, the officially recommended deployment approach is to use a supernode composed of 64 or more accelerators.

The logic behind this recommendation is straightforward:

1. Total VRAM must be sufficient: Based on previous estimates, weights plus runtime overhead require TB-level memory, necessitating dozens of high-VRAM accelerator cards.
2. Interconnect bandwidth must be sufficient: During MoE inference, token routing causes frequent communication between experts. If inter-GPU bandwidth is insufficient, communication becomes a bottleneck, and compute power cannot be fully utilized.
3. Supernode architecture solves communication issues: High-bandwidth interconnects pool multiple accelerator cards into a logical, large memory pool, keeping the communication overhead of expert parallelism and expert routing manageable.

In other words, 64 cards is not a “recommended configuration,” but rather the “minimum to run.” Below this scale, either the VRAM cannot hold the model, or communication bottlenecks will kill throughput.

5. Local Deployment Cost Estimation Framework

If you need to budget for the local deployment of a trillion-level MoE model, you can estimate layer by layer using the following framework:

Step 1: Determine the weight VRAM lower limit

Weight VRAM = Total Parameters × Actual Bit Width per Element ÷ 8

Note the additional overhead of scale and zero-point factors in quantization formats; do not calculate using only the raw bit width.

Step 2: Add non-weight overhead

• KV cache: Related to context length, concurrency, number of layers, and attention head configuration.
• Runtime workspace: Usually a certain percentage of the weights.
• Metadata and alignment padding: Depends on the specific implementation.

Step 3: Determine the number of accelerator cards

Required Cards = Total VRAM Demand ÷ Usable VRAM per Card

Note that usable VRAM per card is not equal to the nominal VRAM; system reservations must be deducted.

Step 4: Evaluate interconnect topology

• Whether inter-card bandwidth meets the communication demands of expert routing.
• Whether supernode-level interconnects (e.g., intra-high-bandwidth domain interconnects) are required.
• Whether the network topology supports full-mesh or near-full-mesh communication.

Step 5: Calculate Total Cost of Ownership (TCO)

• Hardware procurement or rental costs.
• Power and cooling costs.
• Operations and maintenance (O&M) and labor costs.
• For cloud deployments, consider the billing differences between on-demand and reserved instances.

6. Often Overlooked Engineering Details

Beyond VRAM and compute power, the following aspects are equally critical in actual deployment:

• Expert routing load balancing: Among the 896 experts, poor routing strategies can lead to some experts being overloaded while others are idle, causing actual throughput to fall far below theoretical values.
• KV cache management: In long-context scenarios, the KV cache can occupy tens of GB or more of VRAM, requiring optimization techniques like paged attention.
• Balancing quantization precision and performance: While 4-bit quantization reduces VRAM, it may introduce precision loss. Thorough evaluation is necessary before deployment.
• Cold start and loading time: Loading TB-scale weights from storage to VRAM can take tens of minutes, requiring service pre-heating strategies.

7. Summary

For the local deployment of trillion-parameter MoE models, the core constraint is not compute power, but VRAM capacity and interconnect bandwidth. The expert activation ratio only affects computation, not storage requirements. The overhead of quantization formats makes actual VRAM usage higher than raw data estimates. Supernode-level hardware configuration is a fundamental prerequisite for running such models.

When planning a deployment, it is recommended to first estimate the weight lower limit using “total parameters × actual bit width,” then add KV cache and runtime overhead, and finally determine the number of cards and topology based on the usable VRAM per card and interconnect capabilities. This methodology is independent of specific vendors or products and can be reused for deployment estimation of any large-scale MoE model.