How to Estimate VRAM and Costs for Local Deployment of Trillion-Parameter MoE Models
Estimating VRAM and costs for local deployment of trillion-parameter MoE models requires distinguishing between total and activated parameters. This article details how to calculate the VRAM lower limit with 4-bit quantization, calibrate model size using a reference point, understand multi-GPU communication bottlenecks, and provides a reusable cost estimation framework.
1. Distinguish Two Key Metrics: Total Parameters Dictate VRAM, Activated Parameters Dictate Compute
Before deploying a trillion-parameter MoE model, it is crucial to distinguish between two numbers. The first is the total parameter count, which determines how much VRAM the model weights will occupy. The second is the actual number of activated parameters per token, which dictates the theoretical compute required per step.
Take a typical model as an example: it has a total of about 2.8 trillion parameters, includes 896 expert networks, and routes each token to 16 of these experts, resulting in an expert activation ratio of about 1.79% (16 ÷ 896). There are two points to note here:
- The expert activation ratio does not equal the parameter activation ratio. Shared components such as attention, normalization, and embedding layers participate in computation during every inference step, so the actual ratio of activated parameters is higher than the expert activation ratio.
- The router may select any expert at any step, so the weights of all 896 experts must reside in VRAM or be loaded quickly. VRAM planning must be based on the total parameter count.
Many teams see an “activation ratio of less than 2%” and estimate that a few dozen GB of VRAM is sufficient to run the model. This is a misunderstanding of the MoE inference mechanism.
Additionally, activated parameters only indicate the theoretical compute requirement. Actual throughput is also limited by memory bandwidth, all-to-all communication, batch size, and parallelism efficiency. In many scenarios, the bottleneck lies in memory access and communication rather than the multiply-accumulate operations themselves.
2. How to Calculate the VRAM Lower Limit After 4-bit Quantization
Large models often introduce quantization-aware training during the training phase, such as 4-bit weights paired with 8-bit activations. Using 2.8 trillion parameters as an example, the estimation involves three steps:
- Base weight lower limit: 2.8T × 4 bit ÷ 8 = 1400 GB (calculated in decimal, where 1 GB = 10^9 bytes).
- Quantization scaling factor overhead: Assuming a group size of 32 with an 8-bit scaling factor per group, the overhead per element is 8 ÷ 32 = 0.25 bit, making the equivalent bit-width 4.25 bits. The weight footprint is approximately 2.8T × 4.25 ÷ 8 ≈ 1487.5 GB. If the format also includes zero-points or uses a different group size, recalculation based on the actual format is required.
- Unaccounted portions: Unquantized tensors (such as embedding layers and output heads, which some frameworks keep in high precision), metadata and alignment padding, runtime workspace, and KV Cache.
The KV Cache footprint can be estimated using the following formula:
KV Cache bytes = 2 × number of layers × number of KV heads × head dimension × sequence length × concurrency × bytes per element
The coefficient 2 corresponds to the two copies of caches for K and V. Assuming a sample model with 60 layers, 8 KV heads (Grouped-Query Attention), a head dimension of 128, a 16-bit KV Cache (2 bytes), a single request context of 128K tokens (131072), and a concurrency of 16:
- Per token per layer footprint: 2 × 8 × 128 × 2 = 4096 bytes
- Per token across all layers: 4096 × 60 = 245760 bytes
- Single 128K request: 245760 × 131072 ≈ 32.2 GB
- 16 concurrent requests: approximately 515 GB
If the model does not use Grouped-Query Attention, the number of KV heads equals the number of attention heads, doubling the footprint.
In engineering planning, VRAM requirements are the sum of “weights + KV Cache + workspace + overhead,” with the overhead typically set to 20% to 30%. The weight portion should be calculated at the 1.5 TB scale, while the KV Cache and workspace should be determined based on actual business testing.
3. Calibrating Size Using a Hypothetical Reference Model
In the absence of official benchmarks, existing deployment data can be used for analogy. The figures below represent a hypothetical scenario, not an officially published benchmark.
Assume a 671B parameter MoE model. At 8-bit precision, the main and auxiliary parameters take about 685 GB; after 4-bit quantization, the weights take about 335 GB; adding activation footprint, the total is about 400 GB. Using 80 GB-class accelerators, 8 cards are required (640 GB in total).
Extrapolating linearly by parameter count, a model roughly 4 times that size would have a weight lower limit of about 1.34 TB at the same quantization precision. After adding KV Cache and overhead, the total approaches the 1.5 TB scale. A single node with 8 cards (640 GB) cannot handle this; local deployment requires multi-node, multi-GPU interconnection.
The VRAM lower limit is just the starting point. The number of cards also depends on parallelism strategy, interconnect bandwidth, batch size, and Service Level Objectives (SLOs). In scenarios demanding high throughput or low latency, a common practice is to build 64-GPU-class super-nodes, but this is merely an empirical starting point, not a universal threshold. A Proof of Concept (POC) should be conducted before launch to measure actual throughput and tail latency.
4. The Bottleneck in Multi-GPU Deployment Lies in Communication
Large-scale MoE inference is highly sensitive to all-to-all communication bandwidth. The processing flow for each token is roughly as follows:
- The router selects top-k target experts for the token (in this example, k = 16).
- The hidden state of the token is sent to the cards hosting the target experts.
- Once the experts complete the computation, the results are transmitted back to the original card for subsequent computation.
Each token typically undergoes two rounds of cross-GPU transmission: dispatch and merge. The communication volume is roughly proportional to the number of tokens, top-k, hidden dimension, and the proportion of cross-GPU access. When the number of cards increases, the proportion of remote experts rises, increasing the per-GPU communication volume, but the total volume still grows approximately linearly rather than exponentially.
The real risks come from two areas. First, load imbalance: cards hosting popular experts become congestion points, leading to increased tail latency. Second, insufficient interconnect bandwidth: if the system relies solely on standard Ethernet, communication time may exceed computation time, causing throughput to plateau or degrade as more cards are added.
Therefore, the importance of interconnect topology and bandwidth in hardware selection is no less than that of per-GPU VRAM capacity. Before purchasing, the following metrics should be confirmed:
- High-speed direct connection bandwidth within a single node, as well as inter-node bandwidth;
- Actual measured bandwidth and latency of all-to-all communication under expert parallelism;
- The ability of load balancing strategies to handle popular experts;
- Actual throughput and time-to-first-token at the target batch size and context length.
5. Cost Estimation Framework
Before weights and benchmark tests are made public, all figures are mere estimates. The table below provides a reusable framework; replace the parameters with your own data when using it in practice.
| Item | Estimation Formula | Key Assumptions | Major Risks |
|---|---|---|---|
| Weight VRAM | Total parameters × quantization bit-width ÷ 8 | Group size, scaling factor bit-width, high-precision retained layers | Deviations caused by different quantization formats |
| KV Cache | Calculated using the formula in Section 2 | Number of layers, number of KV heads, context length, concurrency | Underestimation in long-context scenarios |
| Total VRAM | (Weights + KV Cache + Workspace) × (1 + overhead) | Overhead is 20% to 30% | Insufficient overhead leading to VRAM overflow |
| Number of Accelerators | Total VRAM ÷ per-GPU VRAM, rounded up to node granularity | Per-GPU capacity, cards per node, parallelism strategy | Topology constraints increasing the actual number of cards |
| Interconnect & Network | Calculated based on node granularity | Topology requirements, parallelism strategy | Insufficient bandwidth limiting actual throughput |
