Trillion-Parameter MoE Models: Quantized Deployment and GPU Memory Planning
This article explains how to estimate and plan GPU memory for deploying trillion-parameter Mixture-of-Experts models. It breaks memory requirements into weights, KV cache, and runtime overhead; distinguishes total parameters from active parameters; discusses long-context sparse attention and low-bit quantization; provides a Python estimation example; and closes with a multi-GPU deployment checklist and practical recommendations.
1. Start with the Memory Budget: Deployment Begins with GPU Memory
When deploying very large models, the first question is usually not speed but whether the model fits. Model parameters, intermediate states during inference, and long-context caches all consume GPU memory. If the estimate is far off, later tuning has little chance of succeeding.
In engineering terms, the problem can be split into three parts:
- Weight storage: determined by the total number of parameters and the number of bits per parameter.
- Inference cache: determined by the number of layers, attention head configuration, context length, and concurrency.
- Runtime overhead: including activation peaks, communication buffers, and framework overhead.
After adding these three parts, reserve a certain percentage as headroom to get the minimum GPU count. The following sections explain the model design factors that affect these three parts.
2. Distinguish Total Parameters from Active Parameters
The defining characteristic of sparse Mixture-of-Experts (MoE) models is that the total parameter count is very large, while only a subset is used in each computation. The model contains many expert sub-networks, and a routing module selects a small number of experts for each input token, leaving the rest idle.
This produces two numbers:
- Total parameters: determines how much storage the weights require.
- Active parameters: determines the computation per token and affects per-step latency and throughput.
Consider a hypothetical model with hundreds of experts, where each token is routed to only a dozen or so of them. In this case, the total parameter count may be more than twenty times the active parameter count. The gap means storage demand should be calculated from total parameters, while compute cost should be calculated from active parameters.
MoE models also often include a small number of shared experts. Shared experts apply to all tokens and learn common foundational transformations, while routed experts handle more specialized tasks. Although shared experts account for only a small share of parameters, they can improve training stability and reduce output variation caused by uneven routing.
During deployment, keep the following points in mind:
- With many experts, expert parallelism introduces complex communication patterns, and cross-GPU all-to-all bandwidth can become a bottleneck.
- When routing is imbalanced, some GPUs are overloaded while others sit idle, reducing overall throughput.
- At small batch sizes, each expert receives very few tokens, which lowers matrix operation efficiency.
Therefore, an MoE deployment plan cannot rely only on the parameter table. It must be evaluated together with the parallelism strategy and batch size.
3. Compute Bottlenecks in Long Contexts and Sparse Attention
The compute cost of standard self-attention grows quadratically with sequence length. As context expands from tens of thousands of tokens to one million tokens, attention computation becomes a major cost, and GPU memory usage grows with it.
Sparse attention is a common response. Instead of letting every token attend to all previous tokens, it focuses only on a subset of positions, such as a local window, fixed-interval positions, or key positions dynamically selected by an indexing module. This can reduce attention computation from quadratic to near-linear or sub-quadratic.
In practical models, a common design is a hybrid structure: most layers use sparse attention, while a few layers retain full attention to preserve the ability to aggregate global information. This balance saves compute while maintaining long-range dependencies.
One point is easy to confuse: sparse attention mainly reduces attention computation, but it does not automatically reduce KV cache storage. As long as the key and value vectors for each historical token still need to be stored, cache memory usage depends on context length and layer count. Compressing the cache at the same time usually requires additional compression or selective retention mechanisms. When evaluating long-context deployment, these two concerns should be calculated separately.
4. Low-Bit Quantization: Separate Roles for Weights and Activations
Quantization is the most direct way to reduce deployment cost. It maps high-precision values to lower-bit representations, reducing storage and bandwidth requirements.
4.1 Microscaling Formats
A common microscaling format shares one scale factor across a group of elements, such as 32 elements. Each element is represented with 4 or 8 bits, and the shared scale factor adds a small number of extra bits. This approach preserves numeric dynamic range relatively well at low bit widths.
For example, with 4-bit elements, an 8-bit shared scale, and a block size of 32, the average storage per parameter is:
effective bits = 4 + 8 / 32 = 4.25 bit
4.2 Quantization-Aware Training
Post-training quantization (PTQ) converts precision directly after model training is complete. It is simple to apply, but accuracy loss can be noticeable at low bit widths. Quantization-aware training (QAT) simulates quantization error during training so that model parameters adapt to low-precision representations. QAT has higher training cost, but in low-bit scenarios it usually preserves more quality.
4.3 Precision Combinations for Weights and Activations
During inference, weights and activations can use different precisions. Weights are usually static and can be stored in more aggressive low-bit formats. Activations are generated dynamically and are more sensitive to numeric range, so they often use higher bit widths. This combination balances memory savings and numerical stability.
During deployment, you must confirm whether the inference framework supports the chosen format combination. If the framework must first dequantize weights to higher precision, the memory savings are offset, and actual usage may exceed expectations.
5. GPU Memory Estimation Method
The following example combines the factors above into an executable estimation process. The numbers are hypothetical and are used only to illustrate the calculation method; they do not represent any specific model.
import mathdef weight_bytes(total_params: float, weight_bits: int = 4,block: int = 32, scale_bits: int = 8) -> float:"""Bytes for weights, including shared scale factor overhead per block."""effective_bits = weight_bits + scale_bits / blockreturn total_params * effective_bits / 8def kv_cache_bytes(layers: int, kv_heads: int, head_dim: int,context_len: int, elem_bytes: int = 1,batch: int = 1) -> float:"""Bytes for KV cache; keys and values are stored separately, hence factor 2."""return 2 * layers * kv_heads * head_dim * elem_bytes * context_len * batchdef cards_needed(total_bytes: float, card_gb: float,usable_ratio: float = 0.9) -> int:"""Number of GPUs needed based on usable memory per GPU."""per_card = card_gb * 1e9 * usable_ratioreturn math.ceil(total_bytes / per_card)# Assumptions: 2T total parameters, 4-bit weights; 60 layers, 8 KV heads,# head dimension 128, FP8 cache, single 1M-token context, 192GB per GPU.w = weight_bytes(2e12)kv = kv_cache_bytes(layers=60, kv_heads=8, head_dim=128,context_len=1_000_000, elem_bytes=1)total = (w + kv) * 1.15 # Reserve 15% for activation peaks and runtime overheadprint(f"Weights: {w/1e12:.2f} TB")print(f"KV cache (single sequence): {kv/1e12:.3f} TB")print(f"Estimated total requirement: {total/1e12:.2f} TB")print(f"GPUs needed: {cards_needed(total, 192)}")
The output is:
- Weights: about 1.06 TB
- KV cache for a single 1M-token sequence: about 0.12 TB
- Estimated total requirement: about 1.36 TB
- At 192GB per GPU with 90% usable memory, 8 GPUs are required
This result is only an estimate. Actual deployment must also account for the number of concurrent sequences, batch size, whether prefix caching is enabled, and framework memory reservations. As concurrency increases, KV cache grows proportionally and can quickly exceed the footprint of the weights themselves.
6. Multi-GPU Deployment Checklist
After estimating the GPU count, confirm each item in the checklist below.
Hardware
- Whether each GPU has enough memory for its weight shard plus cache after partitioning.
- Whether inter-GPU interconnect bandwidth can support the communication volume of expert parallelism or tensor parallelism.
- Whether the intra-server topology is balanced enough to avoid cross-node communication becoming a bottleneck.
Parallelism Strategy
- Tensor parallelism is suitable for reducing per-GPU weight memory, but it involves frequent communication.
- Expert parallelism suits MoE structures, but routing imbalance must be handled.
- Pipeline parallelism can split layers, but it introduces bubble overhead and is unfriendly to latency-sensitive scenarios.
Inference Configuration
- Whether the maximum context length matches actual business requirements. Longer contexts increase prefilling latency.
- Whether the batching strategy aligns with concurrency goals.
- Whether the quantization format is natively supported by the inference framework and whether implicit dequantization occurs.
Validation Method
- Test peak GPU memory with requests at real lengths, not only with short text.
- Record prefilling latency and per-token generation latency, and evaluate them separately.
- Run regression tests at the target precision to confirm that output quality after quantization meets business requirements.
7. Practical Recommendations
For deploying very large sparse models, the core task is converting parameter scale into hardware requirements. This conversion depends on three calculations: weight storage is determined by total parameters and bit width, cache is determined by context and layer structure, and compute latency is determined by active parameters and attention design.
Proceed in the following order:
- Use formulas to estimate weights and cache first, and determine the lower bound for GPU count.
- Reserve at least 10% to 20% headroom, and measure peak usage with real workloads.
- Choose parallelism strategies and batching parameters based on business context length and concurrency requirements.
- Complete quality regression after quantization before gradually opening traffic.
For teams, the most easily overlooked issue is cache growth. Model weights are fixed, but cache changes continuously with user requests. Designing cache limits, timeout-based reclamation, and request admission control in advance is more effective for stabilizing service quality than simply adding more GPUs.
