MoE Model Deployment: How to Calculate GPU Memory Correctly and Avoid Common Pitfalls
This article explains how to estimate GPU memory requirements for deploying an ultra-large sparse MoE model, using a publicly described model (~2.8T total parameters, ~104B activated) as a case study. It covers the difference between total and activated parameters, KV cache estimation for long contexts, the real memory impact of sparse attention, an 8-GPU feasibility analysis based on 1.56TB of weights, parallelism strategies, and a pre-launch engineering checklist.
Step One: Three Calculations Before Deployment
The first step in evaluating whether an ultra-large model can run locally is to calculate three things: how much memory the weights occupy, how much computation each token requires during inference, and how large the cache grows with long contexts. Many deployment failures happen because teams only look at parameter count without distinguishing between “how much memory it takes to store” and “how much computation one forward pass requires.”
This article uses a publicly described ultra-large sparse MoE model as a case study. We will walk through whether 8 GPUs can hold it, how to estimate the KV cache, and whether sparse attention actually saves memory. Key facts about the model:
- ~2.8T total parameters, with only ~104B activated per forward pass;
- MoE architecture with 896 routed experts; each token is routed to 16 experts, plus 2 shared experts;
- 93 layers, supporting a context of roughly 1 million tokens;
- Includes a vision encoder for image inputs;
- Compared to its predecessor, 24 layers use sparse attention instead of full attention; the remaining 69 layers still use full attention.
These numbers come from public descriptions. Always verify against the model’s official configuration file before actual deployment.
Total vs. Activated Parameters: The Core Difference of MoE
In a dense model, every parameter participates in every computation, so parameter count and compute scale roughly proportionally. An MoE model splits its feed-forward layers into many experts, and a routing network selects only a few experts to compute for each token.
Working through the numbers in this case:
- The activation ratio is about 104B / 2800B ≈ 3.7%, meaning per-token compute is close to that of a 10B-scale dense model;
- The selection ratio of routed experts per layer is 16 / 896 ≈ 1.8%. Different tokens select different experts, so all expert weights must stay resident in GPU memory—otherwise, incoming requests would trigger constant weight swapping and latency would spike.
This determines the deployment logic: compute depends on activated parameters, memory depends on total parameters. Latency and throughput are mainly driven by activated parameters; whether the model fits and how much concurrency it can handle are determined by total parameters.
Shared experts execute for every token. This case has 2 shared experts, and every token passes through their feed-forward computation at each layer before being merged with the routed expert outputs. When estimating per-token latency, this computation must be included—you cannot count only the 16 routed experts. Merging the results also requires an extra data aggregation step, which adds synchronization overhead in multi-GPU setups.
Long Contexts and Sparse Attention
Standard self-attention compute grows quadratically with sequence length. When context extends from tens of thousands of tokens to the million-token range, attention becomes the main bottleneck. During inference, the key-value cache (KV cache) of each layer must also be stored, and its size grows linearly with context length.
A rough formula for cache size:
KV cache bytes ≈ 2 × attention layers × KV heads × head dimension × sequence length × bytes per element
The “2” accounts for both K and V. This formula applies to the layers that actually store the cache. In this case, 24 layers use sparse attention and 69 use full attention; each type must be calculated according to its own caching strategy. Public descriptions do not provide the KV head count or head dimension, so these must be read from the official configuration.
The core idea of sparse attention: each query position attends to only a subset of key-value positions instead of the full history. Selection methods typically include local windows, content-based selection of important positions, or a combination of both. What it mainly reduces is the number of keys and values involved in computation—i.e., compute. Sparse attention layers usually still need to store the KV cache; the cache does not drop to zero. How much cache is actually saved depends on each layer’s caching strategy and must be verified layer by layer.
The cost of sparse attention is that the model may miss certain long-range dependencies. A common practice is to keep some full-attention layers or preserve global connections at key positions. This case retains 69 full-attention layers, which is the foundation of its long-text capability.
Three engineering criteria to apply:
- Confirm the attention type of each layer—distinguish full-attention from sparse-attention layers and estimate cache and compute separately;
- Evaluate with real long documents, focusing on long-range retrieval and cross-paragraph reasoning rather than average perplexity alone;
- Set a reasonable maximum context length. Even if the model supports a million tokens, production services should enforce a cap based on business needs, so a single long request cannot exhaust the cache.
GPU Memory Estimation: Working Backward from 1.56TB to an 8-GPU Plan
Weight memory is determined by parameter count and storage precision. The following script gives weight sizes under three common precisions:
def weight_memory_gb(total_params_b, bits_per_param):# total_params_b: parameter count in billionsbytes_total = total_params_b * 1e9 * bits_per_param / 8return bytes_total / 1e9for name, bits in [("BF16", 16), ("FP8", 8), ("INT4", 4)]:print(name, round(weight_memory_gb(2800, bits)), "GB")
Output: BF16 ≈ 5600 GB, FP8 ≈ 2800 GB, INT4 ≈ 1400 GB.
The publicly stated weight size for this case is about 1.56TB. Two things need confirmation: first, whether the unit is decimal TB or TiB; second, whether it includes the vision encoder. In decimal terms, 1.56TB equals 1560GB. If it is actually 1.56TiB, that is about 1715GB—roughly 10% more memory required.
Dividing 1560GB by 2800B parameters gives about 4.5 bits per parameter on average, well below BF16 and FP8. This means the weights are not uniformly 4-bit quantized but use mixed precision—for example, low-bit quantization for expert weights with higher precision retained for attention or embedding layers. When reading the model configuration, check precision module by module.
Per-GPU memory calculation for an 8-GPU plan:
- 1560GB ÷ 8 = 195GB, so each GPU must hold at least 195GB of weights;
- Eight 192GB GPUs total 1536GB—already less than 1560GB, so they cannot even hold the weights. An 8-GPU plan with 192GB cards is not viable;
- With a 20% margin, each GPU needs about 195GB × 1.2 ≈ 234GB;
- If only 192GB cards are available, you need more GPUs—for example, sixteen 192GB GPUs total 3072GB—or offload part of the model to CPU or disk. Offloading significantly reduces throughput and is only suitable for offline or low-concurrency scenarios.
20% is only a rule-of-thumb lower bound. Beyond weights, there are the KV cache, activations, communication buffers, and the inference framework’s own overhead—all of which vary with concurrency and context length and cannot be captured by a single fixed percentage. Before deployment, build a complete memory budget based on target concurrency and maximum length: estimate all four items separately, then decide GPU type and count.
Context length matters just as much. The KV cache grows linearly with sequence length, so going from 8K to 128K multiplies a single request’s cache by 16. Multiply that by concurrency, and cache pressure quickly outpaces the growth of the weights themselves.
Choosing a Parallelism Strategy
Three common parallelism approaches for multi-GPU deployment:
| Parallelism | What is split | Advantages | Main cost |
|---|---|---|---|
| Tensor parallelism | Matrices within each layer | Lower per-GPU weight footprint, lower latency | Inter-GPU communication every layer; bandwidth-sensitive |
| Expert parallelism | Experts distributed across GPUs | Well-suited to MoE; fewer experts per GPU | All-to-all communication after routing; possible load imbalance |
| Pipeline parallelism | Layers split into stages | Less communication; easy across nodes | Pipeline bubbles; higher single-request latency |
MoE models typically combine tensor parallelism with expert parallelism. Expert load imbalance is the main risk in such deployments: if certain experts are selected at high frequency, the GPUs holding them become bottlenecks, and overall throughput is determined by the slowest GPU.
When tuning, monitor the following metrics:
- Tokens processed per expert per second, to detect hot experts;
- Compute utilization of each GPU, to identify the slowest one;
- All-to-all communication latency, to determine whether bandwidth is the bottleneck.
Pre-Launch Engineering Checklist
Before going live, verify each item:
- Weight precision: check module by module and confirm the mixed-precision distribution;
- Memory budget: estimate weights, KV cache, activations, and communication buffers separately, with headroom;
- Context cap: set maximum length and maximum concurrency based on business needs, so no single request can exhaust the cache;
- Parallelism configuration: confirm the tensor/expert parallelism combination and check inter-node bandwidth;
- Framework support: confirm the inference framework and quantization kernels support this MoE structure and sparse attention, with matching versions;
- Load balancing: track expert selection frequency and per-GPU utilization to detect hot experts;
- Quality regression: compare pre- and post-quantization results on business data, with special attention to long-text and multimodal inputs;
- Fallback plan: prepare a smaller model or rate-limiting strategy for traffic spikes.
Common Misconceptions
Misconception 1: Total parameters determine speed. Actual latency depends more on activated parameters, communication overhead, and batching strategy. Total parameters mainly determine whether the model fits.
Misconception 2: A million-token context means you can freely use it. The context limit is the upper bound of model capability. The practically usable length is also constrained by memory, latency budget, and evaluation results.
Misconception 3: Quantization preserves quality. Low-bit quantization loses precision. Mixed-precision models in particular must be validated on target tasks—public benchmarks alone are not enough.
Misconception 4: Eight GPUs will definitely work. GPU count is only one necessary condition. Per-GPU memory, inter-GPU bandwidth, framework support, and concurrency targets together determine whether a plan is feasible.
Summary
Deploying an ultra-large sparse MoE model comes down to estimating total parameters, activated parameters, cache, and communication separately. First use weight precision to establish the memory floor, then use concurrency and context length to size the cache margin, and finally use parallelism strategy and load monitoring to guarantee throughput. With this method, engineers can judge whether a plan is viable before purchasing hardware or launching a service.
