LLM Inference GPU Memory Planning: Don't Only Focus on Active Parameters
When sizing GPUs for large language model inference, many teams look only at active parameters. This article explains why that metric is not enough, especially for Mixture-of-Experts models, and breaks down the full GPU memory footprint: model weights, runtime overhead, KV cache, context length, and concurrency. It also covers key performance metrics such as TTFT and output throughput, along with practical deployment tuning advice.
1. Why “How many GPUs do I need?” is a common trap
When a new large model is released, one of the most common questions in developer communities is: “How many accelerators do I need to run it?” The question itself is reasonable, but simplifying the decision to “active parameter size equals the GPU memory required” can cause serious problems when the model goes into production.
The root cause is that active parameters describe the scale of computation, not the scale of storage. This distinction is especially important in modern model architectures, particularly Mixture-of-Experts (MoE).
2. Active parameters are not the same as GPU memory usage
Active parameters refer to the parameters that are actually activated and used in matrix computation during a single forward pass. In an MoE architecture, a gating network routes each token to only a few expert subnetworks. As a result, while the total parameter count can be enormous, the number of active parameters per inference can remain relatively small.
This is one of the key selling points of MoE: a large-capacity model can be served with relatively low compute cost. However, there is an important misunderstanding here:
Small active parameters do not mean that only those weights need to be placed in GPU memory.
When an inference engine loads a model, it must load all expert weights into GPU memory up front because it cannot know which expert a future token will be routed to. Even if an expert is not currently active, its parameters still occupy GPU memory and must remain available for future calls.
Therefore, deciding whether the hardware is sufficient requires more than counting active parameters. The full set of weights, runtime memory usage, KV cache, context length, and concurrency headroom must all be considered.
3. What makes up GPU memory usage in LLM inference
The GPU memory required for inference can be estimated by breaking it down into the following components:
1. Full model weights
This is usually the largest part of GPU memory usage. For MoE models, the actual memory footprint of the weights is determined by the total number of parameters, not the active parameters. With FP8 precision, for example, memory usage is roughly the parameter count multiplied by the number of bytes per parameter (about 1 byte for FP8), then multiplied by the per-GPU share after parallel sharding.
2. Runtime and framework overhead
The inference engine itself, CUDA context, communication buffers, and temporary activation tensors also consume GPU memory. This part is often not included in parameter tables, but it is real and usually requires reserving a certain proportion of memory.
3. KV cache
This is dynamic memory that grows with context length and concurrency during autoregressive inference. It is also the part most likely to exhaust GPU memory in long-context scenarios. Its size is approximately:
KV_Cache ≈ 2 × layers × KV heads × head dimension × sequence length × concurrency × bytes per element
Sequence length and concurrency are two linear scaling factors. If either is increased significantly, the KV cache can expand quickly.
4. Headroom for context length and concurrency
With ultra-long contexts such as 128K, each request needs to maintain a very long KV cache. If multiple requests run concurrently, GPU memory consumption increases multiple times. As a result, concurrency and context length often require a trade-off.
4. Key performance metrics: don’t focus on just one number
Once the model can run, two performance metrics deserve the most attention because they measure different things:
- Time to First Token (TTFT): the time from sending a request to receiving the first token. This directly affects the response speed perceived by users.
- Output Throughput: after entering the decoding phase, the number of tokens that can be stably generated per unit time. This affects the smoothness of long-text generation.
Interestingly, the performance of these two metrics can vary depending on prompt length and scenario:
| Scenario | Typical behavior |
|---|---|
| Short requests | Solutions with lower TTFT and faster startup have an advantage |
| Long prompts | Solutions with lower TTFT start to show an advantage, but output throughput may still vary |
This means that in long-prompt scenarios, one solution may have lower TTFT while another may achieve higher output throughput. The two metrics may be led by different frameworks or configurations.
This also shows that there is no need to commit to a particular framework in advance. Different inference frameworks make different design choices around scheduling, batching, and prefix reuse. Testing against your actual workload characteristics—short requests versus long generations, low concurrency versus high concurrency—is the only reliable way to decide.
5. Practical tuning advice for deployment
1. Don’t set the maximum context length on the first launch
A common mistake is to configure the maximum context length immediately. This raises the baseline GPU memory required by the KV cache and can make the model barely load, leaving little or no room for concurrency. A better approach is to start with a smaller context length, verify the baseline memory required by weights and runtime overhead, and then increase it gradually.
2. Disable advanced optimizations first to establish a clean baseline
When evaluating performance, disable advanced optimizations such as multi-token prediction (MTP) and prefix caching first to get a baseline “bare run” result. This makes it easier to understand how much benefit each optimization actually provides instead of being confused by combined effects.
3. Match precision and parallelism strategies to the hardware
Using native low precision, such as FP8, can significantly reduce weight memory usage and memory bandwidth pressure. Tensor parallelism (TP) splits each layer’s weights across multiple GPUs and can help when a single GPU does not have enough memory. However, higher parallelism usually increases inter-GPU communication overhead. The right setting depends on the interconnect bandwidth between GPUs; more parallelism is not always better.
4. Use benchmark data to guide decisions
Context length, concurrency, precision, and parallelism are coupled. Changing one setting can affect both memory usage and performance. It is best to build a benchmark matrix and record TTFT and output throughput at different prompt lengths and concurrency levels. Use those results to choose the production configuration that best fits your workload.
6. Summary
Going back to the original question: “How many GPUs do I need?” is never just a parameter-counting question. Success depends on whether you calculate the full model weights, runtime overhead, KV cache, context length, and concurrency headroom; whether you evaluate performance using both TTFT and sustained output throughput; and whether you are willing to make decisions based on benchmarking rather than assumptions. Understanding the difference between active parameters and GPU memory usage is the key first step—establish the baseline first, then optimize and scale.
