How VRAM and Compute Together Determine the Concurrency Limit of LLM Serving
A large language model service can only handle a limited number of concurrent sessions, and that limit is set by two resources: the effective token throughput the GPU can sustain, and how many session states fit in VRAM. This article walks through a three-step estimation method for computing that ceiling, explains why the decode phase is bandwidth-bound, breaks down the KV Cache memory formula, compares the three session-state strategies (keep, offload, and drop-and-recompute), and outlines admission control and tiered service practices for operating near capacity.
1. The Short Answer
How many sessions a service can handle at the same time is usually determined by two kinds of resources: how many effective tokens per second the GPU can output, and how many sessions’ intermediate states the VRAM can hold. Whichever hits its ceiling first forces the server to restrict new sessions, reduce per-session resource usage, or change how historical states are retained.
The estimation method in this article has three steps: first, compute how many active generating sessions the compute budget supports; second, compute how many sessions the VRAM can retain; finally, take the smaller of the two and apply a safety factor. There is no absolute winner among keeping, offloading, and dropping/recomputing session state—the right choice depends on session idle intervals, history length, and latency targets.
2. Why the Bottleneck Lives in the Decode Phase
Inference has two stages: prefill and decode. Prefill processes the user’s entire prompt in one pass, with heavy computation that fully utilizes the GPU’s matrix operations. Decode, by contrast, generates only one token per step and must read the entire model weights plus the historical KV Cache each time—computation is light but memory access is heavy, so this stage is typically bound by memory bandwidth. As a result, in most moments the GPU’s compute units are not fully utilized.
Servers commonly use continuous batching to merge decode steps from multiple sessions into the same batch. Each weight read then serves more sessions, and overall throughput rises. However, as the batch grows, each step takes longer, and the per-session token rate drops. This cost must be factored into any estimation—you cannot look only at total batch throughput.
3. Estimating Concurrency
3.1 The Compute Constraint
Start by defining three quantities:
- T_eff: the effective tokens per second the service can sustain at the target latency. This value must come from load testing—do not use the GPU’s theoretical peak.
- r: the token rate required per active generating session. It is determined by the user’s reading speed and product experience, and must also match the rate the server can deliver under batching.
- N_compute: the number of active generating sessions under the compute constraint, N_compute ≈ T_eff ÷ r.
Keep active sessions and online sessions separate. Online sessions include both sessions currently generating and idle sessions waiting for user input. If the active ratio is a, the number of online sessions is roughly N_compute ÷ a.
3.2 The Memory Constraint
The VRAM budget breaks down as:
VRAM budget = model weights + KV Cache + activation and runtime overhead + fragmentation and safety margin
The per-token KV Cache footprint is:
Bytes per token = 2 × number of layers × number of KV heads × head dimension × bytes per element
The factor of 2 accounts for the K and V caches. The formula uses the number of KV heads, not the number of attention heads. In GQA (grouped-query attention) and MQA (multi-query attention) models, the number of KV heads can be far smaller than the number of attention heads. Mistakenly using the attention head count will overestimate KV usage.
Example: 32 layers, 8 KV heads, head dimension 128, FP16 (2 bytes per element). Per-token usage is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, i.e., 128 KiB. A 4,096-token session occupies roughly 512 MiB. If the same model used MHA with 32 KV heads, the per-token footprint would be 4× larger.
Here is a capacity example. Suppose a single GPU has 80 GB of VRAM: model weights take 16 GB, activation and runtime overhead total 8 GB, and the safety margin is 8 GB. That leaves about 48 GB for KV Cache. Estimating with an average session length of 2,048 tokens, a single session’s KV Cache is about 256 MiB, and 48 GB (using the conversion of roughly 1,024 MB per GB) can hold about 192 sessions. Estimating with a P95 length of 4,096 tokens, only about 96 sessions fit. These numbers are illustrative—run your own measurements with your actual model and traffic.
3.3 Combining Both Constraints
The final concurrency limit is:
Concurrency limit ≈ min(N_compute, N_memory) × safety factor
The safety factor typically falls between 0.7 and 0.9 and should be determined through load testing.
One key point: if every online session’s KV Cache stays resident in VRAM, the memory constraint limits the number of online sessions—not just active ones. Continuing the example above, with a 30% active ratio, compute supports roughly 133 active sessions, corresponding to about 444 online sessions. But VRAM can only retain 192 online sessions, so memory hits its ceiling first. Once idle sessions are offloaded or dropped, they no longer occupy VRAM, and the bottleneck shifts back to compute and offload bandwidth.
4. Three Session-State Strategies for KV Cache
A session’s KV Cache typically moves through four states: active (currently generating, resident in VRAM), idle (the user has paused but the cache is still in VRAM), offloaded (moved to host memory or local SSD), and dropped (VRAM freed; recomputed when needed). Offloading and dropping are the two ways to free VRAM, corresponding to the last two of the three strategies.
4.1 The Three Strategies
Keep: Idle sessions’ KV Cache stays in VRAM. Recovery requires no extra work, giving the lowest time-to-first-token. The cost is high VRAM usage and a directly constrained concurrency count.
Offload: Move the KV Cache to host memory or SSD. VRAM is freed, but on recovery the cache must be transferred back to the GPU. The theoretical lower bound of transfer time is: cache size ÷ effective bandwidth. For a 256 MiB cache at an effective bandwidth of 20 GB/s, the transfer takes about 13 milliseconds; queuing and scheduling time must be added on top of that. Host memory capacity is finite, and SSD bandwidth is far lower than memory bandwidth, so each needs separate evaluation.
Drop and recompute: Free the KV Cache outright. On recovery, prefill must be re-run over the full history. Recompute time scales roughly with the number of historical tokens—the longer the history, the higher the time-to-first-token. If the inference engine supports prefix caching, shared system prompts and other prefixes can be reused, reducing the recompute cost.
4.2 Comparison
| Strategy | VRAM usage | Recovery latency | Extra overhead | Best suited for |
|---|---|---|---|---|
| Keep | High | Lowest | None | Multi-turn conversations with short idle gaps, sensitive to first-token latency |
| Offload | Low | Medium; depends on cache size and bandwidth | Host memory or SSD capacity and transfer bandwidth | Longer idle periods, long histories, tight VRAM |
| Drop and recompute | Lowest | Grows with history length | Prefill compute | Short histories, reusable prefixes, very long idle periods |
4.3 Choosing a Strategy
Use these principles when choosing:
- Short idle times with long histories: prefer keeping—the latency is optimal when VRAM is sufficient.
- Tight VRAM with long histories: use offloading to avoid the compute cost of recomputation.
- Short histories or reusable prefixes: drop-and-recompute is cheap and suits sessions with long idle periods.
Production systems usually make tiered decisions based on idle duration and history length. For example, offload after idle time exceeds one threshold, then drop sessions that remain idle even longer and have short histories. Thresholds should be determined through load testing and production data rather than set purely from experience.
5. Rate Limiting and Tiered Service Workflow
When compute or VRAM approaches its limit, the service needs explicit admission and degradation rules. A recommended workflow:
- Monitor key metrics: request queue length, KV Cache utilization, P95 time-to-first-token, per-token latency, and the counts of active and online sessions.
- Set admission thresholds: for example, when KV Cache utilization exceeds 85%, restrict admission of new sessions.
