GPU Compute Bottlenecks and Concurrency Optimization in LLM Online Serving
When a conversational LLM service suddenly goes viral, operators quickly discover that rising traffic brings not proportional revenue but rapid capacity exhaustion: inference queues grow longer, response latency keeps climbing, and the platform is eventually forced to block new users or even suspend some subscription tiers. On the surface this looks like an operational decision, but it is fundamentally a hard systems-engineering constraint: **given a fixed pool of GPU resources, how many users can a single online inference service actually serve at the same time?** This article does not target any specific product. Starting from the low-level mechanics of an inference engine, it explains this problem thoroughly and offers actionable approaches for scaling and optimization.
Introduction: Why a Popular Inference Service Suddenly “Throttles”
When a conversational LLM service suddenly goes viral, operators quickly discover that rising traffic brings not proportional revenue but rapid capacity exhaustion: inference queues grow longer, response latency keeps climbing, and the platform is eventually forced to block new users or even suspend some subscription tiers. On the surface this looks like an operational decision, but it is fundamentally a hard systems-engineering constraint: given a fixed pool of GPU resources, how many users can a single online inference service actually serve at the same time?
This article does not target any specific product. Starting from the low-level mechanics of an inference engine, it explains this problem thoroughly and offers actionable approaches for scaling and optimization.
1. The Nature of the Compute Bottleneck: GPUs Set the Peak Concurrency Ceiling
1.1 The Full Lifecycle of a Request
After a user presses Enter in the chat box, the server places the request into a batch, the GPU performs the computation, and the model emits the output one token at a time until it hits an end-of-sequence token or reaches the length limit. Throughout this loop, the GPU runs at near-full load against its bottleneck: it must finish generating tokens for the current batch of users as fast as possible so it can free up resources to load the next batch.
As a result, the number of users that can be maintained simultaneously at peak is directly determined by the available GPU resources. This includes both compute throughput (FLOPs) and memory capacity, which together form the ceiling on concurrency.
1.2 The Dual Constraints of Compute and Memory
Many people fixate on “not enough compute” while overlooking memory as a hidden killer. The direct effect of having less memory is that less space is left for the KV cache, which reduces the number of active sessions that can be held at once. But a chatbot has one critical property: the KV cache can be discarded and rebuilt. The vast majority of sessions are one-off interactions (“Will it rain tomorrow?” “Help me rephrase this sentence”), and the probability of a user returning to the same long session is not high.
This makes a pragmatic engineering tradeoff possible: prioritize evicting the KV cache of idle sessions to serve more users, then, when a user comes back days later to continue the conversation, read the cache back from SSD or simply recompute it from the history. This “trade compute for concurrency” approach is the core logic for coping with tight compute resources.
2. Prefill and Decode: Two Fundamentally Different Workloads
LLM generation is divided into two phases whose bottlenecks are completely different:
- Prefill: Processes all prompt tokens of a user’s input in parallel in a single pass, building the initial KV cache. This phase is compute-intensive—the GPU’s compute is fully saturated and the matrix operations have a high degree of parallelism.
- Decode: Generates output tokens one by one. Each step processes only a single new token but must read all of the model weights and the ever-growing KV cache. This phase is memory-bandwidth-intensive.
Understanding this distinction is essential: if memory bandwidth is the bottleneck, then simply piling on more compute (FLOPs) will not improve decode throughput. This also explains why scaling an online service is often not something that adding more cards can solve linearly.
3. Memory Bandwidth: The Underestimated Real Bottleneck
During the decode phase, to generate a single token the GPU must move the large weight matrices from memory to the compute units. This transfer is limited by memory bandwidth (on the order of GB/s for HBM), not by the peak throughput of the compute units.
A simplified way to understand it: at this point the GPU acts more like a “mover” than a “computer,” working at full load against the bandwidth ceiling. What determines single-card decode throughput is memory bandwidth together with the arithmetic intensity of the computation. Batching can amortize the cost of moving weights—loading the weights once to serve multiple requests within a batch—and this is precisely the core technique for improving throughput.
4. The KV Cache: The Key Variable for Concurrency
4.1 How the KV Cache Grows
The KV cache grows linearly with sequence length and concurrency. The KV footprint of each request can be approximated as:
KV bytes per token = 2 (K and V) × num_layers × num_kv_heads × head_dim × bytes_per_elementKV per request = KV bytes per token × context lengthTotal KV footprint = KV per request × concurrency
The “2” represents the two caches for the Key and the Value. As shown, long context and high concurrency can quickly fill up GPU memory.
4.2 Why the KV Cache in Chat Scenarios Can Be “Elastically Discarded”
As discussed above, chat sessions are mostly one-off and short-lived. This gives the inference engine freedom in scheduling: prioritize memory for requests that are actively decoding, and for idle sessions either offload them (to CPU/SSD) or discard them outright and rebuild on demand. Within an acceptable range of perceived latency, this strategy significantly raises peak concurrency.
5. Engineering Techniques for Improving Concurrency
The following are optimization directions commonly adopted in the industry, which can be combined:
5.1 Continuous Batching
Traditional static batching must wait for the entire batch of requests to finish before it can load new ones, which leaves the GPU idling. Continuous batching schedules dynamically at the token level: as soon as a request finishes generating, it exits immediately and frees its slot, and new requests can join at any time. This substantially improves GPU utilization and overall throughput.
# Continuous batching scheduler pseudocodewhile engine.running:batch = scheduler.form_batch(active_requests) # dynamically form the current batchoutputs = model.decode_step(batch) # advance one stepfor req, token in zip(batch, outputs):req.append(token)if req.is_finished(): # exit as soon as finished, immediately free the slotengine.release(req)scheduler.admit_waiting_requests() # new requests fill the empty slots at any time
5.2 KV Cache Paging and Tiered Storage
Splitting the KV cache into fixed-size “pages” for non-contiguous management can significantly reduce memory fragmentation and improve utilization. Combined with tiered storage, cold pages can be pushed down to CPU memory or SSD, forming a multi-tier cache hierarchy across GPU memory, host memory, and storage.
5.3 KV Cache Quantization
Compressing the KV cache from FP16 to INT8 or even lower precision can multiply the context length and concurrency that can be held, with controllable loss in accuracy. This is one of the most direct ways to relieve the memory constraint.
5.4 Prefix Cache Reuse
For common prefixes such as system prompts and long documents, cache their KV and share it across multiple requests to avoid repeated prefill. The benefits are especially pronounced in high-frequency scenarios with a “fixed system prompt plus varying user input.”
5.5 Speculative Decoding
Use a lightweight draft model to quickly generate candidate tokens, then have the main model verify them in a batch, reducing the number of decode steps for the main model and improving single-request speed in the bandwidth-limited decode phase.
6. Capacity Planning: Estimating How Many Users You Can Serve
Below is a memory-constrained estimation template you can apply directly (the values are illustrative only):
def estimate_max_concurrency(num_layers=32, num_kv_heads=32, head_dim=128,bytes_per_element=2, # FP16=2, INT8=1avg_seq_len=4096,gpu_mem_gb=80,kv_mem_ratio=0.6, # reserve 60% of memory for the KV cache):per_token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_elementper_request = per_token * avg_seq_lenkv_budget = gpu_mem_gb * (1024**3) * kv_mem_ratioreturn kv_budget // per_request # theoretical max concurrency under the KV constraint aloneprint(estimate_max_concurrency())
In practice, you also need to apply correction factors for the compute constraint, scheduling fragmentation, and batching efficiency. The theoretical value is therefore usually an upper bound, and actual concurrency should leave a generous safety margin.
7. Best Practices Summary
- Locate the bottleneck first: Determine whether your service is prefill-limited or decode-limited. For the former, look at compute; for the latter, look at bandwidth and memory.
- Maximize utilization first: Continuous batching is the change with the highest return on investment.
- Manage memory as a scarce resource: Combine paging, KV quantization, and prefix reuse.
- Trade compute for concurrency: Actively evict or offload the KV cache for one-off sessions, and rebuild it on demand.
- Leave margin in capacity planning: Estimate the upper bound under the memory constraint, then discount for peaks and fragmentation.
- Use elastic scaling as a safety net: Rely on containerized deployment, monitoring and alerting, and auto-scaling to dynamically add GPU resources during traffic peaks.
Compute pressure is not an unsolvable problem; at its core it is a question of resource scheduling and engineering optimization. Once you thoroughly understand the mechanisms above, you can serve more users on existing hardware and reserve elastic headroom for business growth.
