LLM Chat Service Concurrency Capacity Optimization: Compute Bottlenecks and KV Cache Management
Conversational LLM services are often assumed to be limited by compute, but their true ceiling on peak concurrency is usually memory (VRAM) capacity, decode-phase memory bandwidth, and scheduling strategy. This article explains how autoregressive decoding works, why the decode phase is bottlenecked by memory bandwidth rather than raw FLOPS, how KV Cache scales with concurrency and context length, and how techniques like paged memory management, continuous batching, swapping, eviction, and admission control let a single node serve far more concurrent users. It is intended for engineers building or choosing inference services who want to maximize the number of users served under fixed GPU resources.
1. Why the Capacity Bottleneck of Conversational AI Falls on Compute
In a public-facing online chat service, the response speed users perceive and the peak number of simultaneous users the platform can serve are ultimately governed by the same thing: how many concurrent requests the system can advance per unit of time. The ceiling on that number is usually neither public network bandwidth nor disk I/O, but the supply capacity of compute resources such as GPUs.
The reason is that large-model inference is both compute-intensive and memory-intensive. Every user request occupies a slice of GPU memory to hold runtime state on top of the model parameters, and it continuously occupies compute units to generate text. As concurrent requests increase, that memory fills up request by request with runtime state, and the compute units switch back and forth among many requests. So the question of “how many users can be maintained simultaneously” ultimately becomes an engineering question of “how much GPU resource is available, and how efficiently can it be used.” Understanding this causal chain is the prerequisite for capacity management of a chat service.
2. Autoregressive Decoding: A One-Token-at-a-Time Paradigm
Text generation in large language models follows the autoregressive paradigm: the model predicts one token at a time, appends it to the end of the existing sequence, and then predicts the next token based on the updated full sequence, repeating until an end-of-sequence token is reached or a length limit is hit.
This process can be roughly split into two phases:
- Prefill phase: processes all prompt tokens of the user input in parallel in a single pass. It is compute-heavy with high parallelism and can usually saturate the compute units well.
- Decode phase: generates token by token. Each step processes only one new token but must perform attention against the context of all preceding tokens.
Most of the time cost in an online chat service falls in the decode phase. To avoid redundant computation, the system caches the Key and Value tensors produced by each token at every attention layer. This is the KV Cache. It reduces “generating token N” from “recomputing the previous N-1 tokens” to “computing only token N.” It is a necessary condition for decoding to run at all, but it is also what plants the seed of the concurrency problem on the memory side.
3. The Real Decode-Phase Bottleneck: Memory Bandwidth, Not Compute Peak
A commonly misunderstood point is that slow decoding is often not because compute (FLOPS) is insufficient, but because memory bandwidth is insufficient.
At each decode step, to generate one token, the GPU must move nearly all of the model weights from memory through the compute units. The matrix shape is “1 × weights,” so the arithmetic intensity (the amount of computation performed per byte moved) is extremely low. This means the compute units are frequently waiting on data and remain underfed for long stretches. As a result, in this phase the GPU is actually running at full tilt but hugging the memory-bandwidth bottleneck line, and simply piling on more compute peak does little to improve decode throughput.
This characteristic leads to two inferences:
- Adding compute alone (higher-FLOPS cards) yields diminishing marginal returns for decode throughput, because the bottleneck is on the bandwidth side.
- The technique that genuinely boosts concurrent throughput significantly is serving more requests per single memory move, that is, batching.
4. KV Cache: The Hidden Rival to Concurrency
If the weights are a fixed cost, then the KV Cache is a variable cost that grows linearly with concurrency and context length. Its per-token footprint can be approximated as:
kv_bytes_per_token = 2 (K and V) × num_layers × num_kv_heads × head_dim × bytes_per_element
For a typical order of magnitude (values used only to illustrate the rule): if the KV footprint per layer per token is on the order of hundreds of KB, a session several thousand tokens long can produce a KV Cache reaching hundreds of MB. When concurrent users grow from tens to thousands, the KV Cache alone is enough to exhaust the memory of a single card, or even multiple cards.
This leads directly to the core formulaic understanding of capacity:
- Weight footprint is constant (loaded once, resident in memory).
- KV Cache footprint grows with “concurrency × average context length.”
- Therefore, peak concurrent users ≈ (available memory − weight footprint) ÷ average KV Cache per session.
With a little less memory, the problem does not look serious: as long as the average KV footprint per session is controllable and scheduling is fast enough, room can be found to squeeze in more requests. But if contexts are generally long and concurrency is high, memory will become the bottleneck that breaks the service before compute does.
5. How Memory Constraints Directly Determine Peak Concurrent Users
Putting the relationship above into an engineering decision yields a clear main thread: with the weight footprint fixed, how many users you can maintain simultaneously depends on how much memory is left and how much memory each user consumes on average.
This explains why the key levers of capacity management lie not in a “stronger single card” but in “saving more memory and achieving higher reuse”:
- Reduce the KV footprint per session (more compact representations, quantization, more sensible context management).
- Improve memory utilization (reduce fragmentation, allocate on demand rather than reserve).
- When memory is tight, trade time for space (swap-out, discard).
For chatbot-type scenarios there is one very favorable characteristic: sessions are highly disposable. Many requests are one-off — queries like “Will the weather be good tomorrow?” or “the details of some historical event” are asked and then abandoned, and the probability that users return to the same session is low. Even for longer multi-turn conversations, users rarely iterate repeatedly on the same session within a single day. This provides a practical basis for aggressive KV Cache reclamation strategies.
6. Swapping and Evicting KV Cache: Trading “Recompute” for “More Users”
Since the KV Cache is only an optimization that “speeds up repeated access” and not a “requirement for correctness,” there is a full spectrum of measures, from gentle to aggressive, to free up memory when it runs short:
- Paged memory management: borrowing the idea of operating-system paging, split the KV Cache into fixed-size blocks allocated on demand, and use a block table to maintain the mapping from logical sequences to physical blocks. This significantly reduces memory fragmentation and improves utilization.
- Swap: when memory is insufficient, move the KV Cache of low-priority or long-idle requests to slower storage such as system memory or even SSD, and swap it back when that user interacts again, trading bandwidth and latency for memory.
- Evict and recompute: more aggressive still, directly discard a request’s KV Cache and hand its memory immediately to queued requests. If that user later returns to continue the conversation, the context is reloaded from storage, or the whole chat history is simply recomputed from scratch.
For chatbot businesses, the “cost” of the third strategy is often acceptable: a discarded session most likely never returns, and even if it does, the latency of recomputing once is far smaller than the loss from memory saturation causing site-wide request queuing and timeouts. In essence, this trades “an occasional extra delay for a few individuals” for “a substantial improvement in overall concurrency capacity” — a highly worthwhile capacity-engineering tradeoff.
7. Engineering Practices for High-Concurrency Serving
Combining the principles above, the common general practices on the inference-serving side can be summarized as follows:
1. Iteration-level continuous batching
Traditional static batching must wait until every request in a batch finishes generating before starting the next batch, so short requests get held back by long ones. Continuous batching schedules dynamically at the granularity of “each forward pass (generating one token)”: whoever finishes exits, and the memory they free up is immediately made available to new requests. Illustrative pseudocode:
running, waiting = [], [] # in-progress queue / waiting queuewhile True:# 1) Admit new requests using free memory as much as possiblewhile can_admit(waiting, free_kv_blocks):running.append(waiting.pop())# 2) Run one forward pass over the current batch; each request advances one tokenoutputs = model.forward(running)# 3) Finished requests exit and release their KV Cache memoryfor req in finished(running):free_kv_blocks.release(req.kv_blocks)# 4) When memory is tight, swap out / evict low-activity requestsif not enough_memory(running):evict_cold_requests(running) # swap to memory/storage or drop directly
2. Memory-aware admission control
Maintain an admission check at the entry point: admit a request only when free memory is sufficient to hold its initial KV Cache; otherwise queue it. This avoids the avalanche of “over-admitting so that all requests get dragged down together.”
3. Context-length management
Apply summarization, truncation, or segmentation to overly long contexts, reducing the KV footprint per session at the source — equivalent to expanding concurrency capacity.
4. Priority and fair scheduling
Reserve memory quotas for paid or high-priority requests, and let low-priority requests be swapped out first under pressure, balancing experience and capacity.
5. Horizontal scaling and elastic scheduling
When a single instance reaches its capacity ceiling, achieve elastic scaling through containerized deployment combined with monitoring and alerting. For cold data, general-purpose capabilities such as object storage can carry swapped-out context, leaving memory and GPU memory for hot requests. Baidu AI Cloud’s combination of general-purpose capabilities such as elastic containers, monitoring and alerting, and object storage fits well with this kind of elastic capacity-management need.
8. Final Thoughts
The capacity problem of conversational LLMs looks, on the surface, like “compute is tight,” but at a deeper level it is the combined result of “memory constraints + the decode-phase bandwidth bottleneck + scheduling strategy.” What truly determines peak concurrency is not the compute peak of a single card, but the system’s overall ability to reuse memory, schedule requests, and trade space for time when necessary under high concurrency. When the industry faces fluctuations in compute supply, rather than passively waiting for more GPUs, it is better to proactively mine capacity gains from batching and scheduling, KV Cache management, context management, and elastic architecture. Once you understand this logic, whether you build your own inference service or choose a mature cloud-based inference solution, you can serve more users under limited resources.
