Why LLM Chat Services Hit GPU Limits: KV Cache, Decode Bottlenecks, and Scheduling Optimization
Why do conversational LLM services suspend new sign-ups because of "compute shortage"? This article explains the resource model behind chat inference: prefill is compute-bound, while decode is constrained mainly by GPU memory bandwidth; peak concurrency therefore depends on how much GPU memory and bandwidth are available to hold KV caches. It shows why the one-shot nature of chat sessions makes KV cache eviction and recomputation viable, why admission control shows up to users as paused registration, and which techniques — continuous batching, paged memory management, KV cache quantization, draft-model decoding, tiered context handling — raise per-GPU throughput, along with elastic capacity planning practices for handling peak demand.
1. Compute shortage is a real constraint, not marketing spin
Lately, many conversational LLM services have cited “compute shortage” when suspending or tightening new-user registration and membership sign-ups. To anyone unfamiliar with what runs underneath, this can look like a commercial tactic. From a systems engineering perspective, however, it is a direct expression of supply-demand imbalance in resources. One core fact makes this clear: in the chatbot form factor, the resource that determines a cluster’s peak concurrent serving capacity is almost always GPU compute — more precisely, the combination of GPU memory capacity and memory bandwidth.
This differs fundamentally from traditional web services. Traditional services usually bottleneck on the backend database or network I/O, whereas every conversational inference request consumes a large amount of GPU memory to hold model weights and context cache, and continuously consumes compute units to generate content token by token. As concurrent users increase, memory and compute resources are divided up quickly, and physical limits are soon reached. “Pausing new-user access” is essentially a resource-protection action triggered once that physical ceiling is hit.
2. Two phases of inference: prefill vs. decode
A complete conversational inference pass has two phases with very different resource profiles:
- Prefill: the model processes all prompt tokens of the user input in parallel, computing representations for every input token in one pass. This phase is compute-bound, keeps the GPU’s compute units fully utilized, and is relatively friendly to raw compute capacity.
- Decode: the model generates tokens autoregressively, one at a time, based on the sequence produced so far. Each step emits a single token, appends it to the end of the sequence, and moves on to the next step.
Understanding this difference is the prerequisite for understanding compute shortage: the phase that ties up resources for a long time, step after step, is usually decode.
3. KV cache: the key abstraction behind chat serving
In the decode phase, to avoid recomputing the Keys and Values of all previous tokens every time a new token is generated, the system caches the attention key-value pairs of historical tokens. This is the KV cache. Its value is that it reduces the computational complexity of autoregressive generation from “recompute the entire sequence at every step” to “compute only the one new token at each step” — a key to faster inference.
The KV cache comes at a cost, though: it occupies GPU memory, and its footprint grows roughly linearly with context length and concurrency. The longer the context and the more simultaneous conversations, the more memory the KV cache consumes. This is also why long-context scenarios put particular pressure on GPU memory.
4. Why decode is constrained by memory bandwidth
Decode looks cheap: each step computes only one token, so the arithmetic involved is small. But every step must read the entire set of model weights from GPU memory (HBM) into the compute units, along with the KV cache of the current session. In other words, the bottleneck of decode is often not compute throughput but memory bandwidth — much of the time the GPU is waiting on data movement rather than on computation.
This yields a useful intuition: setting aside the memory-bandwidth bottleneck of decode, you can roughly regard the GPU as running at full utilization against its limits. How many GPUs you have therefore directly determines how many active sessions can be maintained at peak. This is the fundamental reason chat services are so sensitive to memory bandwidth.
5. Peak concurrency: how GPU count determines serving capacity
Chaining the logic above produces a clear causal chain:
- During decode, every active session occupies GPU memory to store its KV cache;
- The memory on a single GPU is finite, so the total KV cache it can hold at once is finite too;
- Once memory is full, requests from new users can no longer be loaded;
- As a result, the peak number of users served simultaneously is roughly proportional to the total GPU memory/bandwidth available.
It is worth stressing that “slightly less memory” is not necessarily fatal. In chat scenarios, the KV cache can be managed dynamically — the key to understanding the scheduling strategies that follow.
6. Can KV caches be dropped? The one-shot nature of sessions
Unlike state in a traditional database, which must be persisted, the session context of a chatbot can often be discarded. The reason is that the vast majority of chat requests have a distinctly one-shot character: a user asks “What’s the weather like tomorrow?” or asks for one detail about a trending event, gets the answer, and leaves — often never returning to that session.
Even for longer conversations, ordinary users are unlikely to return to the same session repeatedly and probe deeply within a single day. This gives providers an important scheduling degree of freedom: to serve more concurrent users, they can proactively evict the KV cache of temporarily inactive sessions and hand the freed memory to new requests.
7. Cache eviction and recomputation: an engineering trade-off
When a “temporarily forgotten” user returns to continue the conversation, the system usually has a few options:
- Reload the previously swapped-out KV cache from a faster tier, such as host memory;
- If the cache was swapped out to a slower tier such as SSD, restore it from there, at the cost of higher latency;
- If no cache is available to reuse, run prefill again with the full conversation history, recomputing from scratch.
These strategies are essentially trade-offs among three things: memory occupancy, recovery latency, and recomputation compute. Chat services generally tolerate some latency elasticity, so “drop first, restore later” is an acceptable engineering compromise in most business scenarios.
8. Admission control and rate limiting: why providers “pause sign-ups”
Once you understand both resource scarcity and the disposability of caches, the rationale behind actions like “pausing new-user access” becomes clear. It is not a refusal to serve, but an admission control mechanism: when active sessions have filled memory and the queue of waiting requests keeps growing, the provider closes the entry point for new users so that limited compute is reserved for users already connected, preventing an overall collapse in service quality.
A simplified admission decision can be expressed as follows:
def try_admit(new_request, gpu_pool):# Estimate the KV cache size the request needs (model weights are usually# handled by the resident region, so the focus here is the KV cache)required_kv = estimate_kv_blocks(new_request.prompt_len)# First, try to allocate memory blocks for the new userif gpu_pool.has_free_blocks(required_kv):gpu_pool.allocate(new_request, required_kv)return "ADMITTED"# If no memory is free, try evicting the least recently active session's cacheelif gpu_pool.evict_idle_sessions_and_retry(required_kv):gpu_pool.allocate(new_request, required_kv)return "ADMITTED"# Still doesn't fit: reject the new request# (surfaced at the product level as "pausing new-user access")return "REJECTED"
The core idea is this: admission depends on whether memory can satisfy the new session’s KV cache requirement, and the eviction mechanism is precisely what cushions this hard constraint.
9. General paths to higher per-GPU throughput
To ease compute pressure, providers can raise effective per-GPU throughput at several levels, indirectly relieving the need to pause sign-ups:
- Continuous batching: static batching must wait for every request in a batch to finish before releasing the whole batch’s resources, so a long request holds back the entire batch. Continuous batching lets finished requests leave immediately and new requests join immediately, substantially improving GPU utilization.
- Paged memory management: borrowing the idea of paging from operating-system virtual memory, the KV cache is split into fixed-size blocks and allocated on demand, reducing memory fragmentation and fitting more concurrent sessions into the same memory.
- KV cache quantization and compression: reducing the numerical precision of the KV cache lowers its memory footprint, allowing more sessions to be served with the same memory.
- Decode acceleration strategies: for example, using a small draft model or multi-candidate parallel prediction to verify several tokens in one pass, which reduces the number of serial decode steps and eases bandwidth pressure.
- Tiered handling of long and short contexts: for very long contexts, sparse attention or chunking strategies keep the KV cache growth rate under control.
10. Scalable architecture for elasticity and cost
Relieving compute shortage cannot rely on single-machine optimization alone; it also requires systematic capacity planning and elastic design:
- Elastic compute scheduling: during business peaks, use container-platform autoscaling to quickly add inference instances; during troughs, reclaim resources to reduce cost.
- Request tiering and prioritization: distinguish paid from free users, and interactive from batch-style requests, so core users receive stronger service-quality guarantees.
- Hot/cold tiering of context cache: offload KV caches of inactive sessions to cheaper tiers such as host memory or object storage, trading capacity for cost.
- End-to-end monitoring and alerting: use monitoring, alerting, and logging services to observe key metrics such as memory occupancy, queue length, and token throughput in real time, triggering scale-out or throttling early rather than passively closing the door only after resources are exhausted.
Conclusion
“Compute shortage, sign-ups paused” looks like a business move, but behind it is an entire set of technical constraints built around GPU memory, bandwidth, and KV cache scheduling. The resource model of conversational inference is distinctive: short requests, many sessions, disposable context — a source of compute pressure, and at the same time exactly the space that fine-grained scheduling can optimize. Understanding the difference between prefill and decode, the lifecycle of the KV cache, and the roles of admission control and batch scheduling makes the causes of compute shortage visible at the system level. For enterprises, using general cloud capabilities such as those offered by Baidu AI Cloud — container platforms, object storage, monitoring and alerting — to plan elastic capacity is a practical way to absorb peak fluctuations and carry workloads steadily.
