LLMCash

    GPT、DeepSeek、ERNIE、Claude、Gemini

    LLM Inference Compute Bottlenecks and KV Cache Scheduling Strategies

    1. Why Compute Is Usually the Bottleneck for Chat Services

    When you build a generative chat service, one question comes up again and again: what resource actually limits how many users you can serve at the same time? The answer is usually not bandwidth, and not the CPU. It is GPU compute combined with GPU memory.

    Here is why. In LLM inference, every token that gets generated requires moving the model weights and the current context state from GPU memory to the compute units, followed by a forward pass. As concurrent requests increase, the GPU is pushed toward its limits and runs at close to full utilization. It has to emit the current user’s remaining tokens as fast as possible and release that user’s memory footprint, then load the next request and keep generating.

    In other words, the amount of GPU capacity you have largely determines how many sessions you can keep alive at peak. When a service announces that it is short on compute and is pausing new user access, what is really happening is that peak concurrency demand has exceeded what the available compute can handle.

    2. Two Phases of LLM Inference: Prefill and Decode

    To understand the bottleneck, you first need to break inference down into its two phases.

    • Prefill: The model processes the entire user prompt in parallel and computes the Key/Value vectors for all positions in a single pass, producing the first token. This phase is compute-bound. The matrix operations are large and make good use of the GPU’s compute capacity.
    • Decode: The model generates tokens one at a time, autoregressively. Each new token requires reading all model weights and the KV cache of the context accumulated so far. This phase is memory-bandwidth-bound.

    The key insight is this: in chat workloads, most of the time is spent in the decode phase. The bottleneck there is not a lack of floating-point compute. It is how fast data can be moved. Arithmetic intensity, meaning the amount of computation performed per byte of data, is very low, so the GPU’s compute units spend a lot of their time waiting for data.

    3. The Memory Bandwidth Bottleneck in Decode

    Why does decode run into the memory bandwidth limit? Token-by-token generation processes only one new token (or one batch of new tokens) per step, yet each step must read:

    1. the model weights, which can amount to billions or even hundreds of billions of parameters;
    2. the entire KV cache that the current session has accumulated.

    That is a huge amount of data to read, while the amount of data actually used in computation is tiny. As a result, GPU memory bandwidth is repeatedly saturated and compute unit utilization drops. This also explains why “having a little less memory is not the end of the world”: even if a session’s cache cannot stay resident, the service can still run at high load as long as it frees resources quickly and puts compute to work on the next request’s throughput. How busy the GPU is, rather than the latency of a single request, is the core way to think about a service’s capacity.

    4. KV Cache: The Core Structure for Reuse in Inference

    In self-attention, every token produces a corresponding Key vector and Value vector. So that the system does not have to recompute the K/V of historical tokens at every step, the inference system caches them. This is the KV Cache.

    The point of the KV cache is to trade memory for compute. Repeated computation turns into a table lookup. The cost is that memory usage grows linearly with context length. Here is a simplified estimate of the memory used by one request’s KV cache:

    1. # Simplified: estimate KV cache memory for a single request
    2. def kv_cache_bytes(num_layers, num_kv_heads, head_dim,
    3. seq_len, dtype_bytes=2): # 2 bytes per element for fp16/bf16
    4. # Each token needs two cached tensors: K and V
    5. per_token = 2 * num_layers * num_kv_heads * head_dim
    6. return per_token * seq_len * dtype_bytes
    7. # Longer contexts and more concurrent sequences increase total usage
    8. total = sum(kv_cache_bytes(32, 8, 128, s) for s in batch_seq_lens)

    This shows that GPU memory capacity directly limits how many active sessions can be resident at the same time.

    5. The Concurrency Ceiling: Compute Sets Switching Speed, Memory Sets Residency

    Putting the earlier analysis together, peak concurrency is determined by two factors:

    • Compute (GPU FLOPS / throughput): This determines how fast tokens are emitted, in other words, how quickly a user can be served and the GPU switched to the next one.
    • GPU memory capacity: This determines how many sessions’ KV caches can be held at the same time.

    When memory runs short, the service does not crash immediately. It can proactively drop the KV cache of some sessions and give the scarce memory to higher-priority requests. When those users come back, the system can reload or recompute their context. This is the core idea of elastic capacity management.

    6. Why You Can Drop a KV Cache

    You might ask: won’t dropping the context hurt the user experience? The answer depends on how real sessions are actually accessed:

    • One-shot sessions are the majority: Many requests are independent Q&A, such as “What’s the weather like tomorrow?” or “Help me understand a concept.” Users rarely come back to the same session with follow-up questions.
    • Long multi-turn revisits are rare: Even in multi-turn conversations, it is relatively unlikely that a user returns to the same session several times within a day and keeps going deeper on the same topic.
    • The cost of dropping is manageable: When a user comes back after a long time away, the system can reload the historical KV from persistent storage (such as object storage or SSD), or re-run prefill over the full chat history. Compared with permanently occupying GPU memory and sacrificing overall throughput, this “trade time for space” strategy is usually the better deal.

    So KV cache is essentially disposable, reconstructible temporary state, which leaves a lot of room for capacity optimization.

    7. Three Common KV Cache Management Strategies

    In engineering practice, cache management approaches generally fall into three categories:

    1. Swap out and swap in: Move the KV cache of inactive sessions from GPU memory to main memory or faster local storage. Swap it back when the session becomes active again, so the system avoids recomputing from scratch.
    2. On-demand recomputation: Drop the cache outright and recompute it from the original prompt when the request comes back. This fits scenarios with infrequent revisits and limited storage bandwidth. It trades compute for memory.
    3. Compression and tiered caching: Lower each session’s memory footprint with techniques such as KV quantization and paged block management, where the cache is split into fixed-size blocks that are allocated on demand. This lets the same amount of GPU memory hold more concurrent sessions.

    Which strategy to choose depends on the balance between revisit frequency, storage bandwidth, and compute headroom.

    8. The Art of Trading Memory Against Compute

    The core engineering philosophy is: protect throughput first, and cache residency second. In practice, this means:

    • When GPU memory is tight, drop the caches of idle sessions rather than let the GPU run below its bottleneck point. Keep it running at high load.
    • Use spare compute to make up for a lack of memory: if you compute faster, you can switch to the next user sooner.
    • Move reconstructible state to storage tiers that are cheaper and larger (object storage, message queues to buffer requests, and cache services to reuse hot content).

    This approach reflects a “service, not session” mindset. The system optimizes for total token throughput and the average experience across users, rather than unbroken continuity for any single session.

    9. Capacity Planning and Engineering Practices for High Concurrency

    To reliably support high concurrency in an LLM service, you can put these practices in place:

    • Continuous batching: Insert newly arrived requests dynamically into the batch that is currently decoding, so the GPU always has work to do. This improves throughput significantly.
    • Request queuing and admission control: When compute is saturated, add queues and rate limiting to prevent overload from causing a cascading failure. This lets the system degrade gracefully instead of rejecting requests outright.
    • Priority scheduling and degradation: Keep resident cache for paid or high-priority sessions, and use swapping or recomputation for low-priority sessions.
    • Elastic scaling: Combine monitoring and alerting with queue depth and GPU utilization metrics to automatically add or remove inference instances. General-purpose capabilities on cloud platforms such as Baidu AI Cloud, including container platforms, load balancing, logging, and monitoring and alerting, can be used to build this kind of elastic inference foundation.
    • Multi-level cache reuse: Share cached prompt prefixes for frequent, reusable prompts, which cuts down repeated computation further.

    Here is a simplified sketch of the scheduling logic:

    1. def schedule_step(running_batch, waiting_queue, kv_pool):
    2. # 1. Admit new requests first (if memory has room to spare)
    3. while waiting_queue and kv_pool.has_free_blocks():
    4. req = waiting_queue.pop()
    5. kv_pool.alloc(req)
    6. # 2. Under memory pressure, swap out the coldest, lowest-priority session
    7. if kv_pool.is_under_pressure():
    8. victim = kv_pool.pick_cold_low_priority()
    9. kv_pool.swap_out(victim) # or mark it for recomputation
    10. # 3. Take one decode step on the active batch to keep the GPU busy
    11. step_decode(running_batch)

    10. Conclusion

    The compute pressure of a generative chat service is fundamentally a resource scheduling and state management problem, not simply a case of “not enough GPUs.”

    If you understand three things, you can build more elastic, higher-throughput inference services with finite resources. These are:

    • how the prefill and decode phases differ in their performance characteristics;
    • the fact that KV cache is disposable and reconstructible;
    • the trade-offs between compute and memory.

    When peak demand exceeds capacity, the right move is not to simply close the door to new users. Instead, use cache scheduling, batching optimizations, admission control, and elastic scaling to spend every unit of compute where it matters most. That is the key step in LLM engineering from “it runs” to “it can scale to serve users.”