KV Cache Trade-offs and Scheduling Under Compute and Memory Pressure

    Serving large language models at scale means managing two scarce resources at once: GPU memory and compute. The concurrency ceiling of an LLM service is largely determined by how many per-session KV caches fit into the remaining GPU memory, while decode throughput determines how quickly sessions finish and release resources. This article explains what KV cache is, how to estimate its size, why its value decays over time, and how to choose between keeping it in GPU memory, offloading it to external storage, or discarding it and recomputing—using an expected-cost comparison as the decision framework.

    1. Background: Concurrency Is Bounded by Both Memory and Compute

    LLM serving typically runs in two phases. The prefill phase processes the entire input prompt in one pass. It is compute-heavy and highly parallelizable. The decode phase generates output tokens one at a time; producing each token requires reading the model weights and the historical cache, so it is primarily bounded by memory bandwidth, with compute units often idle waiting for data.

    In chat scenarios, the metric a service provider cares most about is how many active sessions can be maintained simultaneously. Each session consumes two kinds of resources. The first is model weights, which are shared by all sessions and stored only once. The second is the per-session KV cache. As a result, the achievable concurrency is roughly “remaining GPU memory divided by per-session cache size.” Decode speed determines how fast each request completes and frees its resources. Together, these two factors define the concurrency ceiling.

    2. What KV Cache Is and Why It Takes So Much Space

    When a Transformer generates a new token, it attends over all previous tokens. To avoid recomputing the Key and Value vectors of historical tokens at every step, the system stores them layer by layer—this stored data is the KV cache. Its size can be estimated as:

    Per-session KV cache bytes ≈ 2 × number of layers × number of KV heads × head dimension × context length × bytes per element

    Consider a concrete example. Suppose a model has 32 layers, 8 KV heads, and a head dimension of 128, using 16-bit floats (2 bytes per element). The cache per token is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, about 128 KiB. With a context of 4,096 tokens, a single session occupies roughly 512 MiB. If 40 GiB of GPU memory is available for caching, at most about 80 such sessions can be held concurrently—and that figure excludes activations and memory fragmentation, so the real number is lower.

    The takeaway: the longer the context and the more sessions there are, the greater the memory pressure. Long conversations are the dominant source of memory consumption.

    3. Cache Value Decays Over Time

    The value of a KV cache comes from reuse. If a user continues the same session shortly after, keeping the cache avoids reprocessing the entire history. If the user returns after a long gap—or the session was a one-shot question—the cache is unlikely to be reused, yet it still occupies GPU memory.

    The service therefore needs to decide whether a session’s cache is worth keeping. Common signals include: time since the last interaction, session history length, the user’s historical reuse rate, and whether a shareable common prefix exists (for example, an identical system prompt). These signals jointly determine which storage tier the cache should live in, or whether it should simply be dropped.

    4. Comparing Three Strategies

    Strategy 1: Keep in GPU memory. The session cache always stays resident in GPU memory. Resuming a conversation requires no preparation, and time-to-first-token is minimal. The cost is continuously occupied memory and reduced concurrency. This suits high-reuse, latency-sensitive sessions.

    Strategy 2: Offload to external storage. Idle session caches are moved to host memory or local SSD and read back when needed. This frees GPU memory while preserving fast recovery. Reloading requires data transfer, and its latency depends on storage bandwidth and cache size. If the cache is large and bandwidth is limited, reload time can approach the cost of recomputation, sharply reducing the benefit.

    Strategy 3: Discard and recompute. No cache is stored. When the user returns, the full chat history is re-prefilled. This consumes no extra storage and is the most concurrency-friendly option. However, compute cost grows linearly with history length, long conversations consume significant compute, and users wait longer for the first token.

    None of the three strategies is universally best; the right choice depends on reuse rate, session length, and current resource pressure.

    5. Choosing a Strategy by Expected Cost

    For an idle session, compare two latency figures:

    Reload cost ≈ cache size ÷ storage read bandwidth + queueing delay
    Recompute cost ≈ number of history tokens ÷ prefill throughput

    If reload cost is clearly lower than recompute cost, offload to external storage. If the two are close, or the cache is very large, discarding is more reasonable. Whether to keep the cache in GPU memory depends on the trade-off between reuse probability and memory footprint: keep it when reuse probability is high and memory is plentiful; demote it step by step when reuse probability is low and memory is tight.

    Below is illustrative decision logic; thresholds should be tuned against real load-testing results:

    1. def choose_policy(session, mem_pressure, reuse_prob):
    2. # Estimate the latency of both recovery paths (in seconds)
    3. reload_cost = session.kv_bytes / ssd_bandwidth + queue_wait
    4. recompute_cost = session.history_tokens / prefill_throughput
    5. if reuse_prob > HIGH_REUSE and mem_pressure < LOW_WATERMARK:
    6. return "KEEP_IN_GPU"
    7. if reload_cost < recompute_cost * 0.5: