KV Cache and Inference Capacity Management: Why LLM Services Run Out of Compute

1. The Phenomenon: A Hit App Runs Into the Compute Ceiling

Recently, a large model service saw its user base surge after the release of a new-generation model. The team published a notice saying it was facing an “unexpected compute shortage” and announced that sales of new membership plans would be suspended to prioritize the experience of existing users. The notice also clarified several operational details: features of existing plans are unaffected, continuous subscription and manual renewal remain supported, existing users can still upgrade their current plans, and the reopening date for new plans is to be determined.

Episodes like this are not isolated. Whenever a conversational AI product achieves explosive growth, inference-side compute supply is usually the first link to break. Rather than asking “which provider is short on GPUs this time,” it is more useful to take the problem apart: What actually determines the capacity of an LLM inference service? Why does the KV cache become the variable that can be “sacrificed”? And how can a provider degrade gracefully when resources run short?

2. The Root of the Shortage: A Resource Model for Inference Services

2.1 GPU Count Sets the Concurrency Ceiling

Setting aside the memory bandwidth bottleneck of the decode phase, it is reasonable to assume that GPUs are running close to full utilization at their bottleneck. This leads to a plain conclusion: the amount of GPU resources you own roughly determines how many concurrent users you can sustain at peak. Inference differs from training — training pursues aggregate throughput, while inference has to balance latency and concurrency, and the latter imposes a rigid constraint on resources that cannot be relieved by “squeezing a little harder.”

2.2 The Memory Bandwidth Bottleneck in the Decode Phase

Autoregressive generation has a defining characteristic: generating each token requires reading the full set of model weights once. For a model at the ten-billion-parameter scale, the volume of weight reads per step far exceeds the amount of computation, so the bottleneck is usually not compute (FLOPs) but memory bandwidth (bytes/s). This yields several direct corollaries:

• Increasing the batch size amortizes repeated weight reads and improves bandwidth utilization, but it pushes up per-request latency;
• Once the batch is full and bandwidth is saturated, admitting more requests only makes every user’s response slower at the same time;
• At that point, “not enough GPUs” is really “not enough bandwidth and memory,” and stacking up peak FLOPS numbers accomplishes little.

2.3 The KV Cache: The Elastic Bulk of GPU Memory

Inference memory usage consists mainly of two parts: model weights and the KV cache. Weights are static and non-negotiable; the KV cache grows linearly with context length and concurrency. A rough estimate:

KV memory ≈ 2 × number of layers × number of KV heads × head dimension × bytes per element × sequence length × concurrency

The stronger a model’s long-context capability, the larger the per-session KV cache; the more concurrent users, the more staggering the total. A natural corollary follows: the KV cache is the most elastic part of memory usage — weights cannot be dropped, but the KV cache can. This is the first lever providers pull when compute runs tight.

3. Why the KV Cache “Can Be Sacrificed”

3.1 Session Reuse Is Far Lower Than Intuition Suggests

Chatbot session behavior has a few empirical characteristics:

1. A large share of sessions are one-off queries: “What’s the weather tomorrow?” or “What are the traffic restrictions in a given city today?” — the user gets an answer and leaves;
2. Even in long-running conversations, users rarely return to the same session multiple times within a day to keep iterating;
3. The longer the time since the last activity, the lower the probability the session is resumed — a clear time-decay pattern.

In other words, keeping every session’s KV cache resident in GPU memory means paying 100% of the memory cost for a low-probability “follow-up turn.” That trade-off is negligible when compute is abundant, but it must be re-examined when compute is scarce.

3.2 Two Recovery Paths After Eviction

Dropping a KV cache does not mean losing session data — the chat log remains fully preserved in the storage layer, and there are two paths to recovery:

• Path A (recompute): when the user returns to continue the conversation, re-run prefill with the full history. The cost is one extra prefill computation and the corresponding first-token latency;
• Path B (swap-in): sink the KV cache to secondary storage such as an SSD and swap it back into GPU memory on a hit. The cost is IO time and engineering complexity, but it avoids the recompute cost.

In practice, teams usually apply hot/cold tiering based on session activity: hot sessions keep their KV cache in GPU memory, warm sessions spill to SSD, and cold sessions are evicted entirely with only the chat log retained, to be recomputed when necessary. The remark that “worst case, load it back from the SSD, or accept the loss and recompute from scratch” maps exactly onto these two paths.

3.3 A Quick Capacity Calculation

Suppose that in a long-context scenario a single session’s KV cache reaches the gigabyte scale — not an exaggeration. A single GPU’s usable memory can then hold only a handful of long sessions. If caches belonging to the roughly 80% of low-activity sessions can be evicted, the same memory can support several times as many concurrent users. The logic behind “a bit less memory? No big deal — compute a bit faster, and the next user’s request can keep loading” is precisely this: trade eviction for concurrency, and trade a controlled recompute cost for overall service reach. When compute is scarce, that trade is worth making.

4. Engineering Practices Under Compute Pressure

4.1 KV Cache Eviction Strategy

A practical hybrid strategy looks like this:

class KVCacheManager:
    def __init__(self, gpu_capacity_gb, ttl_s=1800):
        self.capacity = gpu_capacity_gb
        self.ttl = ttl_s
        self.active = {}  # session_id -> (cache, last_access)
    def evict_if_needed(self, incoming_gb):
        # Eviction priority: expired first, then least recently used; offload before dropping
        for sid, sess in sorted(self.active.items(),
                                key=lambda x: x[1].last_access):
            if self.free_gb() >= incoming_gb:
                break
            if now() - sess.last_access > self.ttl:
                offload_to_ssd(sid)   # Path B: offload to secondary storage
            else:
                drop_cache(sid)       # Path A: keep only the chat log; recompute if needed
            del self.active[sid]

Three points matter here: use a TTL as a backstop so zombie sessions cannot squat in GPU memory indefinitely; use LRU to capture the rule that “the longer a session goes unused, the more likely it will never be used again”; and prefer offloading over outright deletion, keeping the recovery cost at the IO level.

4.2 Request Queuing and Priority Scheduling

Cache eviction manages the stock already in memory; scheduling manages the flow still waiting in the queue:

• Set differentiated concurrency quotas and queue priorities by user tier;
• Impose a higher admission bar on extra-long-context requests, since their prefill and KV costs are several times those of short requests;
• Manage user expectations by returning queue position and estimated wait times, avoiding silent timeouts that breed anxiety.

4.3 Product-Side Degradation Through the Membership System

Compute pressure is not only an engineering problem; it is also a product and operations problem. Looking back at the notice described earlier, one can distill a textbook degradation playbook:

1. Protect the installed base: existing plans keep their features and subscribed users keep normal access, preventing panic-driven churn;
2. Keep the renewal channel: continuous subscription remains supported, and users who manually turned off auto-renewal get a renewal entry within a limited time window, confining the impact to the incremental side;
3. Keep the upgrade path: the ability for existing users to upgrade their plans is rolled out in phases;
4. Pause the increment: sales of new plans are suspended with the reopening date TBD, directly cutting off demand-side pressure on compute;
5. Communicate transparently: state the cause and the scope of impact clearly, and keep updating status as the situation evolves.

The core of this playbook is to shrink the service radius of limited compute from “acquisition” to “retention and experience,” trading commercial restraint for system stability.

4.4 Capacity Water Levels and Monitoring

Prevention beats reaction. An inference service should maintain a complete set of water-level metrics: memory utilization, bandwidth utilization, KV cache hit rate and eviction rate, queue length, and first-token latency percentiles. When the eviction rate keeps climbing or queue length crosses a threshold, scaling or rate-limiting plans should be triggered — rather than reacting passively after users start reporting errors en masse.

5. The Bigger Picture: Compute as Strategic Leverage

Viewed in isolation, one notice is an operations event; viewed against the industry backdrop, it is a cyclical phenomenon. The explosion of AI applications and the expansion of compute supply are not linearly synchronized: a hit app’s user growth is measured in days, while data center construction and the capacity ramp-up of advanced-process chips and high-bandwidth memory are measured in years. This mismatch in time scales guarantees that the cycle of “shortage — expansion — shortage again” will recur.

At the same time, AI compute is no longer just a commercial resource; it is deeply intertwined with industrial competition. The supply of high-performance GPUs and high-bandwidth memory is subject to production capacity, export controls, and other factors, and tightening in any single link propagates up the supply chain until it surfaces as some app’s “membership sales paused.” Evaluating AI infrastructure segments (memory chips, for example) therefore requires more than supply-demand cycles; industry-competition variables must be part of the analytical framework. For practitioners, this means two things. First, inference-side efficiency work — quantization and compression, cache management, intelligent scheduling — will retain long-term value because it directly hedges compute costs. Second, elastic compute strategies will become the norm: renting cloud GPUs on demand, hybrid deployments, and rapid scaling. Cloud platforms such as Baidu AI Cloud, which offer GPU rental, container platforms, and monitoring and alerting services, are among the infrastructure options for building this kind of elastic capability.

6. An Action Checklist for Technical Teams

Finally, the whole discussion condenses into a five-step, executable checklist:

1. Map your capacity model: memory = weights + KV cache + activations. Write the concurrency ceiling as a formula instead of a gut-feel estimate;
2. Ship KV cache lifecycle management: hybrid TTL/LRU eviction, SSD cold tiering, and full observability of hit and eviction rates;
3. Prepare tiered degradation plans: tighten long-context admission → queue and rate-limit → degrade features → pause incremental sales, with each level triggered and recovered in order;
4. Design the product side in parallel: protect existing users, keep renewal channels open, and publish transparent notices — translate technical degradation into language users can understand;
5. Build water-level monitoring and elastic scaling: use cloud elastic resources to absorb demand spikes, and use monitoring metrics to detect capacity inflection points early.

Compute pressure is never just a matter of “not enough GPUs.” It is a systemic problem shaped jointly by inference architecture, cache strategy, scheduling systems, and product operations. Understanding the KV cache layer means understanding the core lever of LLM serving capacity management; understanding how capacity interacts with industry cycles makes it easier to see the technical inevitability behind every “temporarily unavailable” announcement.