GPU Inference Bottlenecks and the KV Cache Trade-off

1. From an Announcement: Why Compute Capacity Halts Service Expansion

Recently, a large language model (LLM) platform announced a suspension of new member registrations, citing “compute capacity shortages.” This sparked discussions within the developer community. While some suspected commercial motives, others viewed it as an inevitable outcome of the industry’s supply-demand imbalance in compute resources. Setting speculations aside, this event serves as an excellent technical entry point: in the era of conversational AI, what exactly determines a platform’s service ceiling? Why would a seemingly commercial action like opening memberships be directly bottlenecked by compute issues?

This article attempts to deconstruct the engineering logic behind this phenomenon by examining the resource model of LLM inference.

2. The Resource Model of Chatbots: Compute Determines Concurrency Limits

Unlike traditional internet services, the resource model for chatbot services can be extremely simplified into one sentence: how many users you can serve concurrently at peak times fundamentally depends on how many GPUs you have.

The reason lies in how LLM inference works. A complete conversational request is processed on the server side roughly as follows:

1. Receive the request and perform Prefill on the input context, generating the Key/Value intermediate states required for attention for each token in the sequence.
2. Enter the Decode phase, generating the response token by token in an autoregressive manner.
3. Once generation is complete, unload the KV cache occupied by this session to release VRAM.
4. Load the next user request and repeat the above process.

In this workflow, GPU compute power directly determines “how many tokens can be output per second.” The faster tokens are generated, the shorter the residence time of a single request, and the faster VRAM and scheduling slots can be freed up for the next user. Therefore, from a system perspective, the GPU operates as close to the bottleneck as possible at full load—the number of GPUs determines the peak number of concurrent sessions that can be maintained.

3. The Trade-off Between VRAM and Compute: Why “Calculating Faster” Relieves VRAM Pressure

A common misconception is that the concurrency limit is primarily constrained by VRAM. In reality, the relationship is more nuanced.

Having slightly less VRAM is not necessarily a major problem—as long as compute is fast enough, requests complete quickly, VRAM turnover is rapid, and the next request can be loaded sooner. In other words, VRAM determines the static upper limit of “sessions currently in flight,” while compute determines the turnover speed of these sessions. In engineering practice, VRAM shortages can be mitigated through more compact KV cache allocation or more aggressive eviction strategies. Compute shortages, however, leave almost no room for workarounds—you cannot make a user’s wait “slow but not slow.”

Of course, the Decode phase has an unavoidable physical bottleneck: memory bandwidth. During autoregressive generation, every time a token is produced, the model weights and the current KV cache must be read in their entirety, meaning the compute units are often waiting for data. This is why inference performance tuning mostly revolves around bandwidth: weight quantization (e.g., INT8/INT4), cache compression, and increasing intra-batch concurrency to amortize the overhead of reading weights are all essentially efforts to “feed” the bandwidth.

4. KV Cache: Why Session State is “Disposable”

The KV cache records the attention Keys/Values for each token in the context, enabling the Decode phase to avoid repeatedly recalculating historical parts. It is the key to enabling long-context conversations. However, its “identity” is subtle—it is merely a cache, not data that must be persisted.

This is fundamentally different from session states in traditional online services (like shopping carts or login states). Chat sessions have a strong “one-off” characteristic:

• “What’s the weather tomorrow?”—The session ends once answered, and the user is unlikely to return to continue it.
• “What news did a certain celebrity make yesterday?”—A typical single-turn factual query, same as above.
• Even for long-running, multi-turn conversations, users rarely iterate on the same session multiple times within a single day; most continuations happen days later.

The relatively low probability of users reusing a single session provides the system with massive optimization room: dropping the KV cache first during resource contention is practically a zero-cost capacity expansion method. If a user eventually decides to continue that conversation, the system must either reload the swapped-out cache from slower media like SSDs or re-run the Prefill process with the complete chat history. For a session that has been discarded for a long time, this cost is acceptable.

5. Engineering Trade-offs: Drop, Offload, or Recompute?

Regarding the retention of KV caches, inference services employ three primary strategies, each with a clear cost structure:

1. Drop: Immediately frees up VRAM with zero instant overhead; the cost is betting that the probability of a return visit is zero, requiring a full recomputation if the user comes back.
2. Offload: Writes the KV cache to an SSD or a remote memory pool, preserving the context for quick recovery when the user returns; the cost is IO bandwidth, storage overhead, and recovery latency.
3. Retain: Offers the fastest recovery, but VRAM usage persists continuously, directly squeezing the capacity available for new requests.

Choosing between the three is essentially an expected cost problem:

Retention Cost ≈ VRAM Unit Price × Residence Duration
Drop Cost ≈ P(User Return) × Prefill Recomputation Overhead
Offload Cost ≈ Write IO Overhead + P(Return) × Load IO Overhead

Because the probability of a session continuation decays rapidly over time, pragmatic eviction strategies typically use idle time (TTL) + recent usage as core signals:

# Simplified KV cache eviction strategy illustration
class KVCacheManager:
    def admit(self, new_request):
        # Trigger eviction when VRAM is insufficient to admit a new request
        while not self.enough_memory(new_request):
            victim = self.pick_victim()   # Priority: sessions that are idle beyond timeout and inactive
            if victim is None:
                return self.queue_or_reject(new_request)  # No evictable targets, queue or rate limit
            if victim.idle_time > self.TTL:
                self.offload_or_drop(victim)  # Low-value session: offload or drop directly
            self.free(victim)

Combined with Prefix Caching (calculating common context only once) and keeping system prompts resident in cache, the same hardware can support a higher effective concurrency.

6. Returning to the Announcement: Pausing Expansion is Sound Capacity Governance

Understanding the resource model above clarifies why “compute shortages” lead to “paused member registrations”:

1. Peak concurrency is a hard constraint. Platforms must reserve GPUs based on peak (not average) demand. Conversational traffic has a clear tidal pattern, and the massive gap between peaks and troughs means some GPUs have low utilization during off-peak hours—but they must exist to handle the peaks.
2. High-end GPU supply is tight, and scaling is not instantaneous. Mainstream cloud platforms, including Baidu AI Cloud, generally offer GPU cloud instances and elastic scaling capabilities to provision compute on demand. However, from resource availability to service deployment and stable operation, there is an irreducible lead time. For platforms experiencing rapid growth, “compute failing to keep up with user growth” is a common growing pain.
3. Existing user experience takes priority over incremental scale. When new users would dilute the response speed (lower throughput, longer queues) for existing users, restricting access is the most direct way to protect service quality. Pausing member registrations is essentially a commercial-level admission control, belonging to the same family of capacity governance mechanisms as system-level rate limiting, queuing, and graceful degradation.

7. Conclusion: The Long-Term Challenge Under Compute Constraints

From a longer-term perspective, “compute shortages” will not be a one-off event but will accompany the entire development cycle of LLM services. What engineers can do is continuously increase the effective output per unit of compute: using quantization and sparsification to reduce bandwidth pressure, speculative decoding and small model routing to lower per-request costs, dynamic batch scheduling to improve hardware utilization, and tiered cache systems (VRAM → Memory → SSD) to extend the “reusable window” of sessions. What the business side can do is honest capacity governance—when resources are genuinely insufficient, pausing access is far better than causing a collective degradation in user experience.

For developers, grasping three plain logic principles gives you the core framework: the number of GPUs determines peak concurrency; KV cache is disposable; calculating faster can compensate for less VRAM. Understanding these allows you to see right through the engineering essence behind any LLM platform’s rate limiting, queuing, and capacity announcements.