The Technical Roots of LLM Inference Compute Shortages and How to Address Them

1. The Essence of Online Inference Capacity

Recently, following the release of a new model version, a major LLM conversational service announced that due to compute shortages, it was suspending new paid member subscriptions. Existing users could continue to use the service, renew, or upgrade, but the availability of new subscriptions was postponed.

Setting aside business operations, from a technical perspective, this is a textbook example of an online inference capacity event. A paid membership is essentially a “compute commitment”: every new subscriber represents a new concurrent session requiring stable token throughput during peak hours. When the demand curve spikes sharply and compute supply cannot scale synchronously, throttling new allocations to preserve the existing user experience is the most direct and responsible capacity management approach. The strategy of “preserving existing load, controlling incremental growth, and gradually scaling” is a standard peak-shaving and smoothing tactic in the online services industry, not merely a PR narrative.

So, where exactly is the compute bottleneck in LLM inference? Why is adding more GPUs never enough? Let’s break it down.

2. The Two Phases of Inference: Why the Bottleneck is Always in Decode

The autoregressive inference of large language models is divided into two engineering phases, each with a completely different resource profile:

• Prefill Phase: The entire prompt is read at once, allowing all tokens to be computed in a highly parallel manner. Matrix multiplications are dense, and the GPU’s floating-point compute units are fully utilized. This is a compute-bound workload.
• Decode Phase: Tokens are generated one by one. For every token generated, the entire model weights and the accumulated KV Cache must be transferred from VRAM. The computational intensity is low, but the memory access volume is massive. This is a memory-bound workload.

Using arithmetic intensity (FLOPs/Byte) as a metric: the effective computation per step in the decode phase is relatively fixed, but the data volume that needs to be moved scales linearly with model size. Its arithmetic intensity is far below the “balance point” required by the GPU’s peak compute capacity. In other words, during the decode phase, the GPU spends most of its time waiting for data rather than computing data.

A rough estimation illustrates this: assuming a 10-billion parameter model deployed in FP16 precision, the weights occupy about 20GB of VRAM. If the inference card has a memory bandwidth of 3TB/s, the theoretical decode speed limit for a single replica without batching is about 1,500 tokens/s (20GB ÷ 3TB/s ≈ 67ms/step). This intuitively shows that the key to increasing throughput in the decode phase is not stacking more floating-point compute power, but improving memory bandwidth utilization and batch size—amortizing a single weight read across more concurrent requests.

• With batching, multiple requests share the same weight read, reducing the bandwidth cost per request and potentially multiplying the total throughput of a single card.
• However, as the batch size grows, the KV Cache consumes more VRAM, which in turn limits the maximum concurrency.

This is the core contradiction in inference capacity planning: bandwidth determines how fast tokens are computed, and VRAM determines how many users can be served simultaneously.

3. KV Cache: The “Invisible Giant” in VRAM

Transformer autoregressive inference requires caching the Key/Value vectors for every layer’s attention. The VRAM footprint of a single token’s KV Cache can be estimated using an industry-standard formula:

KV_per_token = 2 × num_layers × num_kv_heads × head_dim × precision_bytes

Taking a hypothetical model as an example: 40 layers, 8 KV heads, 128 dimensions per head, and FP16 precision. The KV footprint per token would be:

2 × 40 × 8 × 128 × 2B = 160KB/token

A long conversation with a 32K context length would require roughly 5GB just for the KV Cache. Assuming a single card has 40GB of VRAM allocated for KV, it can theoretically only maintain 8 such long conversations simultaneously. Short conversations consume less, but given the massive volume of chat requests, the KV Cache VRAM is still rapidly filled during peak hours.

This also explains two industry phenomena:

1. Why services experience “compute shortages” more easily after releasing long-context capabilities: Doubling the context length doubles the KV Cache footprint per session, instantly halving the number of concurrent sessions the same hardware can maintain.
2. Why the industry is heavily investing in KV Cache compression: Techniques like low-precision quantization, sparsification, prefix caching, and paged management are essentially making room for this “invisible giant.”

4. Session Discardability: The Scheduling Art of Chat Scenarios

A highly valuable engineering perspective from this event is worth expanding upon: the KV Cache of a chatbot is discardable. This is based on the statistical characteristics of user behavior:

• A large number of sessions are “one-off”—asking about the weather or a general fact, and rarely continued afterward.
• Even in long-running conversations, users rarely iterate on the same context repeatedly within a single day.
• The proportion of high-value sessions that truly require long-term context retention is limited.

Therefore, mature inference systems adopt tiered scheduling strategies:

1. Active Sessions: The KV Cache resides in VRAM to ensure low-latency responses.
2. Idle Sessions: Evicted from VRAM based on LRU (Least Recently Used) or LFU (Least Frequently Used) strategies, or offloaded to SSDs or CPU memory for cold storage.
3. Returning Sessions: Prioritized for reload from cold storage; if the cold data has expired or reloading is too costly, the system re-runs the prefill phase with the complete chat history, trading one-off compute overhead for VRAM turnover.

This tradeoff is cost-effective in most scenarios: for high-traffic services, VRAM is the scarcest resource during peak times, while re-prefilling merely shifts the cost from “residency” to “replay,” and the majority of sessions will never be replayed anyway. Of course, this has its costs: higher latency for returning users, extra compute consumed by repeated prefilling, and SSD bandwidth becoming a new contention point. The scheduling system must dynamically adjust eviction thresholds based on session length distribution, return rates, and request arrival rates.

“Less VRAM usage, faster computation, and the next user’s request can be loaded”—the essence of this is: for chat workloads, the goal of system scheduling is to maximize the number of sessions per unit of VRAM, rather than allowing every session’s KV Cache to reside indefinitely.

5. Peak Users are Directly Proportional to Available GPUs

Tying the above analysis together yields a simple but practical capacity formula:

Peak Serviceable Concurrency ≈ Available KV VRAM ÷ Avg KV Footprint per Session × (1 / Safety Margin)
Where: Available KV VRAM = Total VRAM − Weight Footprint − Activations & Temp Buffers

This means that once the model architecture, precision, and context length are fixed, peak concurrency is almost entirely determined by the number of inference cards. Memory bandwidth dictates “how fast tokens are generated,” while the number of cards dictates “how many users can be served simultaneously.”

This also explains why new model releases, marketing campaigns, or viral social trends trigger compute shortages: demand is pulsed, while GPU procurement and deployment are long-cycle. The supply of high-end AI compute cards is further constrained by supply chain capacity and export controls, with expansion cycles measured in quarters or even years. Even with the budget to buy cards, they cannot be plugged in and used immediately, exacerbating the supply-demand mismatch. The gap in between can only be bridged by hard measures like rate limiting, queuing, improving batch efficiency, and aggressive KV eviction policies.

6. Implications for Developers and Enterprises

1. Calculate capacity requirements starting with the KV footprint. When evaluating any LLM service or building an in-house inference cluster, first estimate the KV footprint per session using the formula above, then multiply by the target concurrency to derive the VRAM budget. Many “concurrency bottlenecks” are caused not by a lack of compute, but by insufficient KV VRAM.
2. Use engineering techniques to reduce the per-token cost:
   • Inference-side: KV Cache quantization, prefix caching reuse, continuous batching, and Prefill-Decode (PD) disaggregation (deploying prefill and decode on separate instances for individual optimization).
   • Model-side: Adopt architectures like Grouped-Query Attention (GQA) to reduce KV heads, use speculative decoding, and control output length.
   • Scheduling-side: Session eviction and reloading, off-peak scheduling, and tiered priority guarantees for different users.
3. Align commercial strategies with compute supply. Memberships and free quotas are essentially “compute options.” The pacing of issuance must match the pacing of capacity expansion. Pulsed demand can be managed through queuing, rate limiting, and offline batch processing to shave peaks, avoiding commitments that exceed supply capabilities.
4. Keep infrastructure alternatives open. If the cost and operational burden of self-hosted inference are too high, consider managed LLM inference services or dedicated compute instances provided by cloud platforms (such as Baidu AI Cloud). Outsourcing bandwidth scheduling, elastic scaling, and disaster recovery to the infrastructure layer allows teams to focus on business logic and prompt engineering. Whether self-hosted or managed, the capacity estimation methods discussed above remain universally applicable.

7. Conclusion

“Compute shortages and suspended memberships” may appear as operational announcements on the surface, but underneath lies a complete inference capacity event. The memory bottleneck in the decode phase determines the throughput ceiling, the VRAM footprint of the KV Cache determines the concurrency ceiling, and the long procurement cycle of GPUs ensures that the gap cannot be bridged overnight. Understanding this capacity logic—the interplay between bandwidth, VRAM, concurrency, and eviction policies—provides direct value whether you are evaluating LLM services, planning an in-house inference cluster, or designing your cost structure. Compute can be bought, but capacity planning expertise must be cultivated in-house.