From Compute Bottlenecks to Service Tiering: A Technical Review of LLM Inference Capacity Planning

1. The Technical Reality Behind the Incident

When a conversational LLM service experiences user growth far beyond expectations in a short period, the first ceiling it hits is often not model capability itself, but inference-side compute supply. The symptoms are concrete: new users cannot subscribe, some features enter a queue, and response latency rises. The underlying issue is a classic capacity planning problem—given a fixed pool of GPU resources, how many concurrent requests can the system reliably serve per second.

Understanding this requires going back to the basic flow of autoregressive inference.

2. The Two Phases of Inference and the Resource Bottleneck

A single LLM request is roughly divided into two phases:

Prefill phase: The full user prompt is fed into the model, and Key/Value tensors for each layer are computed and cached. This phase is compute-intensive, with latency approximately proportional to input length.

Decode phase: Output tokens are generated one at a time. Each generated token requires reading the KV cache of all previous tokens. This phase is memory-bandwidth-intensive—GPU floating-point compute is often largely idle, and the real bottleneck is whether memory bandwidth can feed the KV cache to compute units in time.

A simplified timing model for GPU occupancy per request:

T_total ≈ T_prefill(input_len) + T_decode(output_len, kv_cache_size)

The memory-bandwidth dependency of T_decode can be approximated as:

T_per_token ≈ (2 × n_layers × d_model × bytes_per_element × seq_len) / memory_bandwidth

This means: the longer the sequence and the more concurrent requests, the greater the memory bandwidth pressure, and the slower the decode speed. When concurrent requests exceed a certain threshold, per-token latency begins to rise non-linearly.

3. Engineering Estimation of Concurrent User Limits

Assume an inference cluster has N accelerators, each with M GB of memory, model weights occupy W GB, and the average KV cache per request is K GB. The theoretical number of requests that can reside simultaneously is:

max_concurrent ≈ N × (M - W) / K

This number looks straightforward, but real systems have several correction factors:

• Memory fragmentation and reservation: Frameworks typically reserve a portion of memory for temporary activations and communication buffers, so actual available memory is lower than the theoretical value.
• Trade-off between batch efficiency and latency: Continuous batching can improve throughput, but excessively large batches increase per-request queueing time.
• Blocking effect of long-tail requests: A small number of very long conversations occupy KV cache for extended periods, preventing short requests from being scheduled in time.

Therefore, real capacity planning cannot just ask “how many requests can fit” but “how many requests can be reliably served under an acceptable latency SLA.”

4. KV Cache Trade-off Strategies

A key engineering decision mentioned in the original discussion: when resources are tight, KV cache can be proactively discarded, forcing the next conversation to recompute historical context. This is a classic space-for-time strategy.

Specifically, KV cache management has several common approaches:

Strategy 1: Full residency. Keep KV cache for all active sessions in GPU memory. The advantage is the best continuation experience; the disadvantage is high memory usage and a low concurrency ceiling.

Strategy 2: Tiered storage. Swap out KV cache for inactive sessions to host memory or high-speed external storage, loading it back into GPU memory when needed. This requires additional transfer overhead but significantly increases concurrent capacity.

Strategy 3: Discard and recompute. Directly release KV cache that has not been accessed for a long time, and re-prefill from the full history when the user returns. For one-shot Q&A scenarios, this is the lowest-cost option.

In engineering practice, most systems combine these strategies: hot sessions reside in GPU memory, warm sessions are swapped to host memory, and cold sessions are discarded. The key lies in the eviction algorithm that determines session activity—commonly LRU, LFU, and time-decay-based weighted strategies.

5. Service Tiering: An Inevitable Choice Under Compute Scarcity

When compute supply cannot meet all demand, service tiering becomes inevitable. Common tiering dimensions include:

Tiering by subscription level. Paying users get priority access to compute; free users queue or are downgraded. This is the most direct commercial strategy but requires product-side coordination.

Tiering by request type. Short requests are scheduled first; long requests enter a lower-priority queue. This protects the interactive experience for most users.

Tiering by time window. New session creation is restricted during peak hours and opened up during off-peak hours. This requires accurate traffic forecasting.

Tiering by model specification. Under resource pressure, some requests are routed to smaller or quantized models, trading some quality for throughput.

These strategies are typically implemented through schedulers and admission control. A simplified admission control logic:

def admit_request(request, current_load, user_tier):
    if current_load > HARD_LIMIT:
        return REJECT
    if user_tier == FREE and current_load > SOFT_LIMIT:
        return QUEUE
    if request.estimated_kv_size > AVAILABLE_MEMORY:
        return DOWNGRADE  # route to a smaller model or require shorter context
    return ADMIT

6. Building Observability for Capacity Planning

To avoid “suddenly discovering that compute is insufficient,” a comprehensive capacity observability system is needed. Core metrics include:

• GPU utilization and memory usage: Distinguish between compute utilization and bandwidth utilization—the latter is often the real bottleneck.
• P50/P95/P99 per-token latency: Latency distribution reflects system health better than averages.
• Queue time and rejection rate: Directly reflects the degree of user experience degradation.
• KV cache hit rate and swap-out frequency: Measures the effectiveness of caching strategy.
• Average token consumption and session duration per user: Used to predict growth trends.

Based on these metrics, a capacity watermark model can be established: when memory usage consistently exceeds 70% and P99 latency exceeds a threshold, scaling or degradation plans are triggered.

7. Implications for Developers

For developers building LLM applications on the cloud, this incident offers several practical recommendations:

First, do not assume inference services will always be available. When designing systems, implement retry, degradation, and circuit-breaking mechanisms. When upstream returns rate-limiting or queueing signals, the application layer should have clear user notifications and alternative paths.

Second, distinguish critical requests from non-critical ones. Allocate core business requests and auxiliary requests (such as summarization or rewriting) to different calling channels to ensure the stability of critical paths.

Third, control context length. Long contexts not only increase per-request cost but also occupy inference resources for extended periods. Context trimming and summary compression at the application layer are effective ways to reduce compute pressure.

Fourth, monitor the link between usage and cost. Establish token-level metering and budget alerts to avoid cost overruns caused by sudden traffic spikes.

Fifth, design for elasticity at peak. If the business has clear peak-valley characteristics, consider supporting multi-model routing at the architecture level—using smaller, faster models during peak hours and full-capability models during off-peak hours.

8. Conclusion

Compute scarcity is not an isolated problem for any single provider but a structural challenge in the commercialization of large models. From a technical perspective, it involves engineering trade-offs across inference optimization, cache management, scheduling strategies, and capacity planning. Understanding these trade-offs not only helps developers better use external services but also provides reference for building self-hosted inference systems.

Ultimately, a system that can reliably serve under compute constraints depends not on a single technical breakthrough, but on continuous fine-grained balancing of resources, latency, cost, and user experience.