LLM Inference Under Load: Resource Scheduling and Capacity Planning for Large Model Services

Why “Compute Under Strain” Is Really a Capacity Signal

When a conversational LLM product announces that it is pausing new subscriptions due to compute shortages, the technical community rarely finds this surprising. For most subscription-based or free-tier dialogue services, such an announcement is primarily a capacity management action rather than a purely commercial decision. It indicates that peak concurrent users have approached or reached the limit of the available compute pool.

The key point is not about any single product’s operating strategy. It is about a general engineering question: for an online conversational LLM service, what resource actually determines how many users can be served at peak concurrency?

The answer is usually inference compute, and more specifically the GPU resources responsible for the decode stage. This article starts from the resource model of inference serving and explains why concurrency limits emerge, how KV Cache trade-offs work, and what scheduling and scaling strategies are available in practice.

---

Peak Concurrency and the Resource Model: Throughput Is Capacity

The Lifecycle of a Single Request

In a typical conversational inference service, a user request goes through the following stages:

1. Admission and queuing: the request enters a gateway and is placed in a scheduling queue.
2. Prefill: the user prompt is fed into the model at once, generating the KV Cache.
3. Decode: tokens are generated one by one through autoregressive decoding.
4. Streaming response: tokens are pushed to the client as they are generated.
5. Resource release: the request ends, and its KV Cache and related GPU memory are freed.

For a service dominated by streaming output, stage 3 is the longest and most compute-intensive phase. Every generated token requires a full forward pass. Although only one new token is processed per decoding step, the model still reads all weights and the current KV Cache.

Why Tokens per Second Determines Concurrency

Assume a single GPU can produce T tokens per second during decoding, and an active session needs on average r tokens per second for the user to perceive the response as smooth. The theoretical number of active sessions a single GPU can support is approximately:

max concurrency ≈ T / r

This is a rough but directionally correct estimate. It highlights two facts:

• Concurrency limits are determined by total compute capacity, not by registered user counts or network bandwidth.
• As long as a user request can be completed within a given time window, the resources can be released and assigned to the next user.

In other words, conversational service capacity is fundamentally a throughput scheduling problem. The number of GPUs determines the sustained token output capacity. Dividing that output capacity by the minimum acceptable per-session throughput gives the peak concurrency.

There Is More Than One Bottleneck

The statement “ignoring the memory bandwidth bottleneck in the decode stage” is a necessary simplification. In real systems, decode-stage bottlenecks typically include:

Bottleneck	Description	Impact
GPU memory bandwidth	Every decoding step reads all weights and KV Cache	Limits token generation speed per GPU
GPU memory capacity	KV Cache grows linearly with context length	Limits the number of resident sessions
Compute utilization	Small batches lead to underutilized compute units	Limits total throughput
Scheduling and queuing	Uneven requests and long-tail jobs accumulate	Reduces effective throughput

Therefore, “the number of GPUs determines peak concurrency” is a correct high-level conclusion, but practical engineering still requires careful trade-offs among memory capacity, context length, and throughput.

---

KV Cache: A Cache That Can Be Sacrificed

What KV Cache Actually Is

During autoregressive generation, to avoid recomputing attention over all previous tokens at every step, inference frameworks cache the Key and Value tensors of each layer. This is the KV Cache. Its memory usage can be approximated as:

KV Cache ≈ 2 × layers × attention heads × head dimension × sequence length × precision bytes

It is directly proportional to sequence length. Longer conversations consume more KV Cache, and more concurrent sessions consume correspondingly more memory. With fixed GPU memory capacity, KV Cache directly competes with the number of sessions that can reside in memory at the same time.

Why It Can Be Dropped First

An important engineering insight is that KV Cache in conversational scenarios is highly expendable.

This follows from user behavior patterns:

• Many sessions are one-off, such as “What will the weather be tomorrow?” or “What did this public figure do yesterday?”
• Even in longer conversations, users rarely iterate repeatedly on the same context within a very short period.
• When users return to an older session, they can usually tolerate recomputation or reloading from persistent storage.

This means the server does not need to retain KV Cache for every session throughout its lifetime. When GPU memory is under pressure, the system can proactively evict caches for some sessions to admit more concurrent users. If an evicted session becomes active again, the system can either:

1. Load historical KV data from external storage such as SSD, if it was persisted.
2. Re-run the full conversation history through a new Prefill.

Both approaches introduce additional latency or compute overhead, but compared with rejecting new users entirely, they are better degradation paths.

Engineering Trade-offs in Eviction Policies

In real systems, KV Cache management usually combines several strategies:

• LRU / LFU eviction: prioritizes retaining recently active or frequently used sessions.
• Tiered storage: keeps hot data in GPU memory, moves warm data to host memory, and cold data to SSD.
• Session reuse detection: identifies short versus long conversations and gives longer conversations higher residency priority.
• Preemptive scheduling: when a new request arrives and memory is insufficient, the cache of a lower-priority session is evicted.

The point is not that any single policy is universally optimal. The goal is to find a balance among throughput, latency, and user experience. Dropping cache comes at the cost of recomputation, and recomputation consumes compute. But when compute is already scarce, sacrificing some experience to preserve overall availability is often the more reasonable choice.

---

From Resource Model to Capacity Planning: Practical Design Considerations

Once these principles are understood, it becomes easier to design more reasonable capacity and elasticity plans for self-hosted inference services.

Separate the Admission Layer from the Inference Layer

Decouple request admission, authentication, and rate limiting from inference execution. The admission layer can enforce admission control based on queue depth, GPU utilization, and other metrics, preventing all requests from directly overwhelming the inference cluster. A common approach is to use a message queue or scheduling queue as a buffer, with the inference layer pulling tasks according to actual throughput.

Define Explicit Service Levels

Different users or scenarios can be assigned different priorities:

• High priority: lower queueing time and longer context residency.
• Normal priority: allowed to queue, with KV Cache eviction permitted.
• Low priority: processed only when resources are idle, or treated as best-effort.

This makes it possible to degrade low-priority requests during compute shortages instead of stopping all new users indiscriminately.

Observability and Scaling

Capacity management requires observability. At minimum, the system should collect:

• Decode throughput per GPU, in tokens/s.
• GPU memory usage and KV Cache distribution.
• Request queueing time and P99 latency.
• Frequency of eviction and recomputation events.

Only with these metrics can teams determine whether the system is truly compute-constrained and identify the threshold for scaling. Containerization and elastic scheduling platforms can complete scale-out actions within minutes, but GPU procurement and deployment cycles may still delay immediate expansion. This is why temporarily limiting new subscriptions is sometimes a necessary stopgap.

Request-Side Optimizations

Beyond the server side, there is also room for optimization on the request side:

• Control context length: avoid unnecessary concatenation of long histories.
• Set reasonable maximum output lengths: prevent extremely long generations from occupying decode resources for extended periods.
• Use streaming consumption: clients should read tokens promptly to avoid hanging connections that hold scheduling resources.

These measures cannot fundamentally solve a compute shortage, but they can significantly improve the effective serving capacity of existing compute.

---

Conclusion: Compute Shortages Are the Norm, Not an Exception

The core tension in conversational LLM services is between user expectations for instant responses and long contexts on one side, and limited GPU throughput and memory capacity on the other. KV Cache can be dropped, requests can be queued, and sessions can be recomputed. These are all valid engineering measures, but they all have costs.

When a service pauses new subscriptions, the signal is that expanding concurrency under the current resource pool would harm the core experience of existing users. Rather than degrading the experience for everyone, the service controls new demand first. This is pragmatic capacity management, not a technical failure.

For developers, the more valuable lesson is to treat compute as a core constraint from the early stages of system design. Establish a clear resource model, priority system, and degradation path. That way, when traffic rises or resources become constrained, the system can respond in a controlled manner instead of being forced into binary choices.