Capacity Planning and Scheduling Strategies for Chat Services Under Compute Bottlenecks

I. The Core Problem: What Determines Peak Concurrency?

In public-facing conversational AI systems, the factor that truly dictates “how many users can be served simultaneously” is typically not the model’s parameter size or the API layer’s concurrency framework. Rather, it is the total underlying inference compute capacity. More specifically, it is the volume of tokens the GPU cluster can generate per unit of time.

Understanding this requires a basic queuing model perspective: upon arrival, each user request goes through prefill and decode stages. The system streams the generated tokens back to the user until the request is completed or interrupted. The resources occupied by a request are essentially a GPU compute time and memory bandwidth over a specific time window. Therefore:

• Total compute per unit time ≈ Number of GPUs × Effective throughput per card
• Peak concurrent users ≈ Total compute per unit time ÷ Average resource consumption per request

This means that when total compute is fixed, increasing concurrency capacity requires focusing on two directions: reducing the average resource consumption per request and improving resource reuse efficiency.

II. The True Bottleneck in the Decode Stage

Many mistakenly believe that the inference bottleneck lies in peak compute (FLOPS). However, during the autoregressive decode stage, the actual limit is often memory bandwidth.

To generate each token during decoding, the system must read the model weights and the request’s KV Cache. While the computational volume for a single token is small, it requires frequent memory access. When the batch size is relatively small, the GPU’s compute units remain largely idle, and the system is effectively bottlenecked by memory bandwidth. This is known as the “memory bandwidth wall.”

This leads to a key conclusion: during the decode stage, the GPU is often already operating at near-full load. Continuously increasing concurrency will not linearly improve throughput. Instead, it will lead to requests being rejected or queued due to insufficient memory.

Therefore, the core issue of capacity planning shifts to: given limited memory and bandwidth, how can we sequence the loading, execution, and unloading of requests to ensure the GPU rarely sits idle?

III. KV Cache: A Disposable “Temporary Asset”

To understand high-concurrency scheduling, it is essential to grasp the positioning of the KV Cache.

The KV Cache is an intermediate result stored in memory during decoding to avoid recalculating attention keys and values for historical tokens. Its characteristics are:

1. Size grows linearly with context length: The longer the context, the more memory it occupies.
2. Lifecycle is bound to the request: Theoretically, it can be released once the request ends.
3. Reconstructible: Even if discarded, it can be recovered by recomputing the historical context.

The third point is crucial. It means the KV Cache is not a “state” that must be persistently saved, but rather a performance optimization that can be sacrificed at any time. This opens up possibilities for scheduling strategies:

• When memory is tight, prioritize evicting session caches that will not be reused in the short term.
• When a user initiates the same session again, either reload it from external storage (like an SSD) or simply recompute the historical context from scratch.

IV. Why “Discarding Cache” is Acceptable in Practice

Looking at user behavior, conversational services exhibit distinct “one-off” characteristics:

• A massive number of requests are short, independent queries, such as “What is the weather tomorrow?” or “What are the details of this news?”
• Even in long conversations, users rarely iterate on the exact same context repeatedly throughout the day.
• Overall session reuse rates are low, meaning the benefits derived from cache hits are limited.

This creates an engineering trade-off: sacrificing a small amount of redundant computation in exchange for higher concurrency capacity.

Specifically, the following strategy can be employed:

# Pseudocode: Cache eviction strategy when memory is tight
def schedule_request(request, gpu_memory):
    if gpu_memory.is_tight():
        # Prioritize evicting the least recently used session cache that won't be reused shortly
        victim = select_lru_session(exclude=request.session_id)
        gpu_memory.evict(victim)
    gpu_memory.load(request)
    run_decode(request)

When an evicted session returns, the system has two options:

1. Restore from external storage: Read the previously serialized KV Cache back from the SSD into memory. The cost is primarily I/O.
2. Recompute: Re-run the prefill process with the complete chat history. The cost is additional compute consumption.

Both methods have their applicable scenarios. The former suits sessions with extremely long contexts where recomputation is costly. The latter suits shorter contexts where I/O might actually be slower.

V. Key Variables in Capacity Planning

In actual system design, the following variables must be monitored simultaneously:

Variable	Description	Impact on Concurrency
Effective throughput per card	Tokens generated per unit time	Positive correlation
Average context length	Average token size per request	Negative correlation
Average generation length	Tokens returned per request	Negative correlation
Caching strategy	Whether to retain, and for how long	Affects reuse rate
Eviction and reconstruction cost	Recomputation or I/O overhead	Affects effective throughput

A common pitfall is focusing solely on the number of GPUs while ignoring the distribution of context lengths. In reality, long-context requests can consume tens of times more memory than short requests. Without differentiation, a small number of long requests can squeeze out a large portion of the concurrency quota.

VI. Actionable Scheduling and Optimization Recommendations

1. Tiered Scheduling

Classify requests based on context length and expected generation length. Schedule short requests first, and route long requests to a relatively independent queue to prevent mutual interference.

2. Tiered Caching

Use GPU memory as L1 cache, and SSD or object storage as L2 cache. Retain high-value sessions longer, and evict low-frequency sessions rapidly.

3. Dynamic Batching

Merge requests as much as possible within memory limits to improve GPU utilization. However, note that an excessively large batch can increase per-request latency due to memory bandwidth contention.

4. Rate Limiting and Queuing

When compute approaches its limit, rather than slowing down all requests, queue or rate-limit the excess traffic to guarantee the quality of service for accepted requests.

5. Capacity Water Level Monitoring

Continuously monitor memory usage, batch size, queue length, and token throughput. Establish water level alerts to scale out or degrade service proactively before hitting the bottleneck.

VII. Conclusion

The peak concurrency capacity of conversational services is fundamentally a comprehensive issue of compute, memory, and scheduling strategies. The disposable nature of the KV Cache provides a flexible adjustment mechanism: under resource constraints, trading some cache reuse for a small amount of recomputation cost can yield higher concurrency support. For developers, understanding this trade-off and designing tiered scheduling, tiered caching, and rate-limiting mechanisms accordingly is key to stably supporting more users under finite compute. Compute scarcity is not frightening; what is frightening is the lack of a matching scheduling strategy.