Capacity Planning for Conversational AI Services Under Compute Constraints: Starting From “Pausing New Signups”

1. Behind the Phenomenon: What Determines the Capacity Ceiling of a Conversational Service

When a conversational AI service announces that it is “pausing new user signups due to compute constraints,” most people’s first reaction is “not enough GPUs.” That direction is correct but too coarse. What really determines how many active users a conversational service can support concurrently is a complete capacity model built around GPU resources. Understanding this model is the key to understanding why some services remain available during peak hours while others have to proactively throttle traffic.

Start with a basic fact: inference for conversational services has two phases — prefill and decode. The prefill phase processes the full context of user input, with concentrated computation and high parallelism. The decode phase generates output token by token, with low compute density but sustained consumption of memory bandwidth. The two phases have different resource characteristics and constrain overall capacity in different ways.

During decode, the GPU bottleneck is usually not raw compute but memory bandwidth. Every generated token requires reading model weights and the KV cache from GPU memory. Model weights are fixed, while the KV cache grows linearly with the number of concurrent sessions. This leads to the core tension in capacity planning for conversational services: total GPU memory is limited, and every active session occupies its own copy of the KV cache.

2. KV Cache: The Underestimated Capacity Killer

KV cache (Key-Value cache) is a key mechanism in Transformer architectures that avoids recomputation during autoregressive generation. For each token processed, the model caches the corresponding Key and Value vectors for every layer. For a model with L layers, hidden dimension H, and FP16 precision, the KV cache size for a single token is approximately:

KV Cache per token ≈ 2 × L × H × 2 bytes

Taking a typical 70B-class model as an example, assuming 80 layers and a hidden dimension of 8192, the KV cache per token is about 2.6MB. If the average session context length is 2000 tokens, the KV cache for a single active session exceeds 5GB. When concurrent sessions reach a few dozen, the total KV cache easily exceeds what a single GPU’s memory can hold.

This explains why “pausing new user signups” is a reasonable engineering decision: when KV cache memory usage approaches the limit, continuing to accept new requests will slow down decoding for existing sessions and may even trigger OOM (out-of-memory) errors that crash the service. Rather than degrading the experience for all users, it is better to actively limit new arrivals and preserve the quality of existing service.

3. Why KV Cache Can Be Discarded

A key point in the original discussion is that a chatbot’s KV cache can be discarded. This judgment has engineering merit, but it needs to be discussed by scenario.

One-shot sessions account for a high proportion. Many user requests are “ask and leave” types, such as checking the weather, asking a factual question, or generating a short piece of text. These sessions have short contexts and few interaction turns, so the KV cache lifecycle is naturally short. Once the request ends, the corresponding cache can be released.

Long conversations have a low probability of reuse. Even in multi-turn conversations, users rarely return to the same session multiple times within a day to continue iterating. Most long conversations enter a “cold” state after they end. For such sessions, the system can choose to swap the KV cache out to external storage (such as SSD) and reload it when the user returns; or, more aggressively, discard the cache entirely and recompute from scratch on the next request.

The cost of recomputation is manageable. If a user returns to continue a conversation after a long interval, the system needs to re-run prefill over the historical conversation records. Although this costs more than reading the cache directly, given the low frequency of such occurrences, it is acceptable overall. Trading “occasional recomputation” for “higher concurrency capacity” is a good deal in most scenarios.

But there is an important boundary condition: if the service is positioned for long-range, high-frequency, deep-interaction scenarios (such as code assistants or complex task planning), the KV cache discard strategy needs to be more cautious. Frequent recomputation significantly increases time-to-first-token and hurts user experience. In such scenarios, a more reasonable approach is tiered caching: hot sessions stay in GPU memory, warm sessions are swapped out to high-speed external storage, and only cold sessions are fully discarded.

4. The Throughput-Concurrency Trade-off: Scheduling Strategy Determines Actual Capacity

With the premise that “KV cache can be discarded,” capacity planning for conversational services shifts from “how many sessions can fit at once” to “how to serve more requests per unit of time.” This involves the design of scheduling strategies.

Continuous batching is standard in modern inference engines. Traditional batching requires all requests in a batch to start and finish together, causing short requests to be dragged down by long ones. Continuous batching allows new requests to be dynamically added and completed requests to be removed during decoding, significantly improving GPU utilization.

Paged attention manages the KV cache in pages, avoiding memory fragmentation and improving memory utilization. Combined with prefix sharing, multiple requests that share the same system prompt can reuse the same KV cache, further reducing memory usage.

Preemption and recomputation are the last resort for handling memory pressure. When memory is insufficient, the scheduler can preempt the KV cache of low-priority sessions, swapping it out or discarding it to free space for new requests. Preempted sessions are recomputed the next time they are scheduled. The cost of this strategy is latency jitter, but it keeps the system from crashing under overload.

A simplified scheduling decision can be expressed in the following pseudocode:

def schedule_requests(waiting_queue, running_sessions, free_memory):
    while waiting_queue and free_memory > threshold:
        req = waiting_queue.pop()
        if req.kv_cache_size <= free_memory:
            allocate_kv_cache(req)
            running_sessions.add(req)
            free_memory -= req.kv_cache_size
        else:
            # Try to swap out a low-priority session
            victim = find_low_priority_session(running_sessions)
            if victim:
                evict_kv_cache(victim)  # Swap out to SSD or discard
                free_memory += victim.kv_cache_size
            else:
                break  # Cannot admit new requests; trigger rate limiting
    return running_sessions

This logic shows that rate limiting is not simply “rejection” but active management of request admission order under memory constraints.

5. Capacity Assessment Framework: From GPU Count to Serviceable Users

Answering the question “how many GPUs are needed to support X tens of thousands of users” requires a systematic estimation method. Here is an actionable assessment framework:

Step 1: Determine the resource profile of a single session. This includes average context length, average output length, session duration, and the proportion of concurrently active sessions. These parameters determine the KV cache usage and decode computation for a single session at peak moments.

Step 2: Measure the throughput of a single GPU. Under the target model and precision, measure the maximum number of concurrent sessions a single GPU can support while meeting SLAs (such as time-to-first-token < 1s and decode speed > 20 tokens/s). This number is usually much smaller than “the number of sessions that fit in memory,” because decode speed degrades as concurrency increases.

Step 3: Calculate the required number of GPUs. Required GPUs = peak concurrent sessions / concurrent sessions serviceable per GPU × redundancy factor. The redundancy factor is typically 1.3–1.5 to handle traffic fluctuations and failover.

Step 4: Design degradation and rate-limiting strategies. When actual load exceeds planned capacity, the system needs a clear degradation path: first reduce concurrency quotas for non-paying users, shorten KV cache retention time, enable more aggressive preemption, and only then trigger rate limiting for new users.

6. Engineering Practice Recommendations

For teams building their own conversational services, the following points deserve attention:

Monitor KV cache utilization as a core metric. Do not look only at GPU utilization. High GPU utilization with low KV cache usage means compute-intensive but still room for capacity; KV cache usage approaching the limit is the real capacity bottleneck signal.

Implement a tiered caching strategy. Divide the KV cache into a GPU memory tier, a host memory tier, and an external storage tier. Keep hot data in GPU memory, swap warm data to host memory, and persist cold data to disk. The timing and granularity of swapping need to be tuned to business characteristics.

Set reasonable session timeout and reclamation mechanisms. For sessions with no interaction for a long time, proactively reclaim their KV cache. The timeout should not be too short (which hurts user experience) or too long (which wastes memory). It can be dynamically adjusted based on the session’s historical interaction frequency.

Conduct stress tests for capacity planning. Before launch, use load-testing tools that simulate real traffic distributions to validate system behavior at target concurrency. Focus on P99 latency, error rates, and memory usage curves, not just average throughput.

Design graceful rate limiting and queuing mechanisms. When capacity is insufficient, rather than rejecting requests outright, let requests enter a queue and return an estimated wait time to the frontend. For paying users, a priority queue can be provided. Rate-limiting policies should be multi-level and configurable, not an all-or-nothing switch.

7. Conclusion

“Compute constraints, pausing new user signups” is not a simple operational announcement but an inevitable engineering choice for conversational services under capacity constraints. Understanding the memory usage patterns of the KV cache, mastering the scheduling principles of continuous batching and paged attention, and building an estimation framework from single-session profiles to GPU counts are foundational skills every developer building conversational services needs.

Compute is always scarce, but through reasonable architecture design and scheduling strategies, it is possible to serve more users with limited resources. The key is to treat capacity management as a continuous engineering optimization problem, not a temporary emergency measure.