Capacity Planning for Conversational AI Services Under Compute Constraints

1. Where the Problem Starts: What Determines Peak Concurrency in Conversational Services

In conversational AI services, the core resource that determines how many users a system can serve simultaneously is typically GPU compute. More precisely, it is the throughput capacity of the GPU during the decode phase.

A typical request processing flow can be abstracted as follows:

Receive user request → Assemble context → Prefill → Token-by-token decode → Complete output → Release resources → Load next request

In this pipeline, the decode phase proceeds token by token. Each generated token requires a forward pass. GPU compute determines how many tokens can be generated per second, while memory bandwidth determines how quickly model weights and the KV Cache can be read during each forward pass.

As the GPU approaches full utilization, overall system throughput converges toward an upper bound. This bound maps directly to: the number of active sessions the system can maintain simultaneously under a given latency constraint.

In other words:

Peak concurrent users ≈ GPU decode throughput ÷ Average token consumption rate per user

This is a rough but effective estimation framework. It explains why service providers choose to limit new user access when compute resources are insufficient—because each additional active session divides a token generation budget that is already constrained.

2. KV Cache: The Regulator Between Memory and Throughput

2.1 What KV Cache Is

In autoregressive generation with the Transformer architecture, the system caches the Key and Value matrices of historical tokens to avoid recomputing them for every new token. This is the KV Cache. It significantly reduces redundant computation for long conversations, but at the cost of GPU memory.

The size of the KV Cache is proportional to the following factors:

• Sequence length (more conversation turns means a larger cache)
• Number of model layers
• Number of attention heads and head dimension
• Batch size (number of requests processed simultaneously)

2.2 Engineering Choices When Memory Is Insufficient

When GPU memory cannot hold the KV Cache for all active sessions simultaneously, there are several engineering paths:

Path 1: Discard KV Cache and Recompute on Demand

This is a core strategy mentioned in the original text. An important characteristic of conversational AI is that a large number of sessions are one-off. A user asks a question such as “What will the weather be like tomorrow” or “What is the background of this event,” receives an answer, and then leaves. It is rare for users to repeatedly iterate on the same conversation within a short period.

Therefore, when memory is tight, the system can proactively discard the KV Cache for some sessions and use the freed memory for new requests. If that user speaks again later, the system has two options:

1. Reload the historical KV Cache from external storage (such as an SSD);
2. Give up on the cache entirely, reassemble the full conversation history into the context, and perform a complete prefill from scratch.

Option 2 incurs the cost of an additional prefill computation, but it is simple to implement and does not depend on the latency of external storage. For scenarios dominated by one-off sessions, the expected waste of this strategy is minimal.

Path 2: Tiered Caching and Eviction Policies

A more refined approach is to introduce a multi-level cache:

• GPU memory: holds the KV Cache for the most active sessions;
• Host memory: holds sessions that may be reused in the near future;
• SSD/object storage: holds serialized caches for cold sessions.

Eviction policies can be based on LRU (Least Recently Used), TTL (Time To Live), or session activity scoring. The core objective is to maximize the computational savings from cache hits under memory constraints.

Path 3: Accept Recompute Costs and Optimize Scheduling

When the cache hit rate is inherently low, it may be better to accept recompute costs directly rather than invest effort in maintaining a complex cache hierarchy. Optimization efforts can then focus on scheduling:

• Prioritize short requests to release resources quickly;
• Assign higher priority or longer cache retention to long conversations;
• Implement admission control in the request queue to avoid overload.

2.3 A Simplified Capacity Model

Assume:

• Single-GPU decode throughput is T tokens/s;
• The average token generation rate per active session is r tokens/s;
• The maximum number of concurrent sessions allowed by the system is N;

Then in steady state, the following must hold:

N × r ≤ T

That is:

N ≤ T / r

As N approaches T/r, GPU utilization approaches saturation, and queueing delays for new requests rise significantly. At this point, limiting new user access is an admission control measure that protects the experience of existing users.

3. Session Lifecycle Management: An Overlooked Capacity Lever

3.1 The One-Off Nature of Sessions

The session distribution in conversational AI is typically highly skewed:

• A large number of sessions consist of only 1–3 turns;
• A small number of sessions extend to dozens of turns or more;
• Very few sessions are resumed by users hours or days later.

This means that if the system retains KV Cache for every session over the long term, GPU memory will be occupied by a large number of “zombie sessions” that may never be used again.

3.2 Proactive Expiration and Passive Eviction

A practical strategy is to set an idle expiration time for each session’s KV Cache:

If session idle time > TTL:
    Evict the session's KV Cache
    Release GPU memory

The choice of TTL requires balancing:

• TTL too short: frequent recomputation wastes compute;
• TTL too long: cold sessions occupy memory and reduce concurrent capacity.

In real systems, the TTL can be dynamically adjusted based on historical statistics, or different TTLs can be applied to different types of sessions.

3.3 Balancing Recompute Cost Against Cache Benefit

The cost of recomputing a session is roughly equivalent to performing a prefill over its full history. For a context of length L, the prefill computation is approximately O(L²) due to the attention mechanism, while decoding a single token is approximately O(L).

Therefore:

• Short sessions have low recompute costs, and discarding their caches is nearly imperceptible;
• Long sessions have high recompute costs and should preferentially retain their caches or load them from external storage.

A reasonable policy would be:

if session history length < threshold:
    Allow direct eviction and recompute on demand
else:
    Preferentially retain the cache, or serialize it to external storage

4. Scheduling and Admission Control in Engineering Practice

4.1 Request Queues and Priorities

When GPU resources are constrained, all requests enter a queue and wait. The scheduler must decide:

• Which requests to process first;
• Which requests can be delayed;
• Which requests should be rejected or degraded.

Common priority dimensions include:

• Paid users vs. free users;
• Short requests vs. long requests;
• Interactive requests vs. batch processing requests.

4.2 Admission Control

When system load exceeds a safe threshold, proactively rejecting new requests or limiting new user signups is a protective measure. Its goals are:

• Preventing unbounded queue growth that leads to timeouts;
• Guaranteeing response quality for users who are already connected;
• Buying time for system scaling.

From a user experience perspective, a clear “temporarily unavailable” message is more manageable than being admitted and then facing a long wait.

4.3 The Limits of Elastic Scaling

In theory, adding GPUs can linearly increase throughput. In practice, however:

• GPU procurement and deployment take time;
• The parallel efficiency of inference frameworks may decline as scale grows;
• Bottlenecks such as memory bandwidth may appear before compute becomes the limiting factor.

Therefore, capacity planning cannot rely solely on scaling. Continuous optimization of caching strategies and scheduling efficiency at the software level is also necessary.

5. Summary and Implications

The capacity bottleneck of conversational AI services is fundamentally a dual constraint of GPU decode throughput and memory resources. As the primary consumer of GPU memory, KV Cache management directly affects the peak concurrency that a system can support.

The core conclusions can be summarized as follows:

1. Peak concurrency is determined by GPU decode throughput. When memory is insufficient, discarding caches and recomputing on demand can help alleviate pressure;
2. The one-off nature of sessions makes cache eviction costs manageable. Proactive expiration is an effective capacity lever;
3. Long sessions require differentiated treatment. Preferentially retain their caches or serialize them to external storage;
4. Admission control is a necessary protective mechanism. Limiting new user access when compute is tight is a reasonable choice to safeguard overall service quality.

For developers, understanding this set of trade-offs helps in making more informed resource planning and architectural decisions when designing and operating conversational AI systems.