Capacity Planning for LLM Conversational Services Under Compute Constraints: From Peak Concurrency to Cache Trade-offs

1. The Core Problem: What Determines Conversational Service Capacity

In LLM-powered conversational services, the core resource determining “how many users can be served simultaneously” is typically neither network bandwidth nor disk capacity, but the compute and memory (VRAM) provided by accelerators. More precisely, it is the balance between the throughput capacity during the decoding phase and the number of concurrent sessions allowed by the VRAM capacity.

The lifecycle of a request can generally be divided into two phases: prefill and decode. The prefill phase processes the entire context of the user’s input, featuring high computation and high parallelism, making it compute-intensive. The decode phase generates output token by token; the per-step computation is small, but it requires repeatedly reading model weights and caches, making it memory-bandwidth-intensive. Setting aside the memory bandwidth bottleneck of the decode phase, we can approximate that the accelerator is working at full load near the bottleneck: the more tokens output per second, the faster a user’s query is completed, releasing its session cache to load the next user’s request and continue generation.

Therefore, a straightforward capacity model emerges:

Peak Concurrent Users ≈ Per-GPU Token Throughput per Second ÷ Token Consumption Rate per User per Second

The premise of this model is that every active session must occupy a session cache (i.e., the KV cache). The larger the cache, the fewer concurrent sessions a single GPU can hold; the more easily the cache is released, the more users can be rotated per unit of time.

2. Two Bottlenecks: Compute Throughput and VRAM Capacity

2.1 Bandwidth Constraints in the Decode Phase

Every time a token is generated in the decode phase, the model weights and the current session’s KV cache must be read. Model weights are a fixed overhead, while the cache grows linearly with the number of active sessions. As concurrent sessions increase, the amount of cache reading rises, and memory bandwidth quickly becomes a bottleneck. At this point, adding more compute power will not linearly increase throughput. This means that in a bandwidth-constrained zone, the primary way to increase concurrency is not stacking compute, but controlling the cache size per session.

2.2 Hard Constraints of VRAM Capacity

VRAM capacity determines the upper limit of concurrent sessions a single GPU can maintain. Expressed in a simplified formula:

Concurrent Sessions ≈ (Total VRAM - Model Weight Overhead - Runtime Overhead) ÷ Average Cache Usage per Session

The cache usage per session correlates with context length, number of layers, and attention head configuration. The longer the context, the larger the cache; the longer a multi-turn conversation, the more the cache accumulates. When the context length grows from a few thousand tokens to tens of thousands, the per-session cache can expand by an order of magnitude, directly compressing the number of concurrent sessions.

3. Session Cache Trade-offs: Why We Can “Drop First, Compute Later”

Conversational services have an important characteristic distinguishing them from other online inference scenarios: the session cache is disposable.

The reason lies in user behavior. The vast majority of sessions are one-off, such as “What’s the weather tomorrow?” or “What is the background of a certain news event?” Users ask, get an answer, and leave. Even in long-running conversations, users rarely iterate on the exact same conversation content multiple times within a day. The probability of users reusing the exact same session is relatively low.

This introduces an important engineering degree of freedom: to serve more users, you can actively release the caches of some sessions when resources are tight, freeing up VRAM for new requests. The trade-off is that when a user returns to an old session to ask a follow-up question later, the system needs to:

1. Reload the session’s history from persistent storage; or
2. Recompute the context from scratch with the full chat history.

Both methods incur one-time latency and compute overhead, but they avoid the long-term occupation of scarce VRAM. In engineering terms, this is shifting VRAM pressure to storage and compute pressure.

This scheduling strategy can be described with pseudocode:

def schedule_request(request, gpu_pool):
    if gpu_pool.free_memory() < request.required_cache:
        # Insufficient VRAM, evict caches of some low-activity sessions based on policy
        victims = select_eviction_candidates(
            policy="least_recently_used",
            prefer="single_turn_sessions"
        )
        for session in victims:
            persist_to_storage(session.cache)   # Persist to disk
            gpu_pool.release(session.cache)     # Release VRAM
    if gpu_pool.free_memory() < request.required_cache:
        return REJECT   # Still insufficient, trigger rate limiting or pause new users
    gpu_pool.allocate(request.cache)
    return ACCEPT

This logic explains the rationale behind “pausing new user registrations when compute is tight”: when eviction and disk persistence cannot free up enough VRAM, the system can only choose to reject new requests to ensure the experience of currently served users does not crash.

4. Capacity Assessment: A Practical Estimation Method

To move capacity planning from intuition to measurable metrics, you can establish a baseline following these steps.

Step 1: Measure per-GPU throughput. Under the target model and target context length, measure the number of tokens a single GPU can generate per second. Record data for both short-context and long-context scenarios separately, as the difference can be significant.

Step 2: Measure per-session cache usage. Measure the actual VRAM usage of a single session’s cache under different context lengths, and plot a “context length vs. cache usage” curve. This curve is a critical input for capacity estimation.

Step 3: Estimate peak concurrency. Combine total VRAM, model weight overhead, and runtime overhead to calculate the number of concurrent sessions. Then divide throughput by the per-user consumption rate, taking the smaller of the two as the peak concurrency limit.

Step 4: Set watermarks. Do not wait until VRAM is exhausted to evict. It is recommended to start eviction when VRAM usage reaches 75%–85%, reserving a buffer to handle sudden traffic spikes.

Step 5: Prepare degradation measures. These include: shortening the default context window, limiting the maximum turns per session, actively persisting low-activity sessions to disk, and enabling queuing and rate limiting during high-concurrency periods.

5. Degradation and Elasticity: Making “Pausing Access” a Controlled Operation

“Pausing new user access” is the final gate for capacity protection. A more mature approach is to integrate it into a tiered degradation strategy:

• Tier 1 Degradation: Actively evict low-activity session caches, sacrificing a small amount of traceback latency in exchange for concurrency space.
• Tier 2 Degradation: Limit the default context length for new sessions, or queue requests with ultra-long contexts.
• Tier 3 Degradation: Rate limit or queue new users, prioritizing the experience of existing active or paid users.
• Tier 4 Degradation: Pause new user access, concentrating resources on serving existing users.

Each tier should be accompanied by clear trigger metrics, such as VRAM watermarks, decode queue length, or time-to-first-token latency percentiles. Metric-driven degradation is more timely than manual judgment and easier to review post-mortem.

In the long run, there are several conventional paths to alleviate capacity pressure: increasing per-GPU throughput (quantization, batching optimization, more efficient attention implementations), reducing per-session cache (cache quantization, hierarchical caching, on-demand loading), and flattening peak-to-trough differences through elastic scaling. However, these methods all have boundaries. When the growth rate of demand exceeds the growth rate of resources, capacity constraints will still manifest in some form—be it queuing, degradation, or pausing new users.

6. Conclusion

The capacity problem of conversational services is essentially a joint optimization problem of compute throughput, VRAM capacity, and session cache strategy. Understanding that “how many tokens can be output per second determines how many users can be rotated,” that “VRAM determines how many sessions can be maintained simultaneously,” and that “caches are disposable so traceback latency can be sacrificed for concurrency,” explains why a seemingly simple “pause new user access” announcement is actually an engineering signal that an entire resource model has reached its critical point. For developers, turning capacity assessment into a measurable, degradable, and reviewable process is more valuable than pursuing single-point perfection.