Compute Bottlenecks and Concurrency Control in LLM Inference Services

1. Core Constraints: GPU Compute and Peak Concurrency

In large-scale chatbot services, a frequently overlooked yet critical fact is that the peak concurrent user limit is often determined by underlying GPU compute capacity, not the application layer architecture.

From a user’s perspective, a conversational request is simply “input a prompt, wait for the answer.” From the server’s perspective, however, this process involves several critical steps:

1. Receive user input and complete tokenization and encoding.
2. Load the input tensors into GPU memory (VRAM).
3. Execute forward inference on the model.
4. Generate output token-by-token during the decoding phase.
5. Write the generated tokens back to VRAM or cache.
6. Release the user session state from active resources after completing the response.

Step 4—the decoding phase—is a strictly sequential, token-by-token process. Generating each token requires accessing model weights and processing the current context’s KV Cache. If GPU compute is insufficient, the system cannot output tokens fast enough, causing user requests to queue up and subsequently reducing peak concurrency.

Therefore, the number of GPUs and their compute levels directly determine how many active user sessions an inference service can maintain simultaneously during peak traffic.

2. Two Key Bottlenecks in Decoding: Compute and Memory Bandwidth

To understand the resource bottlenecks of inference services, it is necessary to distinguish between two easily confused concepts: compute-bound and bandwidth-bound scenarios.

Compute-Bound Scenarios

In cases with large model parameters, small batch sizes, or short sequence lengths, the decoding phase is primarily limited by the GPU’s floating-point compute capacity. At this point, the GPU’s compute units are operating at full load, attempting to complete each token generation step as quickly as possible.

Bandwidth-Bound Scenarios

As sequence lengths increase or concurrent batch sizes expand, the decoding phase gradually shifts to being bandwidth-bound. This happens because every token generation requires reading model weights and the current context’s KV Cache from VRAM. The read speed depends on memory bandwidth rather than the theoretical peak of the compute units.

A common engineering observation is that GPU compute is not always the first resource to max out during decoding; memory bandwidth often becomes the bottleneck earlier. This is why evaluating inference service capacity requires looking beyond just the “number of GPUs” to also consider memory bandwidth, VRAM capacity, and KV Cache management strategies.

3. Why the KV Cache Can Be “Discarded”

The KV Cache is the VRAM area used during Transformer decoding to store historical Key and Value tensors. Its purpose is to avoid recomputing attention for historical sequences at each token generation step, thereby saving compute.

However, the KV Cache has an important characteristic: it is disposable.

In chatbot scenarios, a large number of user sessions exhibit the following traits:

• Short session lifecycles, leaving after asking one or two questions.
• Weak correlation between questions (e.g., “Will it rain tomorrow?”).
• Users rarely return to the exact same session to ask follow-up questions later in the day.
• Long-running conversations exist but represent a small proportion of total traffic.

Based on these traits, inference services can proactively release portions of the KV Cache when VRAM is constrained. Specifically, if a user session remains inactive for a certain period, the system can evict that session’s KV Cache from VRAM. If the user eventually returns to the session, the server can recover the state in two ways:

1. Reload the session’s cache from SSD or persistent storage.
2. Discard the cache and recompute—feeding the historical chat logs back into the model to rebuild the KV Cache from scratch.

While the latter consumes extra compute, it is an acceptable trade-off given the low probability of users resuming inactive sessions. The core logic of this strategy is to trade renewable compute for limited VRAM space, thereby serving more new users.

4. How “Decoding Faster” Alleviates VRAM Pressure

When VRAM capacity is insufficient, a more direct strategy is available: accelerate decoding speed to increase resource turnover.

Assume each user session requires a certain amount of VRAM to store the KV Cache, and the GPU’s total VRAM is fixed. If the decoding speed increases, the completion time for individual user requests shortens, and the duration the KV Cache occupies VRAM decreases accordingly. The system can release the VRAM for that session faster and allocate it to the next user waiting in the queue.

This leads to an important conclusion:

With fixed VRAM capacity, faster decoding speeds enable the system to serve more users per unit of time.

This is why inference services invest heavily in optimizing decoding efficiency, including but not limited to:

• Using more efficient attention operators.
• Optimizing batching and scheduling strategies.
• Applying quantization techniques to reduce the VRAM footprint of weights and the KV Cache.
• Using speculative decoding to reduce sequential steps.

The common goal of these optimizations is to increase the GPU’s token output speed without significantly increasing VRAM usage, thereby improving overall throughput.

5. Scheduling Strategies During Compute Shortages

When an inference service experiences compute tension, it usually means GPU resources are already under high load. In such situations, the platform may implement the following scheduling measures:

1. Restricting New User Access

Pausing new user registrations or limiting access is the most direct control method. This prevents the experience of existing users from being further degraded.

2. Reducing Per-User Resource Quotas

This involves limiting the maximum response length, reducing the context window, or shortening the session retention time. These measures reduce the amount of time a single session occupies GPU and VRAM resources.

3. Activity-Driven Cache Eviction

For sessions with no interaction over a prolonged period, their KV Cache is evicted first, releasing resources to active users. This strategy is based on a simple assumption: inactive users have a low probability of returning in the short term.

4. Tiered Service Levels

Providing different priorities for different user groups. High-priority users receive more stable response speeds, while low-priority users may face queuing during peak hours.

Essentially, these strategies are forms of demand-side management to maintain basic service availability when compute supply cannot be rapidly expanded.

6. Engineering Paths for Compute Scaling

From a longer-term perspective, the fundamental way to resolve compute tension is to scale underlying resources. This process is not simply “adding GPUs” but involves synergistic optimization across multiple layers:

• Infrastructure Layer: Adding GPU nodes to increase the cluster’s total compute capacity.
• Scheduling Layer: Optimizing request distribution and batching strategies to reduce GPU idle time.
• Model Layer: Reducing single inference costs through techniques like distillation, pruning, and quantization.
• Caching Layer: Caching results for high-frequency questions or similar queries to reduce redundant inference.
• Architecture Layer: Degrading some long-tail requests to be processed by lower-cost models.

For developers, the significance of understanding these optimization paths is that a dynamic balance exists between the cost and experience of inference services, and this balance point continuously shifts with changes in compute supply.

7. Implications for Developers

When using or deploying LLM inference services, the following points are worth noting:

1. Accurately assess peak concurrency demands: Do not design systems based on average load; peak load is the critical factor determining compute sufficiency.
2. Monitor the KV Cache lifecycle: Understand the platform’s session cache retention policies, especially whether long sessions might be unexpectedly evicted.
3. Design stateless requests: If the business allows, make each request independent without relying on server-side session state. This ensures seamless recovery through recomputation even if the cache is cleared.
4. Prepare queuing and degradation fallbacks: During periods of compute tension, proactively design queuing prompts, degraded responses, or feature toggles to prevent outright request failures.
5. Monitor token output rate: Use the token output rate as a core operational metric. It reflects user experience much more directly than simple GPU utilization.

8. Conclusion

Compute tension and restricting user access are not merely operational events; they are inevitable outcomes of resource constraints in LLM inference services. Understanding the relationship between GPU compute, VRAM capacity, KV Cache management, and peak concurrency helps developers view these phenomena rationally and provides a judgment framework for building or using inference services.

In the era of AI chatbots, compute is not an abstract noun; it is translated into the output speed of every token, the survival time of every session, and the waiting time of every user during peak hours. Only by deeply understanding these underlying mechanisms can we find a truly viable engineering path between resources and user experience.