Overcoming LLM Inference Compute Bottlenecks: High Concurrency Optimization Practices

Introduction: The Compute Supply and Demand Paradox in High-Concurrency LLM Scenarios

When major LLM platforms throttle or pause certain user services during peak compute periods, it inevitably sparks broad discussions across the industry regarding “GPU compute limits.” In the era of conversational LLMs (chatbots), the core resource dictating the ceiling for peak parallel users is typically compute power—specifically, GPU throughput and memory capacity.

Whether a platform can smoothly support concurrent requests from massive user bases fundamentally depends on the number of tokens it can output per unit of time, as well as its scheduling system’s efficiency in managing context memory (KV Cache). When compute supply cannot match the influx of user requests, the system hits a bottleneck and is forced to adopt protective measures like rate limiting or service suspension. This article delves into the root causes of compute bottlenecks in LLM inference and explores how to maximize system throughput under finite hardware resources through architectural optimization and scheduling strategies.

1. Tracing the LLM Inference Pipeline and Compute Bottlenecks

The LLM inference process primarily consists of two stages: Prefill and Decode. These stages have vastly different compute and bandwidth consumption characteristics, which directly determine the system’s concurrency ceiling.

1. Prefill Stage: Compute-Intensive Context Initialization

When a user submits a long prompt containing historical context, the model must compute the KV Cache for all input tokens at once and write them to memory. This stage is a classic compute-intensive operation, where GPU compute power (FLOPS) becomes the primary bottleneck. The system needs to complete this batch of matrix multiplications as quickly as possible to enter the generation state promptly.

2. Decode Stage: Memory-Intensive Step-by-Step Generation

Once the token generation phase begins, the model generates only one or a few tokens at a time and continuously reads the existing KV Cache. At this point, the GPU’s compute units are often “starved,” and the true bottleneck shifts to memory bandwidth.

During the Decode stage, the GPU essentially operates at full load, pushing right against the memory bandwidth limit. If compute resources are sufficient and scheduling is properly handled, the system can quickly unload the memory space occupied by a current user’s query tokens after generation is complete, load the next user’s request, and thereby maintain high throughput.

2. KV Cache Lifecycle Management: The Core Lever for Throughput

In chatbot scenarios, the KV Cache management strategy directly dictates how many active users the system can maintain concurrently. The “discard and reload” mechanism is essentially a common overload protection and resource swapping strategy used across the industry.

1. Memory Footprint Characteristics of KV Cache

The longer the context length of an LLM, the larger the generated KV Cache footprint. Taking common large-scale parameter models as an example, the memory consumed by a single user in a multi-turn long conversation can easily reach the gigabyte (GB) level. When thousands of concurrent requests pour in, GPU memory capacity can be exhausted instantly, preventing new requests from being scheduled.

2. Eviction Strategy: Trading Time for Space

To meet the initial usage needs of a broader user base, systems often employ aggressive KV Cache eviction strategies. Because chatbot scenarios exhibit a prominent “one-off” characteristic—such as simple information queries (“Will it rain tomorrow?”) or single-pass long-text analyses—the probability of a user iterating on the same session multiple times within a short period is relatively low.

Therefore, when memory is critically low, the system prioritizes discarding KV Caches that are in an idle state. This action frees up precious memory resources, allowing the GPU to immediately accept new user requests. While this sacrifices some long-term conversational coherence, it is a necessary compromise to ensure overall service availability.

3. State Recovery: SSD Loading and Recomputation Mechanisms

When a user reactivates a previously discarded session after some time, the system needs to restore the context. The industry currently relies on two primary strategies:

• Tiered Storage Loading: Infrequently used KV Caches are downgraded from the foreground memory and transferred to SSDs (Solid State Drives) or object storage. When the user initiates a conversation again, the system asynchronously loads the historical state from the SSD. Although this introduces microsecond to millisecond-level latency, it avoids the overhead of recomputation.
• Recomputation: In scenarios of extreme resource scarcity or where the context state has been completely purged, the system will recompute the Prefill stage from scratch using the user’s complete chat history. This approach consumes extra compute power but offers simple implementation logic and high fault tolerance.

3. Engineering Practices to Break Through Compute and Memory Walls

Faced with inevitable compute constraints, enterprises and developers need to approach the problem from a systems engineering perspective, combining cloud-native architectures with low-level optimization technologies to build more elastic inference services.

1. Intelligent Request Scheduling and Batching

The core of improving GPU utilization lies in efficient batching. By implementing continuous batching technology, the scheduling system can dynamically insert newly arrived requests into the currently executing batch during each Decode iteration cycle, while simultaneously removing requests that have finished generation. This mechanism eliminates the compute waste seen in traditional static batching—where the system waits for the longest request to finish—ensuring the GPU consistently outputs tokens at maximum load.

2. Memory Optimization and Operator Fusion

At the underlying computation level, a series of universal techniques can be applied to compress the KV Cache footprint. For instance, utilizing low-bit quantization (such as INT8/INT8 KV Cache quantization) can halve memory consumption with almost no loss in the model’s generation quality. Furthermore, applying operator fusion to merge multiple non-linear activations and matrix operations into a single kernel can drastically reduce GPU memory access times, alleviating the bandwidth pressure during the Decode stage.

3. Elastic Cloud-Native Architecture Support

In scenarios with pronounced peaks and valleys in compute demand, relying on a single cluster to hard-carry the load will inevitably lead to rate limiting. By leveraging container platforms and elastic computing resources provided by cloud infrastructures like Baidu AI Cloud, inference services can scale on demand.

• Decoupled Inference Deployment: Utilizing cloud-based message queues and distributed storage services, the microservice decoupling of request reception, preprocessing, model inference, and result return can be achieved. When requests pile up, the container platform automatically spins up additional inference pods to shoulder the pressure.
• Heterogeneous Compute Scheduling: Combining various compute instance specifications provided by cloud vendors, the platform can schedule compute-intensive Prefill tasks to compute-optimized GPU nodes, while directing memory-intensive Decode tasks to nodes with high bandwidth and large memory capacities, achieving optimal utilization of hardware resources.

4. Future Outlook: Combating Resource Scarcity Through Architectural Evolution

The growth curve of compute power will always struggle to perfectly match the exponential explosion of user requests. As developers and architects, beyond squeezing every drop of GPU performance at the engineering level, we must also pay attention to algorithmic evolution. For example, using model distillation and pruning to obtain smaller yet equally efficient lightweight models to handle massive volumes of low-difficulty, one-off Q&A requests; or introducing speculative decoding technology, where a lightweight model quickly generates candidate tokens that are then verified in parallel by the larger model, thereby multiplying the token output rate.

Compute scarcity should not merely trigger a crisis of service suspension; it should serve as a catalyst forcing the upgrade of underlying architectures and refined operations. Through smarter KV Cache lifecycle management, more extreme low-level operator optimization, and the elastic empowerment of cloud-native architectures, we are fully capable of building robust AI services that support hundreds of millions of concurrent interactions on a limited compute foundation.