LLM Concurrency Bottlenecks: GPU Compute, KV Cache, and Engineering Optimizations

1. Background: Service Degradation and Rate Limiting Under Traffic Spikes

Large language models evolve quickly, and user growth is often nonlinear. When a new model makes a breakthrough in long-context processing or logical reasoning, request volume can grow exponentially within a very short period, overwhelming existing compute capacity and forcing service providers to take emergency measures such as service degradation or rate limiting.

From a systems engineering perspective, this is not merely a business adjustment. It is a self-protection mechanism triggered when the underlying GPU compute infrastructure hits physical limits. Understanding the technical principles behind this behavior helps developers plan resource architectures more rationally when building AI applications and avoid core business interruption caused by compute exhaustion.

2. Core Bottleneck: The Dynamic Interaction Between GPU Compute and KV Cache

In LLM inference, the peak number of concurrent users is primarily constrained by GPU memory capacity and parallel compute capability. Request processing is divided into two phases: Prefill and Decode. The competition between these two phases for hardware resources is the root cause of concurrency bottlenecks.

2.1 The Conflict Between Token Generation Rate and Throughput

The number of tokens generated per second directly determines the lifecycle of an individual request. Higher throughput means the system completes the current query faster and releases resources sooner. However, GPU compute capability is limited by chip-level physical constraints. When the request queue far exceeds the throughput ceiling, queuing latency becomes unavoidable.

2.2 The KV Cache Memory Trap

Autoregressive generation in LLMs requires maintaining historical context. The system caches the Key and Value matrices from each attention layer in GPU memory, commonly referred to as KV Cache. As conversation turns and context length increase, KV Cache grows linearly or even exponentially in size.

The key engineering contradiction is this: insufficient compute can be handled by queuing, but once GPU memory is full, new requests fail immediately with “Out of Memory” errors. Memory scarcity is a hard constraint that cannot be resolved by simply slowing down computation.

2.3 Cache Eviction Strategies for One-off Sessions

The industry commonly adopts a “discard and recompute” strategy. A large number of user requests are one-off in nature, such as weather queries or simple fact lookups. When a request is completed or memory pressure rises, the system actively evicts inactive KV Cache entries. If a user resumes a long-running conversation hours later, the system can either reload historical KV Cache from SSD, which adds latency but avoids recomputation, or have the model perform a fresh Prefill pass with the full history. This approach sacrifices some experience for returning users while freeing up valuable concurrency capacity for new requests.

3. Engineering Practices for Resource Scheduling Under High Concurrency

Simply adding more GPUs is not a sustainable long-term solution. High hardware costs and supply chain constraints make coarse-grained scaling impractical. Developers must introduce fine-grained scheduling and storage mechanisms.

3.1 Separating Memory and Compute: PagedAttention

Drawing inspiration from virtual memory paging in operating systems, modern inference frameworks have introduced paged attention mechanisms. KV Cache is divided into fixed-size pages that are allocated on demand, eliminating fragmentation caused by reserving contiguous GPU memory. According to public benchmarks from frameworks such as vLLM, based on A100 80GB with batch size 256, paging can reduce memory fragmentation to below 10%, significantly improving effective utilization and allowing more concurrent request contexts to be processed on the same GPU.

PagedAttention vs. traditional contiguous allocation:

Dimension	Traditional Contiguous Allocation	PagedAttention
Fragmentation rate	30%–50%	Below 10%
Max concurrency (same memory)	Lower	2–4x improvement
Supported frameworks	Early HuggingFace	vLLM, TGI, Text Generation Inference

3.2 Multi-tier Storage and Asynchronous Offloading

To relieve GPU memory pressure, engineering teams adopt a multi-tier storage architecture. When GPU memory reaches a high-water mark, the scheduler asynchronously offloads KV Cache from waiting or low-priority sessions to CPU memory. When CPU memory is nearly exhausted, data is pushed further down to NVMe SSD. Data is copied over high-speed PCIe 4.0/5.0 buses without blocking GPU computation, effectively creating an extended memory pool constrained by PCIe bandwidth and SSD IOPS.

The following is a logical illustration, not production code:

# Pseudocode: water-mark-based asynchronous KV Cache offloading logic (illustrative only)
import logging

logger = logging.getLogger(“kv_offload”)
HIGH_WATERMARK = 0.90  # Typical alert threshold; calibrate for the actual GPU model

def check_memory_and_offload(request_queue, gpu_memory_usage):
    try:
        if gpu_memory_usage > HIGH_WATERMARK:
            target_req = find_low_priority_request(request_queue)
            if target_req:
                offload_kv_cache_to_cpu(target_req)  # Asynchronous copy, non-blocking
                free_gpu_memory(target_req)
                logger.info(f”Offloaded KV Cache for request {target_req.id}”)
                return True
    except Exception as e:
        logger.error(f”Offload failed: {e}”)
        # Trigger fallback: reject new requests or initiate emergency release
    return False

Risk notes: Although asynchronous offloading extends available memory, it introduces several risks: ① Reloading from CPU or SSD increases request latency, typically by tens to hundreds of milliseconds. ② If a node fails during offloading, KV Cache data may be lost. Persistent snapshots and retry mechanisms are required. ③ PCIe bandwidth and SSD IOPS are the real throughput bottlenecks and should not be treated as unlimited expansion. Multi-level alerting and graceful degradation strategies are recommended so that, in extreme cases, the system rejects new requests cleanly rather than crashing entirely.

3.3 Continuous Batching

Traditional batching waits until enough requests are collected, forcing earlier requests to wait. Continuous batching performs dynamic scheduling at the token level. Each time the model finishes generating a token, the scheduler checks whether new requests can be inserted into the current batch or whether completed requests can be removed. This fine-grained scheduling maximizes GPU compute density and keeps the hardware operating near its bottleneck capacity. The workflow is as follows:

1. Initialize: place ready requests into the first batch and send it to the GPU.


2. Iterate: after each token is generated, check the queue for newly arrived requests and completed requests.


3. Insert/evict: new requests join immediately; completed requests release memory and context slots.


4. Repeat: continue until the queue is empty, keeping GPU utilization high throughout.

4. Cloud-Native Elastic Architectures for Enterprises

Building a large private GPU cluster requires significant upfront investment and is difficult to justify during periods of low utilization. Leveraging cloud-native architectures and mature cloud services is often a more practical way to handle compute shortages.

4.1 Elastic Scaling Based on Container Platforms

Once inference services are containerized, autoscaling policies such as Horizontal Pod Autoscaling can be configured based on GPU utilization and request queue length. When monitoring detects a concurrency surge, the container platform can launch new inference pods within minutes and distribute traffic through load balancing. Enterprises do not need to continuously hold peak-level compute capacity, which substantially reduces operating costs.

4.2 Decoupling Storage and Compute

Object storage and message queues can be used to build a highly available request buffer. User requests enter a message queue first rather than hitting the GPU directly. When compute is constrained, requests wait safely instead of causing a thundering-herd failure. Object storage can also persist conversation history, providing high-throughput read capability for restoring KV Cache from SSD.

4.3 Log-Driven Model Iteration and Resource Optimization

By collecting metrics such as request length distribution, average generation time, and KV Cache hit rate through logging services, engineering teams can make targeted optimizations. A common quantization comparison is shown below:

Quantization	Accuracy Loss	Memory Savings	Use Cases
INT8	Minimal (<1%)	Approx. 50%	General inference, conversational generation
INT4	Moderate (1%–5%)	Approx. 75%	High-concurrency, cost-sensitive scenarios; use with caution for high-precision tasks

In addition, pruning and efficient attention operators such as FlashAttention can further reduce per-request compute consumption, improving both cost and performance. Quantization may introduce accuracy loss and compliance risks, so thorough evaluation on critical business workloads is recommended before rollout.

5. Summary: Three Key Takeaways

Compute bottlenecks in LLM services are an inevitable challenge as the industry scales. At the core is the complex interplay among GPU compute throughput, memory capacity, and KV Cache lifecycle management. Developers and architects should focus on the following three areas:

1. Technical layer: Understand core techniques such as PagedAttention, asynchronous multi-tier memory offloading, and continuous batching to maximize concurrent capacity on limited hardware.


2. Architecture layer: Combine cloud-native elastic scaling with storage-compute decoupling to build highly available services that can withstand traffic spikes.


3. Optimization layer: Use log-driven quantization, pruning, and operator optimization to fundamentally reduce per-request compute consumption and continuously improve cost efficiency.

6. FAQ

Q1: What should I do if KV Cache is too large?

You can use PagedAttention to reduce fragmentation, set a reasonable context length limit, enable asynchronous offloading for inactive sessions, and apply truncation or summarization at the application layer for very long conversations.

Q2: How should I choose a quantization approach?

For general inference and conversational generation, INT8 is usually acceptable. For high-concurrency, cost-sensitive scenarios, INT4 may be worth trying, but accuracy loss must be thoroughly evaluated. For high-precision domains such as healthcare or legal applications, INT4 should be used with caution; INT8 or mixed precision is generally preferred.

Q3: Can asynchronous offloading cause data loss?

The offloading process itself does not lose data, but loss may occur if a node fails and no persistent snapshot has been taken. Regular snapshots, retry mechanisms, and graceful degradation strategies are recommended to ensure recoverability.