LLM Inference Bottlenecks and High-Concurrency Optimization Practices

The Compute-Bandwidth-Memory Triangle in Large-Scale Inference

In LLM inference, system resources are constrained by three dimensions: compute capacity (FLOPS), memory bandwidth, and VRAM capacity. These three factors form a tightly coupled resource triangle. A bottleneck in any one of them directly limits the system’s peak concurrency.

KV Cache: The VRAM Consumer in High-Concurrency Scenarios

In autoregressive generation, the model must access the key-value pairs of all previous tokens when generating each new token. This is known as the KV Cache. Without caching, every new token would require recomputing the entire preceding context, which is computationally prohibitive.

However, caching comes at the cost of significant VRAM usage. For a model with L layers, hidden dimension d, context length s, and FP16 storage (2 bytes), the KV Cache memory for a single request is approximately:

Memory_KV = 2 × L × s × d × 2 bytes

For modern LLMs that support context lengths of hundreds of thousands of tokens, KV Cache consumption can far exceed the model weights themselves once concurrent requests reach a certain scale. When GPU VRAM is fully occupied, new requests cannot be admitted even if the compute cores are idle. This is the classic memory wall problem.

Resource Patterns in Prefill and Decode

LLM inference typically consists of two phases:

• Prefill phase: Processes the user’s input prompt and computes the initial KV Cache. This phase is compute-intensive, with the GPU’s matrix multiplication units running at full capacity.
• Decode phase: Generates subsequent tokens one by one. This phase is memory-bandwidth-intensive. For each generated token, the large KV Cache must be read from VRAM into the compute units. GPU compute utilization is often very low during this phase, with the bottleneck entirely on memory bandwidth.

Under extreme concurrency, the system must simultaneously handle large numbers of prefill and decode requests. This causes intense contention for compute and bandwidth resources, leading to request queueing and increased response latency.

Dynamic Resource Scheduling and Eviction Strategies Under Extreme Traffic

When hardware expansion cannot keep pace with a sudden traffic spike, the system must rely on software-level dynamic scheduling to maintain availability of core services.

KV Cache Eviction and Recomputation

Chatbot scenarios have a notable characteristic: many sessions are short-lived and one-off. For example, when a user asks about tomorrow’s weather or a historical fact, the context is unlikely to be accessed again after the response is generated.

Based on this pattern, the system can adopt an aggressive KV Cache eviction strategy. When VRAM usage reaches a threshold, the system actively offloads KV Cache from inactive sessions to free up space for new requests. This approach trades response time for long-running conversations in exchange for throughput on short-lived ones.

If a user whose context was evicted returns hours later to continue the conversation, the system faces two options:

• Cold loading from SSD: Reload the KV Cache that was previously offloaded to NVMe SSD back into VRAM. This is limited by PCIe bandwidth and can be slow.
• Recomputation: Recompute the entire prefill from scratch using the full conversation history. This is computationally expensive for very long contexts, but is often accepted to admit new users.

Continuous Batching and Dynamic Memory Management

To maximize GPU utilization, modern inference engines widely adopt Continuous Batching. Unlike traditional static batching, continuous batching allows the system to dynamically remove completed requests from a batch during the decode phase and insert waiting requests into the same compute batch.

A simplified scheduling flow looks like this:

class DynamicBatchScheduler:
    def __init__(self, max_batch_size, vram_manager):
        self.max_batch_size = max_batch_size
        self.running_queue = []
        self.waiting_queue = []
        self.vram_manager = vram_manager
    def add_request(self, request):
        self.waiting_queue.append(request)
    def schedule(self):
        # 1. Remove finished or aborted requests and free VRAM
        finished = [req for req in self.running_queue if req.is_finished()]
        for req in finished:
            self.vram_manager.free_kv_cache(req)
            self.running_queue.remove(req)
        # 2. Check VRAM usage and try to pull new requests from the waiting queue
        while self.waiting_queue and len(self.running_queue) < self.max_batch_size:
            # When VRAM is insufficient, evict the oldest inactive session's KV Cache
            if not self.vram_manager.check_available_vram(prompt_length):
                evicted = self.vram_manager.evict_oldest_inactive_kv()
                if not evicted:
                    break # VRAM is fully occupied; cannot accept new requests
            req = self.waiting_queue.pop(0)
            self.vram_manager.allocate_kv_cache(req)
            self.running_queue.append(req)
        return self.running_queue

Through this dynamic scheduling, the system can smooth out high-concurrency request handling as much as possible within the physical limit of available GPU resources.

Cloud-Native Elastic Scaling and System-Level Optimization

When software-level scheduling reaches its limit, the fundamental solution to compute shortages remains infrastructure expansion. In AI-era infrastructure management, elastic cloud-native architecture and heterogeneous compute management become critical.

Decoupling Compute and Memory: Disaggregated Inference Architecture

To overcome the VRAM limits of a single GPU, the industry has begun adopting architectures that separate prefill from decode. The compute-intensive prefill phase is handled by compute-optimized nodes, while the memory-bandwidth-intensive decode phase is handled by nodes with large VRAM capacity or high memory bandwidth.

Using cloud-native message queues and container platforms, the KV Cache generated by prefill nodes can be transferred to decode nodes over high-speed networks such as RDMA. This decoupling allows each phase to scale elastically and independently, significantly improving overall system throughput.

Elastic GPU and Cloud-Native Scheduling

For sudden LLM traffic spikes, traditional manual capacity expansion is often too slow. With mature cloud-native infrastructure, teams can build automated elastic scaling mechanisms:

• Metrics monitoring and alerting: Continuously collect GPU VRAM utilization, request queue depth, and token generation latency metrics such as TTFT and TPOT using cloud-native monitoring components.
• Autoscaling policies: Define HPA (Horizontal Pod Autoscaler) policies based on multiple correlated metrics. When queue depth exceeds thresholds and VRAM usage is critical, new inference instances are automatically launched.
• Resource pooling and sharing: For non-core or offline inference workloads, container platform priority preemption can automatically yield resources to high-priority online chat services during compute shortages.

Multi-Tier Storage and Offloading Optimization

When VRAM resources are extremely scarce, proper storage tiering is critical. Modern inference frameworks typically use a three-tier storage system consisting of GPU VRAM, CPU memory, and NVMe SSD.

• When GPU VRAM is full, KV Cache with low access frequency is offloaded to CPU memory.
• When CPU memory also reaches its limit, it is further compressed and offloaded to SSD.
• Combined with prefetching mechanisms, KV Cache can be loaded back from SSD to memory or VRAM before the user is likely to start the next round of conversation.

This mechanism depends on efficient coordination between the underlying operating system and cloud storage services to maximize concurrent capacity while preserving user experience.

Summary and Outlook

An LLM platform suspending service access due to compute shortages may appear to be an operational incident caused by a traffic surge, but it reflects the serious challenges that large-scale AI inference services face in resource scheduling, memory management, and cloud-native architecture.

For developers and architects, understanding the KV Cache memory model, the resource characteristics of prefill and decode, and dynamic batching and offloading strategies is a prerequisite for building highly available LLM applications. As competition in the LLM space intensifies, extracting maximum inference throughput from limited hardware resources will become a core technical capability for both cloud providers and application developers. Deep optimization of inference engine architecture, combined with cloud-native elastic scheduling, remains the fundamental way to address compute shortages.