Deep Dive: LLM Compute Bottlenecks and KV Cache Management

With the rapid expansion of Large Language Model (LLM) applications, compute resource scarcity has become a universal industry challenge. In conversational AI scenarios, the core factor determining the peak limit of concurrent users is often not simply the number of physical servers, but the fine-grained scheduling capability of GPU compute and video memory (VRAM) resources. This article delves into the resource trade-off mechanisms during LLM inference and explores engineering optimization practices under compute-constrained scenarios.

1. Resource Supply and Demand Dynamics During Inference

In autoregressive generative models, the inference process is primarily divided into two stages: Prefill and Decode. These two stages have significantly different demands on GPU hardware resources, forming the core contradiction in system performance optimization.

1. Prefill Stage: Compute-Bound Initialization

When a user sends a request, the model must process the input prompt in parallel and generate the initial KV Cache. This stage is a typical compute-intensive task, where the GPU’s floating-point operations per second (FLOPS) act as the primary bottleneck. The system leverages the GPU’s parallel computing capabilities to process the context as quickly as possible.

2. Decode Stage: Memory-Bound Generation

During the token-by-token generation process, each newly generated token requires reading the entire preceding KV Cache. At this point, although the computational workload is small, memory bandwidth becomes the absolute bottleneck. The GPU often cannot operate at full load and is instead limited by VRAM read speeds. Accelerating the token generation rate allows the system to complete the current request swiftly, thereby freeing up VRAM space to load the next user’s request.

3. Direct Correlation Between VRAM Capacity and Concurrency Limits

The VRAM usage of conversational LLMs consists primarily of two parts: model weights and the KV Cache. Assuming a fixed model parameter size, the number of concurrent users directly depends on how many sets of KV Cache the available VRAM can hold. While VRAM capacity determines how many user contexts the system can “host” simultaneously, compute power and bandwidth dictate the efficiency at which the system outputs results and rotates users.

2. KV Cache Lifecycle Management and Eviction Strategies

In real-world business scenarios, a large volume of conversations exhibits “read-and-discard” characteristics. For instance, in queries like checking the weather or retrieving news, users rarely continue asking follow-up questions based on the original context. To maximize concurrent capacity, the system must adopt aggressive KV Cache management strategies.

1. Priority Eviction and Dynamic Unloading

When GPU VRAM is on the verge of exhaustion, the system will proactively evict the KV Caches of low-activity sessions based on predefined eviction policies (such as Least Recently Used, LRU). This mechanism trades the coherence of some long-running conversations for the ability to admit new user requests. If a user whose context was evicted initiates a conversation later, the system must bear the cost of context reconstruction.

2. Cost Evaluation of Context Reconstruction

When an old session is reactivated, the system typically faces two options:

• Archive Reloading: If the previous KV Cache was transferred to a slower storage tier like an SSD, the system must reload it into VRAM via high-speed I/O. This introduces some latency but avoids redundant computation.
• Recomputation: If no archive exists, the system must treat the user’s chat history as a new prompt and execute the Prefill computation from scratch. This approach not only consumes significant compute power but also causes noticeable response latency for the user.

3. Engineering Optimization Practices Under Compute Constraints

Faced with sudden compute and resource crunches, platforms typically need to optimize comprehensively across three dimensions: scheduling strategies, storage architecture, and infrastructure. The following are standard technical solutions in the industry:

1. Hot-Cold Data Separation and SSD Swapping Mechanism

To balance VRAM capacity and context retention rates, a hot-cold tiered storage architecture can be introduced.

# Pseudo-code example: KV Cache hot-cold tiered scheduling logic
class CacheManager:
    def __init__(self, gpu_memory_limit, ssd_storage):
        self.gpu_cache = LRUCache(gpu_memory_limit)
        self.ssd_storage = ssd_storage
    def get_context(self, session_id):
        if self.gpu_cache.has(session_id):
            # Hot data: read directly from VRAM
            return self.gpu_cache.get(session_id)
        elif self.ssd_storage.has(session_id):
            # Cold data: load asynchronously from SSD to VRAM
            context = self.ssd_storage.load(session_id)
            self.gpu_cache.put(session_id, context)
            return context
        else:
            # No record: trigger full recomputation
            return None
    def evict_if_needed(self):
        if self.gpu_cache.is_full():
            # Evict least recently used data to SSD when full
            evicted_id, evicted_data = self.gpu_cache.evict_lru()
            self.ssd_storage.save(evicted_id, evicted_data)

Through this mechanism, the system can smoothly migrate inactive KV Caches to SSDs when VRAM space is insufficient, rather than discarding them outright. This ensures high concurrency while reducing the compute waste associated with recomputation.

2. Elastic Scaling and Containerized Deployment

Infrastructure elasticity is crucial for handling traffic spikes. Through the integration of container platforms and monitoring alert systems, auto-scaling can be triggered when GPU utilization or VRAM usage reaches a threshold, adding new inference nodes to the load balancer. Conversely, the system automatically scales in during off-peak nighttime hours to optimize resource costs. This cloud-native architecture effectively mitigates resource tension caused by sudden traffic surges.

3. Request Queuing and Intelligent Rate Limiting

When underlying resources reach their physical limits, the application layer must implement intelligent rate limiting and queuing mechanisms. Dynamic scheduling is performed based on request priority (e.g., VIP users, lightweight short conversations, heavy long-text analysis). Requests exceeding current compute capacity are buffered in a message queue to prevent system-wide crashes caused by overloading.

4. Multi-Tier Storage and Compute-Network Collaboration Architecture

For long-text processing scenarios, the size of the KV Cache grows exponentially. Modern AI infrastructure often utilizes object storage for archiving historical sessions, local NVMe SSDs for hot data caching, and GPU VRAM for currently active computation data. Simultaneously, high-speed RDMA networks bridge the storage tiers and compute nodes, ensuring minimal latency during data swapping.

4. Conclusion

The compute scarcity in LLM applications is fundamentally a multi-dimensional trade-off between compute capacity, memory bandwidth, and storage I/O. Under the realistic constraints of limited hardware resources, optimizing the KV Cache lifecycle management, introducing hot-cold data separation mechanisms, and implementing intelligent scheduling and elastic scaling strategies are the critical paths to ensuring high system availability and concurrency. In the future, with the evolution of underlying compute architectures and the deep integration of heterogeneous computing resources, the inference efficiency of large models will unlock even broader optimization potential.