LLM Inference Bottlenecks and Dynamic KV Cache Scheduling Strategies

As large language models (LLMs) are deployed across more industries, user concurrency on chatbots and intelligent Q&A systems has grown explosively. Recently, a leading LLM platform announced compute resource shortages under extreme concurrency pressure and suspended some membership services, sparking widespread discussion in technical circles. On the surface, this reflects a conflict between business-side capacity planning and sudden traffic spikes, but from an infrastructure perspective, it is an extreme challenge involving GPU compute allocation, memory scheduling, and KV Cache lifecycle management.

1. Core Bottlenecks of Peak Concurrency in Chatbot Services

In traditional API services, throughput limits are typically determined by CPU frequency, network bandwidth, or the number of concurrent backend database connections. In LLM inference services, however, the core resource that determines the upper bound of concurrent users has fundamentally shifted—GPU compute, especially memory capacity and compute cores, has become the absolute bottleneck.

Specifically, LLM inference consists of two main stages: Prefill and Decode. In the Prefill stage, the system processes the user’s input prompt in parallel, which is a compute-intensive task. In the Decode stage, the model generates the answer token by token, which is a memory-bandwidth-intensive task. The number of tokens the system can emit per second directly determines how quickly it can complete the current user’s query.

When GPU compute throughput approaches its limit, the system can quickly finish processing a request, then offload the KV Cache occupied by that request, freeing memory space to load the next user’s request. In this ideal loop, the GPU operates at full load as close to the bottleneck as possible. Therefore, the amount of GPU resources directly determines how many active user inference contexts can be maintained simultaneously during peak periods.

2. Throughput Trade-offs Under Memory Constraints and KV Cache Scheduling

In production environments, GPU memory is extremely expensive and limited. If memory runs slightly short, does the system crash immediately? The answer is no—this is actually a trade-off between throughput and latency.

Chatbot request processing has certain peculiarities. Because LLM inference is autoregressive, the token generated in the previous step depends on historical context, so the Key and Value tensors of the historical conversation must be stored in memory—this is the so-called KV Cache. If the system’s GPU compute is fast enough, it can quickly finish emitting the current user’s request and immediately release the KV Cache space occupied by that user to serve the next request.

User behavior patterns in chatbot scenarios make this “compute fast, release fast” strategy feasible. Many user queries are one-off, such as “Will the weather be good tomorrow?” or “What was a certain public figure’s schedule yesterday?”—these requests do not require long-term context memory. Even for long conversations, users may not iterate at high frequency multiple times within a day. Therefore, to satisfy more concurrent users with limited compute, systems often adopt aggressive KV Cache eviction policies.

3. Engineering Practices for KV Cache Discarding and Rebuilding

When compute or memory runs critically low, actively discarding the KV Cache of some inactive users is a common degradation strategy to preserve availability. But this decision brings subsequent engineering challenges: when a user suddenly returns days later to continue a previous conversation, how should the system respond?

1. Storage Tiering and Cold Data Loading

To balance memory occupancy and user experience, the industry commonly adopts a storage tiering architecture. GPU memory serves as the most expensive Tier 0 layer, retaining only the KV Cache of currently active sessions. When a session is idle beyond a certain threshold (e.g., 5 minutes), the system offloads its KV Cache to SSD (Tier 1). If the user reactivates that session, the system must reload the state from SSD.

The following simplified pseudocode for KV Cache lifecycle management illustrates this tiered scheduling logic:

class KVCacheManager:
    def __init__(self, gpu_memory_limit, ssd_storage):
        self.gpu_memory = LRUCache(capacity=gpu_memory_limit)
        self.ssd_storage = ssd_storage
        self.idle_threshold = 300  # 5-minute idle threshold
    def get_cache(self, session_id):
        # Prefer reading from GPU memory
        if self.gpu_memory.has(session_id):
            return self.gpu_memory.get(session_id)
        # GPU memory miss, load from SSD into GPU memory
        elif self.ssd_storage.exists(session_id):
            cache_data = self.ssd_storage.load(session_id)
            self.gpu_memory.put(session_id, cache_data)
            return cache_data
        # Both miss, trigger full recomputation or return context timeout
        else:
            return self.recompute_context(session_id)
    def release_memory_if_idle(self, session_id, idle_time):
        if idle_time > self.idle_threshold:
            # Evict GPU memory data to SSD
            cache_data = self.gpu_memory.evict(session_id)
            self.ssd_storage.save(session_id, cache_data)

2. Cost Analysis of Prefill Recomputation

If the SSD has not retained the state, or the system determines that loading from SSD is too time-consuming (e.g., for very long contexts), the fallback is to recompute the Prefill from scratch with the full chat history. In modern compute frameworks, this is called context recomputation. Although it significantly increases the time to first token (TTFT), compared with directly denying service, this “bite the bullet and recompute” strategy offers greater advantages in overall system availability.

4. Architectural Optimization Recommendations for High-Concurrency Compute Shortages

When facing extreme situations like “compute shortage suspending services,” beyond passively limiting user access, technical teams should introduce more general cloud-native and distributed optimization solutions at the system architecture level.

1. Elastic Scaling with Compute-Storage Separation

Modern LLM inference platforms should avoid monolithic architectures and adopt a design that separates compute from storage. Container platforms such as Kubernetes can elastically scale inference nodes: when monitoring and alerting systems detect that GPU utilization or memory levels persistently exceed 80%, new inference Pods are automatically launched to share the load. Meanwhile, session state (KV Cache) is externalized to high-performance distributed caches or object storage, allowing any compute node to seamlessly take over a user’s subsequent sessions.

2. Group Scheduling and Priority Queues

Under overload, uniformly suspending membership services is not elegant. A more reasonable approach is to introduce message queues and priority scheduling. High-priority requests (e.g., paying enterprise users) and low-priority requests (e.g., free trial users) are assigned to different request queues. When underlying GPU resources are strained, the system uses a token bucket algorithm to limit the throughput of the low-priority queue, ensuring the stability of core business; meanwhile, Dynamic Batching merges multiple short low-priority requests into a single batch for inference, maximizing GPU compute density.

3. Optimizing the Memory Bandwidth Bottleneck in the Decode Stage

As mentioned earlier, the Decode stage is memory-bandwidth-intensive. Under memory-constrained scenarios, low-level optimization techniques can reduce the size of the KV Cache. For example, quantization can compress the KV Cache from FP16 to INT8 or even INT4, increasing the in-memory concurrent capacity severalfold without significantly sacrificing model accuracy; alternatively, general techniques such as PagedAttention can manage the KV Cache in a fragmented manner through a page-table mechanism, much like an operating system manages virtual memory, greatly reducing resource waste caused by memory fragmentation.

Conclusion

An LLM platform limiting access due to compute shortages is essentially a manifestation of GPU resource supply-demand imbalance under high concurrency. For developers and architects, understanding the Prefill and Decode mechanisms in LLM inference, the KV Cache lifecycle, and the trade-off logic of memory scheduling is the foundation for building stable AI services. Facing continued user growth, a systematic engineering approach combining tiered storage, dynamic batching, elastic scaling, and low-level memory optimization is the ultimate answer to breaking through compute bottlenecks and achieving smooth commercialization.