Why LLM Serving Is Constrained by Compute: The Underlying Logic of Concurrency Limits

    This article explains why conversational LLM services hit concurrency ceilings even after training is complete, arguing that online inference is a resource-intensive, real-time workload. It breaks down the two generation phases—compute-bound prefill and memory-bandwidth-bound decode—shows how KV Cache imposes a hard memory constraint on concurrency, and explains why chat sessions make the KV Cache effectively disposable. It then covers continuous batching, paged memory management, and a tiered storage model with recomputation trade-offs, closing with practical architecture takeaways for capacity planning and elastic scaling.

    1. What “Pausing New Sign-Ups” Reveals About Compute Constraints

    When a conversational LLM service announces that it is limiting new user sign-ups because of “compute pressure,” many people wonder: the model weights have already been trained, so why does compute still bottleneck the serving stage? The answer is that online inference is a resource-intensive, real-time service. Unlike offline batch processing, which can run slowly, inference must continuously produce tokens for every concurrent request within a latency budget measured in milliseconds to seconds.

    The key insight is this: in a chatbot-style deployment, what determines how many users can be served simultaneously at peak is often not the model size, but the combined effect of effective throughput per unit time and memory capacity. How many accelerator cards you have available largely determines how many active sessions you can “hold” at the same instant. So “pausing new sign-ups” is essentially an explicit admission control mechanism—rather than letting all users queue until they time out, the service proactively shrinks the concurrency level to protect the quality of service (SLA) for existing users.

    2. Prefill and Decode: Two Distinctly Different Workload Profiles

    To understand the compute bottleneck, you must first recognize the two phases of a single generation request, whose resource characteristics are almost mirror images:

    • Prefill: The entire user prompt is fed into the model in parallel at once, computing the Key/Value for all tokens and writing them to the cache. This phase is compute-bound—matrix multiplications have high arithmetic intensity, saturating the GPU’s compute capacity (FLOPS).
    • Decode: Autoregressive, token-by-token generation, where each step processes only one new token but must read the entire historical KV Cache. This process is memory-bandwidth-bound, and the real bottleneck is memory bandwidth rather than compute.

    A rough illustration to express the difference between the two phases:

    1. def generate(request):
    2. # Prefill: parallel computation over a long sequence, saturates compute
    3. kv_cache = model.prefill(request.prompt_tokens)
    4. # Decode: token-by-token generation, saturates memory bandwidth
    5. output_tokens = []
    6. for _ in range(max_new_tokens):
    7. logits, kv_cache = model.decode_step(kv_cache)
    8. next_token = sample(logits)
    9. output_tokens.append(next_token)
    10. if next_token == EOS:
    11. break
    12. return output_tokens

    Precisely because decode is bandwidth-constrained, engineers often say that “once you set aside the memory-bandwidth bottleneck of the decode phase, you can approximately treat the GPU as running at full capacity against the bottleneck.” The faster you emit tokens, the sooner you finish the current user’s generation, release their KV Cache, and load the next request—trading higher turnover for greater concurrency capacity. A little memory pressure is not alarming; as long as you compute fast enough and turn over requests frequently enough, you can serve more users.

    3. KV Cache: A Double Ledger of Memory and Bandwidth

    KV Cache is central to understanding concurrency limits. It caches the Key and Value vectors for each generated token, so decode does not need to recompute the historical part. But its cost grows linearly with sequence length × concurrency × number of layers × hidden dimension, often becoming the single largest consumer of memory. Its footprint can be approximated as:

    1. KV_bytes ≈ 2 × num_layers × num_heads × head_dim × seq_len × batch_size × dtype_bytes

    Here the 2 accounts for the two copies of Key and Value. The longer the sequence and the more concurrent requests, the larger the footprint. This is the “hard constraint” memory imposes on concurrency: when the sum of all active sessions’ KV Cache approaches the memory limit, the system can no longer admit new requests and must queue or reject them.

    4. Why Chat KV Cache Is “Disposable”

    The most elegant point in the original argument is this: a chatbot’s KV Cache has low temporal-locality reuse value. This contrasts with scenarios like code completion or RAG retrieval, which often need to retain context for a long time. But the chat scenario has its own characteristics:

    • Most sessions are one-off: Queries like “What’s the weather tomorrow?” or “What did a certain public figure eat yesterday?” are often short, throwaway interactions, and users are unlikely to revisit the same session repeatedly.
    • Even long conversations are not high-frequency iterations: Even in long-context conversations, users are unlikely to return to the same session many times a day to ask follow-up questions continuously.

    So a pragmatic engineering judgment is: to accommodate more users, you can first discard the KV Cache of some idle sessions. When a user resumes the conversation days later, you can reload from persistent storage, or simply accept the loss and recompute prefill with the full chat history. The extra overhead of these two fallback approaches is usually far smaller than the cost of scaling up just to retain all caches long-term.

    A schematic expression of this eviction-and-recovery strategy:

    1. def admit_request(new_req, kv_pool, max_capacity):
    2. if kv_pool.size >= max_capacity:
    3. victim = lru_evict(kv_pool) # Evict by least recently used
    4. offload_to_storage(victim) # Offload to cheap storage, or just drop
    5. kv = load_from_storage(new_req.session_id) # Reuse on a hit
    6. if kv is None:
    7. kv = model.prefill(new_req.history) # Recompute prefill on a miss
    8. return kv

    This reflects two levels of trade-off: the eviction policy (LRU/LFU, etc.) decides “who gets dropped,” and the recovery policy (offload vs. recompute) decides “how to bring it back.”

    5. A Scheduling Perspective: How Continuous Batching Amplifies Effective Compute

    Looking at a single request alone rarely saturates a GPU; real concurrency comes from the scheduling layer. The widely adopted idea of continuous batching is to fill new requests into the batch the instant old requests finish, rather than waiting for the whole batch to align. Compared with static batching, it can significantly improve the utilization of both memory and compute:

    1. while system_running:
    2. batch = scheduler.form_batch() # Dynamic assembly: requests from mixed phases
    3. # A batch may contain both prefill and decode requests at the same time
    4. model.step(batch)
    5. for finished in batch.finished_requests():
    6. release_kv_cache(finished) # Release immediately, making room for new ones
    7. admit_new_requests() # Fill the vacancies from the waiting queue

    Combined with paged management of the KV Cache (similar to virtual-memory paging in an operating system, where the cache is split into fixed-size blocks and allocated on demand), memory fragmentation can be effectively mitigated, allowing more sessions to “fit in.” This is exactly the scheduling philosophy behind the phrase “a little less memory is not a big deal—compute fast enough and I can load the next user.”

    6. Engineering Practice: Trade-offs Between Tiered Storage and Recomputation

    When the cache cannot all remain resident in memory, mature systems build a tiered storage architecture that schedules KV Cache across different media:

    • GPU memory (HBM): Lowest latency, holding only currently active sessions—a scarce resource.
    • Host memory (DRAM): Serves as the first-level swap-out area, with large capacity and acceptable latency.
    • Low-cost storage (such as object storage or local SSD): The fallback tier, used for long-term residency or cross-request reuse.

    The trade-off between recomputation and swapping in/out can be summarized as a decision logic:

    1. def recover_kv(session):
    2. if in_hbm(session): return hbm_read(session) # Fastest
    3. if in_dram(session): return swap_in(session) # Next fastest
    4. if in_storage(session): return prefetch(session) # Slower
    5. return recompute_prefill(session) # Slowest, fallback

    The general principle is: prefer loading when the hit rate is high; recompute directly when the hit rate is low. The shorter the session and the sparser the reuse, the smaller the cost of recomputation—in which case “just drop it and recompute when needed” becomes globally optimal. This also explains why the chat scenario can afford to boldly discard caches—because the worst-case fallback cost is controllable.

    7. Conclusion: Compute Scarcity Is the Norm, Architecture Is the Cure

    Returning to the original phenomenon: pausing member sign-ups due to compute pressure is essentially peak concurrent demand exceeding the physical limit of current resource capacity. This is almost an inevitable stage in the large-scale growth of LLM services, because it is simultaneously squeezed by three constraints: compute, memory capacity, and memory bandwidth.

    For developers and enterprises, there are three transferable takeaways. First, clarify your business’s session reuse characteristics and design the retention and eviction strategy for KV Cache accordingly. Second, make good use of continuous batching and paged memory management to push effective throughput close to the bottleneck. Third, build tiered storage and elastic scheduling capabilities. On cloud platforms, such needs can usually be addressed with general capabilities like container-platform elastic scaling, using object storage and message queues for cache swap-out and request queuing, and monitoring and alerting for capacity forecasting. General infrastructure capabilities offered by cloud platforms such as Baidu AI Cloud can also support capacity planning and elastic hosting for this kind of inference service. Ultimately, compute pressure is not the end of the road, but a normal constraint that a mature inference architecture must design for in advance.