Why Popular LLM Services Keep Crashing: The Logic Behind the Compute Shortage

    This article explains why popular large language model services often become overloaded or unavailable despite rapid improvements in model capability. It examines the compute-demand dynamics behind success, including Jevons Paradox, the prefill and decode phases, context length, KV cache, sparse mixture-of-experts architectures, and batching. It also covers practical capacity planning, autoscaling, rate limiting, queuing, graceful degradation, and cloud-based capabilities that help teams absorb traffic spikes.

    Introduction: A highly capable model can overwhelm its servers

    Imagine a scenario in which a large language model (LLM) keeps improving its benchmark scores, performing well across code generation, logical reasoning, and long-context processing. As its reputation spreads, user request volume can surge within hours—or even tens of hours—eventually exceeding the cluster’s capacity. The operator is forced to pause new-user onboarding.

    This may seem contradictory: the more successful the product, the closer the service comes to collapse. Yet it is a recurring reality in online LLM services.

    To understand this phenomenon, it helps to clear up a common misconception: improvements in model efficiency do not automatically lead to lower total compute consumption.

    1. A counterintuitive truth: stronger capabilities can demand more compute

    Market intuition often assumes that a smarter model should require less compute because it produces more value per unit of compute. In reality, the opposite tends to happen. This is the AI-era version of Jevons Paradox.

    The original Jevons Paradox describes how, when steam engines became more efficient and produced more power from the same amount of coal, total coal consumption did not fall. Instead, it rose—because greater efficiency lowered the barrier to use and encouraged broader adoption.

    Large language models follow the same pattern:

    improved model capability → better user experience → broader application scenarios → a surge in API calls → rising total compute demand

    In other words, efficiency gains can be quickly consumed by explosive demand.

    Demand-side growth is further constrained by rigid supply-side limits. Inference compute for LLMs is determined by multiple factors, several of which are often overlooked.

    2. Where inference compute is actually consumed

    An online inference can be roughly divided into two stages: prefill and decode.

    • Prefill stage: This stage processes all tokens of the user input at once. It is compute-intensive because it feeds the prompt into the model in parallel.
    • Decode stage: This stage generates output tokens one by one. It is memory-access intensive because every generated token requires reading model weights.

    These two stages place different pressures on hardware. As a result, estimating compute usage is much more complex than simply dividing parameter count by some coefficient.

    Several factors also amplify compute demand:

    1. Context length
      As the context grows longer, attention computation and cache usage rise superlinearly. Long-text tasks can consume compute especially aggressively.

    2. KV cache
      To accelerate decoding, the system must cache the intermediate results of historical tokens. Higher concurrency and longer contexts increase KV-cache memory usage, which directly limits the number of concurrent requests that can be served per GPU.

    3. Hidden costs of sparse architectures
      Mixture-of-Experts (MoE) architectures activate only part of the expert networks during each inference, reducing computation per token. However, loading, routing, and scheduling hundreds or thousands of expert parameters still incur significant memory and scheduling overhead. A large total parameter count means that memory capacity and memory bandwidth requirements remain demanding even when activation is sparse.

    4. Concurrent batching
      To improve throughput, servers combine multiple requests into batches. Larger batches can increase throughput, but they also lengthen per-request latency and raise peak memory usage.

    When these factors are combined, the problem becomes clearer: if call volume multiplies within a short period, memory and compute resources can be exhausted quickly. Hitting cluster capacity becomes only a matter of time.

    3. Capacity planning: three calculations that must be done before launch

    To make a service resilient, capacity planning should not be based on guesswork. At minimum, three calculations are needed:

    • Peak calculation
      Use historical or estimated peak concurrency as the baseline, rather than average values. LLM traffic often has sharp spikes, and averages can be highly misleading.

    • Latency calculation
      Define SLA targets for first-token latency and overall generation latency. These targets can then be used to derive the concurrency limit and batching strategy for each instance.

    • Redundancy calculation
      Reserve buffer capacity for traffic bursts, failover, and canary releases. A certain amount of headroom is usually needed to avoid pushing resources to the critical point.

    A simple capacity-estimation approach is:

    1. Required instances ≈ Peak QPS × average resource usage per request / (available capacity per instance × target utilization)

    Target utilization should not be set to 1.0. Leaving margin is necessary to absorb fluctuations.

    4. Elastic scaling: letting the cluster breathe with traffic

    A static resource pool cannot handle steep traffic spikes. Elastic scaling is a core mechanism. Automatic scaling based on key metrics can add compute when traffic rises and reclaim resources when traffic falls.

    Scaling signals can include request queue length, memory or compute utilization, and average latency.

    1. # Illustrative autoscaling logic (pseudocode)
    2. def autoscale_controller(metrics, current_replicas, config):
    3. load = metrics.queue_length / config.max_queue_per_replica
    4. latency = metrics.p95_latency
    5. if load > config.scale_up_threshold or latency > config.latency_sla:
    6. target = min(current_replicas + 1, config.max_replicas)
    7. elif load < config.scale_down_threshold and latency < config.latency_sla:
    8. target = max(current_replicas - 1, config.min_replicas)
    9. else:
    10. target = current_replicas
    11. # Keep a cooldown period when scaling down to avoid repeated scaling due to jitter.
    12. return target

    It is important to note that LLM instances often have slow cold starts. Loading model weights and warming caches take time. Elastic scaling therefore needs to account for lead time: using metric prediction or scheduled policies to complete scaling before peaks arrive, rather than reacting only after the system has already become overloaded.

    5. Rate limiting, queuing, and graceful degradation: flood control for traffic peaks

    Even with elastic scaling, instant unlimited scaling is not possible. When a real traffic flood arrives, it is better to actively manage demand than to let all requests overwhelm the system.

    • Rate limiting
      Set request rate limits by user, API, or globally. Token-bucket and leaky-bucket algorithms are commonly used to smooth traffic.
    1. # Minimal illustration of token-bucket rate limiting
    2. class TokenBucket:
    3. def __init__(self, capacity, refill_rate):
    4. self.capacity = capacity # bucket capacity
    5. self.tokens = capacity # current tokens
    6. self.refill_rate = refill_rate # tokens added per second
    7. def allow(self, now, last_refill, need=1):
    8. # Refill tokens based on elapsed time.
    9. self.tokens = min(
    10. self.capacity,
    11. self.tokens + (now - last_refill) * self.refill_rate,
    12. )
    13. if self.tokens >= need:
    14. self.tokens -= need
    15. return True
    16. return False
    • Queuing and backpressure
      Place excess requests into message queues to smooth out peaks. Combine timeouts and retry policies to avoid unbounded request accumulation exhausting connections.

    • Graceful degradation
      Under pressure, prioritize core paths. Non-critical capabilities, such as ultra-long context or complex reasoning, can be degraded or queued, leaving scarce compute for the most important requests.

    • Circuit breaking and fast failure
      When downstream systems are overloaded, returning a “retry later” response quickly is friendlier than making users wait indefinitely, and it also helps protect the system itself.

    6. Combining cloud capabilities

    On cloud platforms, several common capabilities can be combined to address compute shortages:

    • Container platforms and elastic compute: Host containerized inference services and use auto-scaling groups to handle traffic fluctuations.
    • Load balancing: Distribute requests across multiple instances and remove unhealthy nodes using health checks.
    • Message queues: Act as buffering and peak-shaving layers to smooth sudden traffic bursts.
    • Object storage: Store model weights, datasets, and other large files so newly scaled-out instances can pull them quickly.
    • Monitoring, alerting, and logging: Observe QPS, latency, and memory/compute utilization in real time, and trigger alerts and automatic scaling before thresholds are reached.

    For example, Baidu AI Cloud provides composable cloud capabilities that can be combined as needed, helping teams establish a closed loop of planning before traffic peaks, elastic response during events, and retrospective analysis afterward, without building the entire infrastructure from scratch.

    Conclusion

    The tendency of LLM services to become slower or less stable as they gain popularity is not simply a matter of “not enough GPUs.” It is the combined result of improving capabilities, surging demand, and rigid compute constraints.

    By understanding Jevons Paradox, recognizing the true composition of inference compute, and implementing capacity planning, elastic scaling, rate limiting, and graceful degradation, teams can better absorb traffic peaks.

    At its core, compute is a supply-and-demand problem. The value of engineering lies in making the answer to this problem sufficiently robust.