Why LLM Chat Services Hit Compute Limits: Root Causes and Capacity Scaling

    A practical technical guide to why large language model chat services run into compute constraints as demand grows, and how to plan and scale capacity. It explains the two inference phases (prefill and decode), why decode is memory-bandwidth-bound, and how KV Cache grows with context length and caps concurrency. It examines when the "less memory, faster compute" assumption holds and when it breaks, then covers five levers for raising per-GPU efficiency: continuous batching, paged KV Cache, chunked prefill, tiered KV Cache with prefix reuse, and quantization/speculative decoding/parallelism. It closes with admission control, queuing, priority, load-testing methodology, and capacity-planning formulas that turn business volume into a GPU count.

    1. Reading Compute Constraints Through “Pausing Membership Sign-Ups”

    Recently, several LLM chat services have publicly paused new membership sign-ups citing tight compute. This drew attention because it laid bare a simple fact: even a technically leading model product can hit a hard compute-supply ceiling when demand grows quickly.

    Setting aside any specific vendor’s decision, consider the action itself — pausing sign-ups. It is, at heart, business-level admission control. When service capacity is limited, protecting the experience of existing users and deferring new traffic is the classic distributed-systems move of “reject some requests to preserve overall quality of service.”

    To understand this kind of decision — and to judge whether it is unavoidable — three questions come first:

    1. Where is the actual bottleneck in LLM chat inference?
    2. What determines how many users a single accelerator can serve at once?
    3. How do you quantify the compute gap created by growth, and how do you close it?

    2. What Actually Happens When You Generate a Token

    2.1 Two phases: compute-bound prefill and memory-bound decode

    LLM inference typically splits into two phases:

    • Prefill: process all of the user’s input tokens in parallel, compute their intermediate representations, and produce the first output token. This is dominated by large matrix multiplications with plenty of data reuse, so it is compute-bound.
    • Decode: generate tokens one at a time thereafter. Each step computes the Key and Value for the current position over the full history, runs attention against the cached Key/Value, and emits the next token. Every new token is appended to the KV Cache.

    In the decode phase, a single forward pass produces one token yet still has to read all model weights. The dominant operation is matrix–vector: each weight element participates in a single multiply-add, so data reuse is extremely low.

    Under the Roofline model, an operator’s achievable performance is roughly:

    1. Achievable performance ≈ min(peak FLOPS, memory bandwidth × arithmetic intensity)
    2. where arithmetic intensity = FLOPs / bytes accessed

    Decode-phase arithmetic intensity is roughly proportional to batch size. Assume 16-bit floating point and one token generated per request:

    • For a single request in decode, each byte of weight read corresponds to about 1 FLOP.
    • For a batch of N requests, that ratio rises to roughly N.

    The ridge point of mainstream accelerators is usually on the order of hundreds of FLOP/Byte. In other words, the batch size needs to reach the hundreds before the compute units approach saturation. Until then, decode stays memory-bandwidth-bound.

    2.2 A simplified estimation model

    A simple model helps explain decode throughput:

    • Let W be the memory footprint of the weights (in bytes).
    • Each decode step must read the weights at least once, so you can run at most bandwidth / W steps per second.
    • Each step emits one token for every request in the batch.

    This yields:

    • Per-request generation speed ≈ bandwidth / W.
    • Aggregate output throughput ≈ N × bandwidth / W, where N is the batch size.
    • Concurrent users ≈ N × T / (T + I), where I is the average think interval between turns and T is the duration of a single turn from send to completion.

    Note that this is a simplified model that ignores KV Cache, activations, communication overhead, framework overhead, and scheduling jitter. Real capacity must be calibrated with load testing. Even in this simplified view, one point stands out: as long as the weights fit, memory capacity does not directly determine decode speed — it determines the upper bound of N. Memory decides “how many users you can serve at once”; bandwidth decides “how many tokens per second you can produce.”

    The fact that per-request speed is locked by weight movement does not mean the card’s compute is well used. The only way to raise overall efficiency is to grow the batch size, amortizing the cost of reading weights across more requests — that is, raising arithmetic intensity. This is why “increase throughput” and “reduce queuing” are two sides of the same coin during decode.

    3. The Real Constraint: KV Cache and the Concurrency Ceiling

    3.1 Why KV Cache exists

    Without caching, generating each token would require recomputing the attention intermediate results for all prior tokens, and the work grows quadratically with context length. KV Cache stores the Key and Value for historical tokens at every layer, making per-step work grow linearly with context length.

    This is the classic trade of space for time: memory buys throughput and latency stability during generation.

    3.2 How big KV Cache gets

    A single request’s KV Cache footprint can be estimated as:

    1. KV memory ≈ 2 × num_layers × num_kv_heads × head_dim × context_len × bytes_per_elem

    The factor 2 accounts for two copies, Key and Value. If attention uses grouped-query attention (GQA) or similar, “num_kv_heads” is far smaller than the number of query heads, which can substantially reduce this cost.

    The following snippet estimates KV Cache size:

    1. def kv_cache_bytes(num_layers: int,
    2. num_kv_heads: int, # under GQA, KV heads are fewer than query heads
    3. head_dim: int,
    4. seq_len: int,
    5. bytes_per_elem: int = 2) -> int:
    6. # 2: one copy each for Key and Value
    7. return 2 * num_layers * num_kv_heads * head_dim * seq_len * bytes_per_elem
    8. # Example params: 80 layers, 1024 KV dims read per decode step in total, 16-bit precision
    9. per_token = kv_cache_bytes(80, 8, 128) / 1024 / 1024 # convert to MiB
    10. for ctx in (2048, 8192, 32768, 131072):
    11. print(f"{ctx // 1024:>3}K context: {per_token * ctx / 1024:6.2f} GiB/request")

    Output:

    1. 2K context: 0.62 GiB/request
    2. 8K context: 2.50 GiB/request
    3. 32K context: 10.00 GiB/request
    4. 128K context: 40.00 GiB/request

    Two conclusions follow:

    • Concurrency ceiling is inversely proportional to context length. The longer the context, the more memory each request holds. With the weight footprint fixed, the number of requests served simultaneously drops roughly in proportion to context length.
    • With long contexts, KV Cache can outweigh the weights themselves. For example, a model with ~20 GiB of quantized weights would, at 128K context, have a single request’s KV Cache worth roughly twice the weights.

    Every time the product side exposes a longer context tier, the capacity side should immediately recompute the single-card concurrency ceiling. This is the variable most often underestimated in capacity planning.

    4. Does “Less Memory, Faster Compute Is Enough” Hold?

    Some argue that in chat scenarios, KV Cache is disposable transient state rather than data that must be preserved exactly. Most conversations are one-off; even long-running ones may not be resumed quickly, so when memory is tight you can discard the cache and reload it from storage when the user returns — or simply recompute it. As long as decode is fast enough, a bit less memory is fine.

    From an engineering standpoint this judgment is largely reasonable, but it needs three preconditions filled in.

    First, the longer the context, the less “recompute is cheap” holds. A discarded KV Cache can only be recovered from two places: loading from persistent storage, or doing a fresh prefill. And prefill is not merely “reading the weights once”:

    • In each layer’s attention, the scale of the query-times-Key matmul grows with context length, so long contexts add substantial compute.
    • In workloads with a high share of long contexts, prefill compute can exceed decode itself.
    • More importantly, a request being swapped back in or re-prefilled still occupies precious compute and bandwidth, directly slowing the decode already in progress.

    Second, disposability only makes sense if you can recover the state more cheaply. If a session begins by loading a fixed system prompt and tool definitions, or repeatedly references the same long document across turns, then that prefix is highly reusable. Recomputing it “from scratch” every turn means redoing the most stable, most predictable part of the work. A better approach is a tiered cache across GPU memory, host memory, and local storage that prioritizes prefix reuse.

    Third, the goal is not maximizing single-card throughput but maximizing the number of users served while meeting experience targets. The extreme case: push the batch size to the limit and throughput looks high, but every user’s token speed is unbearably slow. The metric must satisfy response-quality requirements — time to first token, token speed — as well as the number of users served.

    The table below compares two memory strategies with three sets of hypothetical figures. The data illustrates the method only and does not represent any real system.

    Item No offload Tiered offload
    Weights per request 1 unit 1 unit
    KV Cache per request (8K context) 1 unit 1 unit
    Single-card concurrency N ~30 ~120
    Weight read time per step 20 ms 20 ms
    KV Cache read time per step 5 ms ~0.3 ms
    Average think interval I 30 s 30 s
    Single-turn duration T 15 s ~15.2 s
    Steady-state users per card ≈ N × T / (T + I) ~10 ~40

    Correspondingly:

    • No-offload strategy: per-token time is roughly 20 ms + 5 ms = 25 ms.
    • Tiered-offload strategy: per-token time is roughly 20 ms + 0.3 ms = 20.3 ms, about 19% lower than 25 ms.
    • Under these assumptions, tiered offload raises single-card concurrency by moving the bulk of the KV Cache to host memory and swapping back only a few requests on demand, which increases the number of users served.

    The reason: weight reads are relatively independent of batch size, while KV Cache reads vary with context and concurrency. When 8K-context KV Cache does not yet dominate overall bandwidth pressure, tiered offload reduces the bandwidth burden on the decode path and raises effective concurrency. Conversely, when contexts stretch to tens of K and KV Cache reads take up a growing share of bandwidth, the per-request slowdown from larger batches becomes more pronounced. At that point you should scale out with more instances.

    So what determines the experience is per-token time, not how large the batch is.

    5. Five Ways to Raise Per-GPU Capacity

    5.1 Iteration-level scheduling: continuous batching

    Traditional static batching waits to assemble a full batch before computing, and can’t release memory until the slowest request finishes. Short requests get “stuck” behind long ones, and memory stays occupied for a long time. A better approach is continuous batching: schedule at the granularity of “one token generated,” letting finished requests leave immediately and waiting requests join at any time. Which requests a given step processes is decided anew at every step.

    5.2 Paged KV Cache management

    KV Cache should be allocated and reclaimed in fixed-size blocks rather than reserving a contiguous slice of memory per request, which greatly reduces fragmentation and improves utilization.

    The snippet below shows the core structure of a continuous-batching scheduler; variable names and semantics mirror established inference-engine practice. It is illustrative pseudocode, and a real system must handle more edge cases:

    1. def schedule_step(running, waiting, cache, max_seqs, max_budget):
    2. # 1. Admit new requests: bounded by both max concurrency and in-batch token budget
    3. while waiting and len(running) < max_seqs:
    4. req = waiting[0]
    5. if current_tokens(running) + req.prefill_tokens > max_budget:
    6. break
    7. if not cache.can_allocate(req):
    8. # Out of memory: evict/swap by priority, recency, or SLO tier
    9. victim = select_victim(running, cache)
    10. if victim is None:
    11. # Nothing to preempt; reject or wait for the next round
    12. return "reject_or_wait"
    13. cache.offload_or_discard(victim) # swap to host memory, or mark for recompute
    14. cache.allocate(waiting.pop(0))
    15. # 2. Advance one step: compute one token for every sequence in the batch
    16. engine.step(running)
    17. # 3. Reclaim: requests that finish, get preempted, or exceed the limit release KV Cache immediately
    18. for req in list(running):
    19. if req.done or req.preempted or req.exceeds_limit:
    20. cache.free(req)
    21. running.remove(req)

    Key edge-case handling:

    • select_victim may return nothing, in which case you should reject new requests or queue them rather than keep allocating.
    • When cache.can_allocate fails, distinguish “swappable” from “must reject.”
    • Preempted requests must save state; on recovery, either load the KV or re-prefill.
    • Over-long prefill requests should have a token budget so a single step doesn’t block.

    With this machinery in place, “how many users you can serve at once” becomes a configurable, dynamically adjustable scheduling parameter rather than a fixed value.

    5.3 Chunked prefill: stop long inputs from clogging the pipeline

    Prefilling a long input can take hundreds of milliseconds or more. If that step runs alone, every decoding request stalls, showing up as jittery token speed. Splitting a long prefill into chunks and mixing them into the same step as decode requests lets long-input requests “prefill while queuing,” keeping other users’ output smooth.

    5.4 Tiered KV Cache: offloading, prefix reuse, and on-demand recompute

    For context reuse, there are three engineering layers:

    1. Prefix caching: for shared prefixes such as system prompts, tool definitions, and knowledge-base documents, compute once and reuse the corresponding KV blocks on subsequent hits. Since chat requests usually share a long fixed prefix, hit rates are often high.
    2. Tiered offloading: KV blocks that don’t fit in GPU memory cascade down to host memory and then to local high-speed storage; long-idle sessions can be persisted further to cloud data storage services. Reactivation loads directly, which is far faster and more stable than recomputation.
    3. On-demand recompute: when a block has been evicted too long, when persistence isn’t cost-effective, or when the context is short, discarding and recomputing on demand is more economical. The right design lets the system choose automatically between reload and recompute based on eviction time, context length, and load-in bandwidth — rather than blanket-keeping or blanket-discarding everything.

    One caveat: KV Cache can contain user inputs, conversation content, or business knowledge snippets, so persistence should account for redaction, encryption, access control, and lifecycle management to avoid turning a transient cache into an unnecessary data-retention risk.

    5.5 Quantization, speculative decoding, and parallelism

    • Quantization: storing weights and KV Cache at lower precision reduces both bandwidth pressure and memory footprint. Halving weight size roughly doubles the theoretical decode-speed ceiling; compressing KV Cache directly expands the concurrency ceiling. The trade-off is accuracy loss, which usually needs evaluation to confirm acceptability.
    • Speculative decoding: a lightweight “draft” source guesses several tokens first, then the main model verifies them in one parallel pass. Accepted tokens amount to producing multiple tokens in a single step, breaking the “read weights only once per step” limit at the single-request level. The cost is extra compute for drafting and verification.
    • Parallelism: when a single card can’t hold the model, you need tensor parallelism, pipeline parallelism, or distributing different layers across instances connected by communication. Higher parallelism means more communication overhead, which can actually raise per-token latency, so it must be chosen with real measurements.

    There is no silver bullet among these; their combined effect is only one thing — raising effective per-card throughput at the same response quality.

    6. Admission Control, Queuing, and Priority: Turning Capacity into an Operable SLA

    Even with per-card efficiency pushed to the limit, queuing is unavoidable when demand outgrows supply. The service’s job then is to turn “disordered congestion” into “orderly, predictable degradation.” These mechanisms are widely used in industry; the specific thresholds should be set against business SLOs:

    1. Rate limiting at the entry: set rate caps per tenant and per endpoint, and give over-long inputs a separate quota. If long-context requests can be submitted without limit, a tiny number of requests can exhaust an entire card’s memory. Because of retry amplification, rate-limit thresholds should leave a safety margin.
    2. Admission-controlled queuing: based on current memory headroom, queue length, and estimated wait, decide whether to accept immediately, queue, or fast-fail with a “retry later” prompt. Fast failure beats making a user wait a long time only to time out.
    3. Priority and preemption: assign priority to different user or service tiers, and when memory is tight, evict or preempt lower-priority requests first to protect core paths. This is essentially the institutionalized version of “discard KV Cache first,” but executed automatically by the system under rules rather than passively waiting for a crash.
    4. Graceful degradation: in extreme cases, shortening max output length, lowering the context cap, routing non-critical workloads to a smaller model, and turning long-form generation into asynchronous tasks all reduce per-request resource consumption.
    5. Capacity watermark monitoring and tiered playbooks: bind actions — “full open, queue, rate-limit, pause new sign-ups, emergency scale-out” — to metrics such as queue length, P95 time to first token, and KV Cache utilization, and trigger them ahead of time. A public “pause membership sign-ups” is, in essence, one of the stricter tiers of this playbook.

    7. Capacity Planning: From Business Volume to GPU Count

    7.1 Three core formulas

    Formula 1: number of cards required

    1. Cards required = daily total tokens × peak factor / (86400 × effective per-card throughput) × headroom

    Here, the peak factor reflects the intra-day peak-to-average gap; effective per-card throughput must come from load testing under the SLO, not from converting the vendor’s nominal peak FLOPS. The headroom factor covers growth, failures, and planned maintenance (the headroom in the code below).

    If input and output differ greatly in volume, a finer approach is to account for them separately:

    1. Input cards = daily total input tokens × peak factor / (86400 × effective per-card input throughput) × headroom
    2. Output cards = daily total output tokens × peak factor / (86400 × effective per-card output throughput) × headroom
    3. Final cards ≈ max(input cards, output cards) or weighted by the mixed workload

    Formula 2: steady-state concurrent users

    A common simplified estimate:

    1. Steady-state users per card ≈ concurrency ceiling N × turn duration T / (turn duration T + average think interval I)

    This is suited to a first-pass capacity screen; actual planning should still rely on load-tested active batch size, time to first token, and per-token latency.

    Formula 3: the utilization “safety line”

    Queueing theory has a basic result: as utilization ρ approaches 1, queueing delay rises sharply. For LLM serving:

    • At 90% utilization, queueing delay is about 5× that at 50%.
    • At 95%, about 10×.
    • At 98%, up to 25×.

    So capacity planning must target an “operable utilization,” typically recommended between 70% and 85% depending on the business SLO — not peak traffic divided by the theoretical ceiling.

    7.2 A worked example

    The example below defines effective per-card throughput as the mixed-token throughput that still satisfies the SLO, i.e., the total token-processing capacity after combining input and output:

    1. def estimate_cards(dau=1_000_000, # daily active users
    2. req_per_user=20, # requests per user per day
    3. in_tokens=1500, # average input tokens (incl. system prompt & history)
    4. out_tokens=500, # average output tokens
    5. peak_ratio=3, # peak factor: peak-period load / daily average load
    6. per_gpu_tps=10_000, # effective total per-GPU throughput (tokens/s, from load testing)
    7. headroom=1.5): # headroom factor: covers growth, failures, planned maintenance
    8. tokens_per_day = dau * req_per_user * (in_tokens + out_tokens)
    9. avg_tps = tokens_per_day / 86400
    10. cards = avg_tps * peak_ratio / per_gpu_tps * headroom
    11. print(f"Daily token total: {tokens_per_day / 1e9:.0f} billion, avg load: {avg_tps:.0f} tokens/s")
    12. print(f"Peak output token demand: {avg_tps * peak_ratio * out_tokens / (in_tokens + out_tokens):.0f} tokens/s")
    13. print(f"Recommended GPU count: {cards:.0f}")
    14. estimate_cards()

    Output:

    1. Daily token total: 40 billion, avg load: 463000 tokens/s
    2. Peak output token demand: 347000 tokens/s
    3. Recommended GPU count: 208

    When reading these results:

    • Peak card count is about 3× the average. This is exactly the root of “idle in quiet times, short in busy times.”
    • Input and output should be accounted separately. Input tokens mainly consume prefill compute; output tokens mainly consume decode bandwidth and memory, so their conversion factors differ. This example uses mixed total throughput for simplicity; a finer approach estimates input cards and output cards separately.
    • 70% of tokens are input. If a substantial share is a shared prefix, prefix caching can markedly cut prefill pressure, so the actual card count may fall below the estimate. Conversely, if the business is all about very long contexts, the card count will be far higher.

    7.3 Cloud elasticity: baseline capacity plus burst capacity

    Scaling LLMs has a special problem: high cold-start cost. Bringing a new instance online means loading tens to hundreds of GB of model weights into memory and then warming up — often several minutes or longer. For second- or minute-level traffic spikes, autoscaling alone is usually too slow.

    A more robust strategy combines “baseline capacity + burst capacity”:

    • Baseline capacity is sized at operable utilization to cover the daily base load.
    • Predictable periodic peaks, such as the evening rush, are handled by scheduled scale-out ahead of time.
    • Trend-driven growth relies on capacity reviews and quarterly scaling plans.

    Mainstream cloud platforms typically offer container autoscaling, monitoring and alerting, and object storage, which can host inference instances, resource-metric alerts, long-term KV Cache persistence, and versioned model distribution. When selecting, evaluate the cluster’s scaling cycle as a capacity risk on its own. If the scaling cycle is measured in months while user growth is measured in weeks, you must reserve a sufficient growth buffer at the planning stage.

    8. Load Testing: Make Capacity Numbers Come from Measurement, Not Specs

    The quality of capacity planning ultimately depends on whether load testing covers real workload characteristics. Design tests across these dimensions:

    1. Realistic length distribution: chat request input and output lengths are usually long-tailed. Testing only with “short input, short output” significantly overestimates real capacity.
    2. Stepped ramp-up: gradually raise concurrency or arrival rate while recording time to first token, per-token latency, throughput, and error rate, and find the inflection point where throughput is still rising but latency starts to degrade noticeably. The load at this inflection point is the true single-instance capacity.
    3. Long context and shared-prefix scenarios: separately measure the impact of 32K and 128K long-context requests on concurrency, and the difference before and after enabling prefix caching.
    4. Bursts and retries: simulate short-term traffic spikes and client timeout-and-retry behavior, and check whether rate limiting and queuing mechanisms hold.
    5. Failure scenarios: simulate partial instance unavailability and check whether remaining capacity can sustain core paths through degradation.

    It’s worth maintaining a load-test record table with at least these fields:

    Scenario Concurrency Input length dist. Output length dist. TTFT P95 Per-token P95 Throughput Error rate
    Short-session baseline
    Long input
    Long context
    Shared prefix
    Burst retry

    From a scaling and selection standpoint, focus on answering three questions:

    • How to scale: first find the load-test inflection point, then scale by the headroom factor; if the model’s cold start is slow, prefer baseline capacity plus scheduled scaling.
    • How to select: if contexts are long and prefix reuse is high, prefer an inference solution that supports paged KV, prefix caching, and tiered offloading; if short sessions dominate, focus more on low latency and batch-scheduling capability.
    • Common mistakes: testing only with short input and short output, looking only at peak FLOPS, ignoring KV Cache’s impact on concurrency, having no rate limiting, and overlooking retry amplification.

    Metrics to watch closely after launch:

    • Time to first token P95/P99;
    • Per-token latency P95;
    • Queue duration and queue length;
    • KV Cache memory usage and eviction rate;
    • Prefix cache hit rate;
    • Number of request preemptions;
    • Rate-limit rejection rate.

    Among these, prefix cache hit rate and KV Cache eviction rate are the most valuable “leading indicators.” Their changes often signal capacity pressure earlier than user complaints do.

    9. Conclusion

    Back to the original question: why do chat services run out of compute? Not because of a vague “the model is too big and the cards aren’t fast enough,” but because of three specific mechanisms stacking up:

    • the memory-bandwidth bottleneck during decode;
    • KV Cache growing linearly with context;
    • residency time jointly determined by user concurrency and think interval.

    The claim “less memory, faster compute is enough” basically holds in short-session, low-prefix-reuse scenarios. But as contexts grow longer and prefix reuse becomes more common, relying solely on “discard and recompute” comes at an ever-higher cost.

    A more complete solution builds a system spanning five layers — scheduling, caching, admission, capacity, and load testing:

    • release memory with iteration-level scheduling;
    • reduce fragmentation with paged management;
    • cut redundant computation with prefix caching and tiered offloading;
    • protect core experience with admission control and priority;
    • support growth with load-test-driven capacity planning.

    For developers and technical decision-makers, the “pause membership sign-ups” moment is a reminder: the competitiveness of an LLM service depends not only on the model’s own quality, but on how many users each unit of compute can reliably serve. And that number can be measured, optimized, and planned.