Rate Limiting and Graceful Degradation for LLM Services Under GPU Scarcity
Large model services scale very differently from traditional web services: their peak concurrency is ultimately bounded by the available GPU compute. This article explains why the decode phase keeps GPUs under sustained load, how KV Cache becomes the invisible ceiling on concurrent capacity, and why KV Cache can be evicted at acceptable cost in short-interaction chat workloads. It then lays out a practical framework for admission control, priority scheduling, graceful degradation, capacity planning, and high availability, so teams can keep serving users reliably even when compute is limited. Key takeaways: memory is a hard constraint while compute is a convertible cost, and eviction is essentially trading compute for memory.
1. Introduction: From “Pausing Membership Sign-Ups”
A large model service recently paused new membership sign-ups on July 19, 2026 due to compute constraints. Regardless of the exact cause, such events point to the same technical reality: the peak concurrency of a large model service is ultimately determined by the available compute.
In traditional web services, a single request typically consumes only milliseconds of CPU time. Engineers can support massive concurrency by adding threads, connection pools, and horizontally scaling instances. Large model inference is completely different. Generating a few hundred tokens requires hundreds of sequential forward passes on the GPU, where every token goes through a full cycle of “read weights, compute attention, emit the next token.” A single request can occupy an accelerator for several seconds to tens of seconds.
For a consumer-facing AI assistant, this difference has two direct consequences:
- Compute is rigid: GPU resources take weeks to months to move from procurement and delivery to cluster networking and model deployment. You cannot scale them in seconds the way you can with stateless web services.
- Per-request cost is high: every conversation is real compute consumption. When request growth outpaces resource growth, peak concurrency inevitably hits a ceiling.
So “pausing membership sign-ups” is essentially a form of admission control: it closes off new traffic entry points, shaves down demand peaks, and prioritizes limited compute for existing users and high-priority requests, avoiding a site-wide performance collapse.
Starting from the scheduling mechanics of the inference engine, this article answers the following questions:
- Why does the decode phase keep the GPU under sustained high load?
- When GPU memory runs low, why can KV Cache be “discarded”?
- Where are the cost boundaries of “discarding KV Cache”?
- How can we design admission, scheduling, degradation, and capacity planning on this basis?
2. The Decode Bottleneck: Why the GPU Stays Fully Loaded
2.1 The Two-Phase Inference Model
Autoregressive large models built on the Transformer architecture split inference into two phases:
| Phase | What it processes | Compute characteristics | Main bottleneck |
|---|---|---|---|
| Prefill | Processes all input tokens in parallel at once | Large matrix sizes, high parallelism, close to the compute ceiling | Compute |
| Decode | Generates output tokens one at a time, each based on the full history | Emits one token per step; low arithmetic intensity when batches are small | Memory bandwidth and memory capacity |
An analogy: Prefill is like reading a book quickly and taking notes—read it all in one pass and process in parallel. Decode is like uttering the text one word at a time, and every word requires re-reading the entire book.
2.2 Why Decode Is Bound by Memory Bandwidth
Every decode step must read the entire model weights once. This puts a theoretical lower bound on per-step latency:
Lower bound on step time ≈ model weight bytes ÷ memory bandwidth
Take a 7B (7 billion) parameter model at FP16 precision, with weights of about 14GB. On an accelerator with roughly 2TB/s of memory bandwidth, just reading the weights once takes about 7ms. If each token requires 512 decode steps, a single request needs at least about 3.6 seconds—and this does not yet account for inefficient memory access, attention computation, sampling, or cross-card communication. The longer the context, the more KV Cache each step must read from the corresponding positions, pushing the overhead even higher.
2.3 Continuous Batching: Filling the “Idle” Time
To improve throughput, inference engines use continuous batching: they pack multiple requests into a single decode step. The cost of reading weights is amortized across the batch, so the GPU’s compute and bandwidth are used more fully. A typical flow:
# Inference engine single-step scheduling flow (illustrative)while serving:# 1. Collect runnable requestsrunnable = pending_queue.pop_up_to(max_batch_size - len(running))# 2. Admission: allocate KV Cache (prefill can be split into chunks for interleaving)for req in runnable:req.alloc_kv_cache(est_len=req.input_len + req.max_new_tokens) # fall back to the wait queue if memory is insufficient# 3. Batched forward: one batch shares a single weight readlogits = model.forward(batch=running + newly_added, kv_cache=kv_pool)# 4. Sample and append new tokensfor req in active_batch:req.append(sample(logits[req]))# 5. Retire and release: free memory immediately, refill from the wait queueretire_finished_and_release(active_batch)
Continuous backfilling keeps the GPU busy. That is exactly why GPU utilization naturally approaches saturation as long as requests keep arriving. High utilization is normal, not a failure signal. The real danger is latency spiraling out of control first: queue times keep growing, TTFT (time to first token) and TPOT (time per output token) keep climbing, and eventually timeouts cascade across the system.
2.4 The Capacity Formula
Peak concurrency capacity can be approximated as:
Peak concurrency ≈ (GPU count × effective per-card decode throughput) / (average per-user token rate × safety margin factor)
- Effective per-card decode throughput: total tokens output per unit time;
- Average per-user token rate: mean tokens output per second in a session;
- Safety margin factor: headroom reserved for traffic jitter, long contexts, and failover (e.g., 1.2–1.5).
Example estimate (assumed values, for illustrating the method only):
- Per-card decode throughput: at a batch size of 32, each request produces about 30 tokens/s, for about 960 tokens/s in total;
- Average user rate: output pace matched to the engine, about 15–30 tokens/s, take 20 tokens/s;
- Target utilization: 75%.
Then a single card can support roughly 960 ÷ 20 × 0.75 ≈ 36 concurrent requests. Supporting 1,000 concurrent requests takes about 28 cards, not counting Prefill compute, the extra overhead of long contexts, and cross-card communication losses. Actual values should be based on load-testing results.
This estimate reveals three transmission paths of compute scarcity:
- Failures reduce effective compute: a single machine or card failure instantly cuts available throughput;
- Long contexts consume memory: they cap batch size and lower per-card throughput;
- Prompt bloat increases compute: every input token participates in full Prefill computation.
So the unit cost of compute does not amortize linearly with scale. Capacity planning must be built around peaks, not averages.
3. Memory and KV Cache: The Invisible Ceiling on Concurrent Capacity
3.1 Where Memory Goes and How KV Cache Grows
Accelerator memory is mainly consumed by three things: model weights, KV Cache, and runtime overhead (activations, communication buffers, framework reservations, and so on). KV Cache usage can be estimated with the following formula:
KV bytes per token = 2 × layers × KV heads × head dimension × bytes per elementTotal sequence usage = KV bytes per token × sequence length × number of concurrent sequences
The factor of 2 accounts for Key and Value.
Take an FP16 model with 32 layers, 8 KV heads, and a head dimension of 128. Each token needs about 128KB of KV Cache:
- An 8K-token context session takes about 1GB;
- Expanding to 32K tokens takes about 4GB;
- If a single card reserves 16GB for KV Cache, it can hold only about 4 full-length sessions.
The takeaway is direct: memory determines how many sessions’ contexts can be maintained at the same time. When memory runs low, the engine must either shrink the batch size or evict some sessions’ KV Cache.
3.2 Memory Is a Hard Constraint; Compute Is a Soft Cost
The original view—“a little less memory is not a big deal, just compute faster”—is only half right, and we need to distinguish two kinds of resources:
- Memory is a hard constraint: if it does not fit, execution cannot proceed. Exceeding the batch-size limit triggers an out-of-memory error outright, and requests can only queue or fail. Paged management (allocating KV Cache in fixed-size blocks) can substantially reduce fragmentation and improve utilization, but it cannot conjure capacity out of nothing.
- Compute is a convertible cost: once KV Cache is evicted, it can be brought back by recomputation, turning a “space problem” into a “time problem.”
So a more precise conclusion is: memory determines “how many sessions can be maintained simultaneously,” while compute determines “how much it costs to rebuild them.” The tighter memory is, the more often the system trades compute for space; when the compute cost of eviction-and-rebuild approaches or exceeds the idle compute available, latency deteriorates sharply. This is what “compute scarcity” really means at the engineering level.
4. Why KV Cache in Chat Scenarios Can Be “Discarded”
4.1 The Premise: Conversations Are Dominated by Short Interactions
The ability to evict KV Cache at a lower priority rests on several premises:
- Requests to mass-market chatbots are mostly “short questions, short answers,” like “Will it be nice tomorrow?” or “What did that public figure eat yesterday?”;
- Such sessions are mostly one-off, with a relatively low probability of being repeatedly appended to in a short window;
- Even for longer multi-turn conversations, users rarely keep iterating at high frequency throughout a single day.
For these sessions, the marginal benefit of continuously retaining KV Cache is low, yet it persistently crowds out the memory new requests need. This means the cost of eviction is often just “a slightly slower first token,” in exchange for a significant boost in overall concurrent capacity.
Conversely, scenarios like long-document analysis, codebase understanding, and ongoing multi-turn agent sessions put a much higher value on KV Cache, which must be protected at high priority. At the same time, developers should measure session reuse rates and average lengths from session logs in their own business and use the data to set protection thresholds.
4.2 Three Strategies for Handling Evicted Sessions
| Strategy | Approach | Cost | Suitable scenarios |
|---|---|---|---|
| Drop directly | Free memory; re-prefill from scratch on the next request | Extra compute, higher TTFT | Short sessions, low-value requests |
| Tiered offload | Swap out to host memory or local SSD, swap back when needed | Swap-in bandwidth cost and higher latency | Medium-length sessions likely to be resumed |
| Rebuild + prefix reuse | Free memory; reuse cached common prefixes (system prompts, long documents) on re-prefill | No recomputation for common prefixes | Shared system prompts, RAG, long-document scenarios |
In practice, a combination of “eviction + tiered storage” is common: first evict low-value sessions with long idle times following a least-recently-used policy; if that is still not enough, swap them out to host memory or local SSD rather than zeroing them out.
4.3 Eviction Order: Ranked by “Rebuild Cost”
Eviction should not be ordered by idle time alone; it must also account for rebuild cost, because the cost of “recomputing once” is closely tied to context length:
- Evict short contexts first: rebuilding once takes only tens of milliseconds and is nearly imperceptible to users;
- Swap out long contexts before dropping them: when the context runs to tens of thousands of tokens, re-prefill can take hundreds of milliseconds or longer, so a naive “drop an old session for each new request” amplifies TTFT jitter for long sessions;
- Weight by session value: for paying users and active multi-turn sessions, set lower eviction probabilities or longer retention times.
The eviction decision can be abstracted as:
# Eviction priority score (higher score = evicted first)evict_score = idle_seconds / max(1, cached_tokens) / session_weight
This formula embodies two principles: the longer the idle time, the earlier the eviction; the more expensive the rebuild, the later the eviction. In practice, teams should strike a balance between “utilization” and “reuse rate,” tuning the weights through online parameter optimization.
4.4 Cost Boundaries: Four Points to Clarify
- The eviction cost for short sessions is minimal. If a conversation is only a single turn, its KV Cache is meant to be discarded after use anyway. If the conversation has already been persisted outside the inference engine (such as session history in object storage or a message queue), the next turn’s input already carries the full history, so eviction adds no extra computation.
- The cost for long sessions can be compressed further. With prefix caching, common parts like shared system prompts and RAG-retrieved documents do not need to be recomputed; combined with chunked prefill (splitting a long prompt into chunks interleaved with decode steps), long-session TTFT jitter can be kept within acceptable bounds.
- Even without prefix caching, there is a “free lunch.” When the architecture has no cache reuse, every turn already recomputes the full history, so the compute is identical before and after eviction—and eviction even saves memory along the way.
- When compute is scarce, “computing faster” is itself the scarcest resource. Evicting and rebuilding KV Cache at that point makes an overloaded system even worse. This is exactly why admission control like pausing membership sign-ups is necessary: the problem is not only memory, but the total amount of compute.
5. Compute Scarcity: Root Causes and Product-Side Responses
5.1 Root Causes
- Demand side: new feature launches, trending events, and free-trial promotions create instant peaks; free and paid requests compete for the same resource pool, diluting the commercial value of each unit of compute.
- Supply side: long GPU procurement and delivery cycles make it hard for cluster expansion to keep pace with traffic growth; when training and inference share resources, the compute available to inference gets squeezed; long-running training jobs occupy resources and make them hard to reallocate.
- Efficiency side: batch size constrained by memory, KV Cache growth, verbose prompts, and poor sampling parameters (such as an oversized max_tokens) all inflate the compute cost per request.
5.2 The Product-Side Logic
Pausing membership sign-ups is intuitive but relatively blunt, affecting all users. A more refined approach is to build a tiered system around “value per unit of compute” and “latency tolerance,” for example:
- Entry-tier separation: free users and paid members use separate quotas and resource pools so the core paid path is never crowded out;
- Model tiering: route simple tasks to smaller models, and invoke larger models only for complex tasks;
- Capability tiering: restrict high-cost capabilities like long context, deep reasoning, and multimodality to certain tiers, or open them only during off-peak hours;
- Elastic scaling: shrink inference instances off-peak and return the freed compute to training jobs, then release the training artifacts before the next peak, forming time-shifted reuse.
6. Engineering Practice: Admission, Scheduling, and Degradation
6.1 Admission Control: Queues and Token Buckets
Rate-limiting and queuing at the entry point is more controllable than passively waiting on the GPU. A simplified token bucket implementation:
import math, timeclass TokenBucket:"""Token bucket limiter: refills tokens at a steady rate over time and supports charging by request cost"""def __init__(self, capacity, refill_per_sec):self.capacity = capacity # bucket capacity, determines tolerable burstself.refill = refill_per_sec # token generation rate (tokens/s)self.tokens = capacityself.updated = time.monotonic()def acquire(self, cost=1.0, max_wait=0.0):# Charge tokens by request cost: long prompts and long outputs consume morewhile True:now = time.monotonic()self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.refill)self.updated = nowif self.tokens >= cost:self.tokens -= costreturn Truewait = (cost - self.tokens) / self.refill# If wait exceeds budget, return a retryable signal (with Retry-After semantics)if wait > max_wait:return Falsetime.sleep(wait)
Three key points:
- Charge by cost: request cost can be estimated from input length and expected output length, so long-context requests and short Q&A share resources fairly;
- Multi-level quotas: members and regular users are rate-limited separately and queued independently, protecting high-priority paths;
- Capacity back-calculation: the token generation rate is derived from the compute budget, for example “tokens producible per second × the average per-request token share.”
6.2 Scheduling Priority: The Tighter It Gets, the More Orderly It Must Be
Under compute scarcity, first-come-first-served lets long jobs monopolize the GPU. Multi-level queues can be introduced:
# Scheduling priority suggestions (lower number = higher priority)QUEUE_LEVELS = [{"name": "paid/member", "max_batch": None, "max_queue": 10_000},{"name": "free-short-QA", "max_batch": 12, "max_queue": 5_000},{"name": "long-context/batch jobs", "max_batch": 4, "max_queue": 1_000},]
Higher-priority queues get larger batch quotas and shorter waits, while lower-priority queues are rate-limited or scheduled with delay first when overloaded.
6.3 Graceful Degradation: Retreating Layer by Layer Under Overload
The goal of degradation is not “refusing service” but “continuing to serve at lower cost.” Common techniques include:
- Shorter outputs: tighten the max_tokens cap to reduce decode steps;
- Smaller context: summarize and compress prior turns to lower Prefill compute and KV Cache usage;
- Model downgrade: route requests to smaller models, or fall back to retrieval-based Q&A;
- Feature downgrade: disable or limit high-cost capabilities like multimodal input, web retrieval, and deep reasoning;
- Result downgrade: on failure, return a cached generic answer or a templated reply to preserve basic availability.
Degradation policies should be tiered, configurable, and able to be rolled back quickly by traffic source.
6.4 Observability: Making “Scarcity” Measurable
| Metric | Meaning | Alerting significance |
|---|---|---|
| TTFT | Time to first token (P50/P95/P99) | Queuing or Prefill congestion |
| TPOT | Average time per output token | Batch too large or memory bandwidth saturated |
| Decode throughput | Tokens output per second per card | Compute utilization efficiency |
| KV Cache occupancy | Used blocks / total blocks | Triggers eviction as it nears the threshold |
| Eviction/rebuild rate | Sessions evicted or rebuilt per unit time | Balance state between memory and compute |
| Queue wait time | Time a request spends in the wait queue | Whether admission control is working |
| Rejection rate | Share of requests rate-limited/degraded | Overload severity and user-experience impact |
When memory usage and KV Cache occupancy cross a threshold (such as 80%), it should trigger automatic scaling or tighter rate limiting; a persistently rising eviction rate indicates the system has entered the sub-healthy state of “trading compute for memory.”
7. Capacity Planning and Cost Optimization
7.1 Structured Load Testing
Capacity planning cannot rely on rules of thumb; it needs structured load testing: vary input length (short Q&A / long documents), output length (short / long answers), and concurrency gradients (stepped ramp-up) in combination, recording TTFT, TPOT, and the throughput inflection point for each combination, forming an “input length × concurrency × latency” capacity matrix. In production, weight by the actual request distribution to get the real sustainable peak concurrency, and re-test periodically to account for model version and business distribution changes.
7.2 Three Levers for Cost Optimization
- Raise decode arithmetic intensity: the larger the batch, the more fully each weight read is amortized. But a larger batch raises TPOT, so you must trade off throughput against latency;
- Raise prefix cache hit rate: stable system prompts, templated instructions, and repeated documents are natural cache assets;
- Raise effective GPU utilization: judge health by metrics rather than “100% utilization.” You can reference model compute utilization (actual FLOPs ÷ hardware peak compute) to identify memory-access bottlenecks, low arithmetic intensity, and communication overhead, avoiding the trap of “seemingly full but actually inefficient.”
7.3 Elastic Resources and Cloud Architecture
Businesses show clear tidal patterns, and fixed resources struggle to cover both peaks and troughs. Elastic scaling, pay-as-you-go billing, and resource scheduling let the inference cluster scale automatically with traffic. Enterprises can build an elastic inference cluster using cloud computing and container products—for example, those offered by Baidu AI Cloud—along with general-purpose cloud capabilities like object storage, message queues, log services, and monitoring and alerting:
- Compute layer: GPU instances host the inference services, with the container platform handling orchestration and elastic scaling;
- Storage layer: object storage holds model weights and session history, supporting fast pull and persistence across multiple instances;
- Traffic layer: message queues smooth burst traffic, shaving peaks and filling troughs;
- Observability layer: log services and monitoring/alerting connect metric collection, anomaly detection, and capacity warnings.
7.4 High Availability
- Multi-availability-zone deployment: avoid a single point of failure causing an abrupt drop in compute;
- Cross-region traffic scheduling: switch quickly on failure to protect core paths;
- Runbook drills: regularly drill rate-limiting, degradation, and scaling paths to avoid scrambling during a real incident.
8. Conclusion
Compute scarcity is not simply “not enough GPUs”; it is the combined result of “memory constraints, compute conversion, scheduling efficiency, and demand fluctuation.” The core conclusions can be summarized in five points:
- The memory-bandwidth bottleneck of the decode phase sets the GPU’s peak concurrency ceiling;
- Memory is a hard constraint and compute is a convertible cost; evicting KV Cache is essentially “trading compute for memory”;
- In chat scenarios dominated by short interactions, KV Cache can be evicted by priority, but long-context sessions need tiered protection;
- When compute is scarce, “computing faster” is itself the scarcest resource; admission control and graceful degradation are more rational than simply toughing it out;
- Capacity planning must be based on load testing and observability data, using elastic scaling to absorb tidal fluctuations.
Facing similar resource challenges, enterprises should govern “compute” as a core capacity resource: benchmark with load-testing data, defend with admission and degradation, and regulate with elastic scaling. Baidu AI Cloud’s related cloud products can provide general support to enterprises, from elastic compute to an observability system, helping them serve every user reliably and efficiently even under compute constraints.
