LLM Inference Capacity Under Pressure: Capacity Planning, Graceful Degradation, and Elastic Scaling for High-Concurrency AI Services
A practical engineering guide to handling large-language-model inference shortages during traffic spikes. Learn why "capacity shortage" is usually a queuing problem rather than a crash, how to plan capacity from measurable SLOs, how to design a multi-level graceful degradation ladder, and how to control the compute amplification of multi-agent workloads. Includes working reference code, capacity formulas, and an operational checklist for keeping AI services responsive at peak.
Introduction: The engineering and business signals behind a “try another tool” message
A recent, widely discussed incident offers a useful starting point. Shortly after a next-generation model launched, a platform repeatedly slowed down or timed out during peak hours. The UI showed a message that compute was short at peak and the service had switched to a fast mode, and the team followed up with a plain-spoken reply: they were scaling up and suggested users try other tools in the meantime.
Most of the attention landed on the posture of actively recommending an alternative. From an engineering standpoint, though, it is more instructive to unpack three layers of the problem:
- Where the compute gap comes from: a single request to the new model often consumes several times the compute of an ordinary conversational model;
- How to degrade gracefully under overload: whether to show a generic “system busy” template, or to give users an actionable, predictable alternative path;
- How the capability foundation caps scheduling headroom: with dynamic autoscaling and multi-cloud resource pools, the real levers for reducing single-node congestion risk are the elastic ceiling and the migration cost.
Using this incident as a thread, this article turns the discussion toward a more general technical question: under high concurrency, how do large-model inference services do capacity planning, graceful degradation, and elastic scaling? Solve that well on the engineering side, and users naturally have no reason to “look elsewhere” at peak.
Problem framing: a peak-hour capacity shortage is queuing, not a crash
When the service misbehaves, users see “unable to answer,” but engineers usually see a very different picture: GPU utilization is near saturation, requests pile up in the queue, wait times exceed the timeout thresholds of the client or gateway, and the backlog surfaces to users as failures. In most cases, a compute shortage is not an outage but queuing plus timeouts.
2.1 What queueing theory tells us intuitively
Model the inference service as a queueing system: let the request arrival rate be λ and the service rate of a single instance be μ, so utilization is ρ = λ/μ. Using the simplest M/M/1 model, the average time a request spends in the system is:
E[T] = (1 / μ) / (1 - ρ) = E[S] / (1 - ρ)
where E[S] is the average service time. This gives:
| Utilization ρ | Avg. time in system (relative to single-request time) | Assessment |
|---|---|---|
| 50% | 2x | Healthy |
| 70% | ~3.3x | Elevated, needs attention |
| 80% | 5x | Critical, prone to backlog at peak |
| 90% | 10x | Overloaded, frequent timeouts |
| 95% | 20x | Severely overloaded |
This curve explains two common misconceptions:
- “Not yet maxed out” does not mean safe. At 80% utilization, average wait is already 5x the single-request time; layer a peak-hour traffic surge on top, and tail latency degrades sharply.
- Scaling up may not stop the bleeding immediately. New instances have to spin up containers, load model weights, and warm up GPU memory and caches. From the scale-up trigger to actually taking traffic is often a delay measured in minutes. Until then, only queueing policy and degradation can hold the experience floor.
2.2 Three dimensions of inference compute
The compute a conversational model consumes is not determined by “how many turns the user asked,” but by how much each interaction actually computes:
- Prefill: a full forward pass over the input context, with compute that grows roughly linearly with input length and is also shaped by the attention mechanism;
- Decode: token-by-token autoregressive generation, where GPU memory bandwidth and KV Cache capacity are the main bottlenecks and time scales roughly with output length;
- Scheduling amplification: a single user request may fan out into “plan → retrieve → multiple tool calls → aggregate results,” and multi-agent parallel orchestration is the most typical case. One top-level task fans out into several sub-agents working at once, and a single request’s compute can balloon to several times that of an ordinary single-turn conversation.
User request: research and generate a competitive analysis report│Top-level agent (planning / aggregation)│┌────┼────┬────┬────┐Sub-agent Sub-agent Sub-agent ... ← parallel fan-out│ │ │Tool call Search Code exec└────┴─────┬────┘Result aggregation → final output
So phenomena like “the servers got hammered two weeks after the release” are not surprising: traffic growth and per-request compute inflation stack up far beyond the linear extrapolation expected from historical conversational models.
Capacity planning: working backward from SLOs
3.1 Define measurable SLOs first
The prerequisite for capacity planning is turning “is the service good?” into quantifiable metrics. Set service-level objectives (SLOs) separately for different interaction types. Example:
| Metric | Meaning | Example target |
|---|---|---|
| Time to first token (TTFT) | From request sent to first token received | P95 ≤ 2 s |
| Time per output token (TPOT) | Average decode time per token | P95 ≤ 50 ms |
| End-to-end completion latency | Total time to finish, strongly tied to output length | Tiered by scenario |
| Request success rate | Share not returning 5xx and not triggering a timeout degradation | ≥ 99.5% |
| Timeout degradation rate | Share degraded to a fast model or queued directly | ≤ 5% (at peak) |
Use P95 rather than the mean, because the long tail of large-model responses is especially pronounced: a few long-output requests can drag up the average without reflecting what most users actually feel.
3.2 Establish a “tokens/s per instance” baseline
The core of capacity estimation is effective throughput per unit of compute. Benchmark two profiles separately:
- Long input + short output: biased toward a prefill bottleneck;
- Short input + long output: biased toward a decode bottleneck.
Weight the two results together to get a single-instance throughput baseline (tokens/s) for your business traffic profile.
3.3 Capacity estimation formula
Required instances N ≈ (peak QPS × avg tokens per request) ÷ (per-instance throughput × target utilization)
Worked example (all parameters hypothetical): an agent feature with a peak QPS of 50, an average of 30,000 tokens per request, a single-instance throughput baseline of 8,000 tokens/s, and a target utilization of 0.7:
N ≈ (50 × 30000) ÷ (8000 × 0.7) ≈ 268 instances
For compute-heavy agent features, also reserve burst headroom (say 20%) and cross-region disaster-recovery redundancy. This example also illustrates a frequently overlooked fact: agentification brings a qualitative change, not just quantitative growth. Extrapolating the compute needs of an agent product from a conversational product’s historical capacity easily produces order-of-magnitude errors.
3.4 A three-tier resource structure
Relying on on-demand scaling alone is both expensive and laggy. Split resources into three tiers:
| Resource tier | Role | Configuration principle |
|---|---|---|
| Reserved baseline | Absorbs everyday traffic | Sized to the P50–P70 of daily peaks |
| Elastic scaling | Handles morning/evening peaks and hot events | Sized to P95–P99 peaks, triggered in seconds to minutes |
| Reserved buffer | Absorbs bursts and localized failures | A share of idle capacity, spread across availability zones |
Load balancing, health checks, and connection draining are the prerequisites for this structure to work reliably: on scale-up, new instances take traffic only after passing a readiness probe; on scale-down, an instance is first pulled out of the load balancer and only released after in-flight requests finish.
Graceful degradation: turning “system busy” into a predictable alternative path
Capacity always has a ceiling. The more important engineering question is how to step back gracefully under overload. Design a four-level degradation ladder:
| Level | Trigger signal (example) | System behavior | User perception |
|---|---|---|---|
| L0 Normal | All metrics nominal | Full capability | Full features |
| L1 Model routing | Queue wait P95 > 2 s for 3 min | Simple requests route to a lightweight model; complex tasks stay on the flagship | Simple questions faster, complex tasks slightly slower |
| L2 Capability reduction | After L1, queue wait still > 5 s | Shorter max context, reduced reasoning depth, lower sub-task concurrency | Complex tasks take longer; “fast mode” notice shown |
| L3 Queueing / rate limit | Instance utilization > 85% for 5 min | Throttle by user tier and feature priority; low-priority requests queue with an estimated wait | Requests wait in queue or are guided to retry later |
4.1 A degradation state machine example
from dataclasses import dataclass@dataclassclass LevelRule:queue_wait_ms: float # Queue wait latency threshold (P95)util: float # Instance utilization thresholdLEVEL_RULES = [("L1", LevelRule(2000, 0.75)),("L2", LevelRule(5000, 0.85)),("L3", LevelRule(float("inf"), 0.90)),]OVER_WINDOW = 3 # Consecutive over-threshold windowsRECOVER_WINDOW = 10 # Consecutive normal windowsdef choose_level(metrics: dict, current: str) -> str:"""Pick the target degradation level from the latest window's metrics (illustrative logic)"""waiting = metrics["queue_wait_p95_ms"]util = metrics["instance_util"]target = "L0"for name, rule in LEVEL_RULES:if waiting > rule.queue_wait_ms or util > rule.util:target = name# Recovery has a higher bar: low utilization must persist longer to recoverif util < 0.60 and waiting < 500 and metrics["recover_windows"] >= RECOVER_WINDOW:target = "L0"if target == "L0" and current != "L0" and metrics["over_windows"] < RECOVER_WINDOW:target = currentreturn target
Four principles are worth holding to when designing a degradation policy:
- Trigger one level at a time: rather than jumping straight from L0 to L3, drop one rung and observe the effect first;
- Add hysteresis and a minimum dwell time: entering a higher level requires 3 consecutive over-threshold windows, while returning requires low utilization for 10 consecutive windows, so the system does not oscillate between two levels;
- Be transparent: clearly tell users what mode they are in, what has changed, and roughly when it will recover;
- Make degradation configurable: thresholds and the model parameters per level should support dynamic adjustment, so you can respond to a traffic surge without shipping a release.
4.2 Client-side cooperation: backoff retries and off-peak guidance
While the server degrades, the client also has to cooperate:
import random, timedef call_with_backoff(request, max_attempts=3, base=0.5, cap=8.0):"""Exponential backoff + randomized jitter retry (illustrative logic)"""for attempt in range(max_attempts):try:return request()except OverloadedError:if attempt == max_attempts - 1:raisesleep = min(cap, base * (2 ** attempt))time.sleep(random.uniform(0, sleep))
Randomized jitter prevents large numbers of clients from retrying at the same instant and forming a “retry storm” that worsens the overload. The UI copy should also shift from a cold “system busy” to something specific and actionable, for example:
Compute is tight at peak right now and we’re scaling up. Simple Q&A can switch to fast mode (faster average response); for complex tasks, please retry a bit later. Full capability is expected to be restored after XX:00.
Model routing: trading a “difficulty judgment” for cost efficiency
Not every request needs the strongest model. First having a lightweight model or classifier judge request difficulty, then deciding the routing target, is a standard cost-control technique.
5.1 A two-tier cascade
User request│▼[Lightweight model / classifier] ──simple──▶ Fast model ──▶ response│└──complex──▶ Flagship model (full context + deep reasoning + tool calls) ──▶ response
Routing features can combine signal strength, context length, whether tool calls or code execution are needed, and the user’s historical preferences. Two engineering points matter here:
- Fallback on misjudgment: set a confidence threshold and default to the flagship model when unsure; also log routing decisions for offline evaluation and iteration.
- Cache up front: query a semantic cache or prefix cache (KV Cache reuse) before routing; on a hit, return directly and avoid a needless model call.
def route(req, classifier) -> str:"""Illustrative routing: prefer the fast path for low risk, with a fallback"""if cache_hit(req):return "cache"difficulty = classifier.score(req) # 0~1, higher means more complexif difficulty < 0.35 and not req.needs_tools:return "fast"if difficulty < 0.70:return "standard"return "flagship"
5.2 The boundary between fast and slow models, and user expectations
Layering fast and slow models should not depend on “difficulty” alone; it can also follow “interaction mode”: real-time chat pursues time to first token, while asynchronously generated reports and code tasks care more about final quality and can use different model combinations and timeout policies. But however you layer, hold one bottom line: the output quality of core capabilities must not shrink because of degradation. Once users feel the “fast mode” is answering the wrong question, the trust loss far outweighs the compute cost saved.
Compute amplification and control in multi-agent scheduling
Multi-agent parallelism is a core selling point of agent products and also the compute source most prone to running out of control. A task that fans out to 10 sub-agents means occupying 10 copies of context and KV Cache at once; without constraints, one user’s task is enough to drag down an entire instance group.
6.1 Bounded concurrency and timeout control
import asyncioMAX_CONCURRENT_SUBTASKS = 4 # Max concurrent subtasks per requestSUBTASK_TIMEOUT_S = 45 # Subtask timeout (seconds)RETRY_LIMIT = 1 # Max retry count on failureasync def run_subtasks(tasks, executor, max_concurrent=MAX_CONCURRENT_SUBTASKS):"""Run subtasks under a concurrency cap and timeout constraint (illustrative logic)"""sem = asyncio.Semaphore(max_concurrent)async def run(task):async with sem:for attempt in range(RETRY_LIMIT + 1):try:return await asyncio.wait_for(executor(task), timeout=SUBTASK_TIMEOUT_S)except asyncio.TimeoutError:if attempt == RETRY_LIMIT:# Timeout fallback: use partial results + a prompt to continuereturn task.partial_result()return await asyncio.gather(*(run(t) for t in tasks))
Companion quota mechanisms matter just as much:
- Per-user quota: cap the number of concurrent tasks and total subtasks for the same user, so a single heavy user cannot exhaust global resources;
- Task priority: interactive requests take priority over background batch jobs;
- Execution deadline: set an overall timeout for the top-level task, with subtasks inheriting the remaining time budget, so a slow subtask does not drag down the whole;
- Idempotency and checkpoint resume: key each subtask in the cache by “input hash + parameters,” so repeat calls hit directly; persist intermediate state for long tasks so they can resume after an instance restart and avoid recomputation.
6.2 Operator-level optimization: raising throughput without sacrificing quality
| Optimization | Common approach | Caveats |
|---|---|---|
| Continuous batching | Dynamic batch assembly to raise decode throughput | Set a max batch size to prevent longer queue latency |
| Pooled isolation | Separate resource pools for long and short requests | Prevent long requests from blocking interactive ones (head-of-line blocking) |
| Prefix cache reuse | Share system prompts and tool definitions, computed once | Manage cache capacity and eviction policy |
| Quantization and sparsification | Low-precision inference such as INT8/FP8 | Run quality regression before launch to prevent accuracy loss |
| Speculative decoding | Small model drafts, large model verifies | Tune draft length to your business distribution |
| Result caching | Return historical results for identical inputs | Set a sensible TTL to avoid stale info |
These optimizations are general engineering practices at the inference-service layer. Teams can build an elastic, observable inference foundation from their own resources, or with general cloud capabilities such as container platforms, message queues, and object storage.
Observability and the capacity loop: making “scaling up” an everyday capability
7.1 Four core observability dimensions
- Traffic: QPS, concurrent sessions, input/output token rates;
- Queueing: queue length, wait-time distribution (P50/P95/P99);
- Resources: GPU utilization, GPU memory usage, KV Cache hit rate, batch size;
- Quality: output quality sampling, user adoption rate, negative human-feedback rate.
Pay special attention to KV Cache hit rate and batch size. High utilization with a low hit rate often means a large share of compute is being wasted on repeated computation — a problem scaling alone cannot fix.
7.2 The capacity operations loop
Traffic forecast → capacity baseline → elastic policy → load-test validation → live monitoring → retrospective tuning▲ │└──────────────────────────────────────────────────────────────────────────────┘
- Traffic forecasting: time-series forecasting by holiday, weekday, and morning/evening peak, adjusting baseline capacity ahead of time;
- Routine load testing: regularly load-test in staging against the production traffic profile to recalibrate the single-instance throughput baseline;
- Failure drills: periodically simulate scale-up failures, availability-zone outages, and dependency timeouts to verify that degradation and recovery logic behave as expected;
- Retrospective mechanism: after each peak event, produce a capacity-gap attribution, an assessment of degradation effectiveness, and improvement items, forming a trackable loop.
7.3 Choosing cloud elastic capabilities
In a multi-cloud or hybrid-cloud environment, elastic compute can be configured by task profile:
- Interactive inference: latency-sensitive and hard to interrupt, suited to stable reserved compute;
- Async batch processing and offline evaluation: interrupt-tolerant, suited to using spot instances to cut costs, paired with task checkpoints and resume mechanisms.
With general capabilities from cloud platforms such as Baidu AI Cloud — elastic containers, message queues, logging services, and monitoring/alerting — teams can quickly stand up a “baseline + elastic + buffer” resource structure and a unified observability system. The core principle: make elasticity part of the product, not a fire drill after a failure.
The business view: capacity itself is part of the product experience
Back to the original question. When a company is willing to guide users to try another option during overload, it usually reflects a few considerations.
8.1 Signaling product confidence by “ceding traffic”
By the intuition that retention is everything, pushing users toward an alternative seems almost incomprehensible. But from a signaling standpoint, letting users leave actually shows confidence in the product’s long-term retention. This posture conveys more conviction to investors and users than any parameter ranking on a launch stage. The prerequisite is clear: that confidence has to be backed by capacity and stability, or it will only amplify a negative experience.
8.2 Candid communication as differentiation against “marketing fatigue”
When launch events are saturated with claims like “leading across the board” or “first on every leaderboard,” users have developed obvious marketing fatigue. A plain-spoken, honest “we’re scaling up, try another tool for now” is more likely to earn goodwill precisely because it is rare. It signals that the team faces problems honestly and values the real user experience. This brand personality is often harder to copy than raw performance leadership.
8.3 The overload itself is the most honest demand validation
- A real test of willingness to pay: if a feature goes unused, its servers never get hammered. Peak-time queuing means users are using it — and using the most compute-intensive premium features.
- A corrective signal for investment direction: when the marginal returns of paid acquisition diminish, shifting resources from channel buying toward model R&D and engineering efficiency is a realistic choice for many teams. This overload actually provided the most direct evidence for that direction.
- Validation of product positioning: moving from “better at chatting” to “better at getting work done” means agent clusters, code execution, office automation, and other compute-heavy scenarios become the main battleground. Capacity planning for these scenarios must be extrapolated on the basis of “per-user compute several times that of conversation.”
8.4 Cost accounting: pay for growth, not for waste
The cost question ultimately reduces to a simple formula:
Cost per effective request = total resource cost ÷ number of effectively completed requests
Here “effectively completed” means requests that did not time out and whose quality was not degraded. The optimization focus should be on raising the denominator:
- Routing and caching let more requests complete at lower cost;
- Operator optimization and pooled scheduling raise single-instance throughput;
- Elastic resources are used on demand only during high-value periods.
Conversely, if degradation causes many requests to finish at low quality, the unit cost actually rises, because the resources were spent but the value was not delivered.
Summary and a practical checklist
A peak-hour compute shortage is an inevitable test as an AI product moves toward large-scale adoption. It tests not only the number of GPUs but the maturity of the engineering system. Teams should self-check against this list:
- SLOs first: define latency, success-rate, and degradation-rate targets per interaction type, measured at P95/P99;
- Throughput baseline: regularly load-test “tokens/s per instance” and recalibrate as the model version changes;
- Capacity structure: build a three-tier “baseline + elastic + buffer” resource structure, with load balancing that supports smooth scale up and down;
- Tiered degradation: implement a state machine with hysteresis and dynamic adjustment, with degradation information transparent to users;
- Model routing: control cost with difficulty judgment and cache-first, while keeping core-capability quality intact;
- Agent constraints: set concurrency caps, timeouts, quotas, and execution deadlines for multi-agent parallelism;
- Operator optimization: apply continuous batching, prefix caching, quantization, and similar techniques under quality regression;
- Observability loop: cover all four metric families — traffic, queueing, resources, quality — with load testing and failure drills;
- Cost metric: evaluate spend by “cost per effective request,” not just the total resource bill.
Back to that opening line — “we’re scaling up, try another tool for now.” For users, it is timely information; for the team, it is a public commitment. Whether it can be honored depends on whether there is a load-tested capacity baseline, a degradation strategy with hysteresis, and elastic resources that can be tapped quickly. Turn “temporary scaling” into “everyday elasticity,” and users won’t be forced to leave at peak. Only then is every instance of “the servers got hammered” genuinely good news.
