# LLM Inference Capacity Planning: Compute Quotas, Admission Control, and Priority Scheduling
When a new model release triggers a request surge and the only remaining lever is to temporarily close sign-ups for new users, the surface event looks like a commercial decision — but the underlying issue is a capacity planning failure. This article explains why generative inference consumes compute differently from a conventional web API: high per-request cost across the Prefill and Decode phases, KV Cache memory that scales with context length and concurrent sessions, and demand that spikes by orders of magnitude without warning. It then walks through the engineering practices that address the resulting mismatch: estimating per-instance concurrency from a memory budget, giving paying tenants explicit scheduling priority, splitting entitlements by consumption profile instead of selling unlimited shared capacity, and building a three-layer stack of rate limiting, admission control, and priority scheduling backed by observability, elastic scaling, and graceful degradation.
1. Background: Why Inference Compute Is a Hard Scarce Resource
When an AI model service is hit by a traffic surge after a new version ships and ends up temporarily closing its new-user entry point, the event looks like a business decision. In practice, it is a capacity planning failure coming to light. Understanding this class of problem starts with a clear view of how inference differs from a traditional web API in the way it consumes compute.
Traditional APIs have a stable, linearly extrapolable compute profile. Generative inference has three characteristics that break that assumption:
- High cost per request: a single generation request passes through both the Prefill (prompt processing) and Decode (token-by-token generation) phases. Decode is memory-bandwidth bound, so GPU memory bandwidth often becomes the bottleneck before raw compute does.
- Context length amplifies memory usage: KV Cache grows linearly with sequence length and with the number of concurrent sessions. Long contexts, multi-turn conversations, and agent clusters multiply the memory requirement.
- Highly unpredictable demand: marketing campaigns, new model launches, and social media attention can produce order-of-magnitude traffic jumps within minutes. Historical averages are nearly useless for peak planning.
GPU cluster expansion — procurement, racking, tuning — is measured in weeks or months, while demand spikes are measured in minutes. This mismatch between supply and demand time scales is the root cause behind nearly every compute shortage event.
2. The Core Challenges of Capacity Planning
Designing a robust capacity strategy requires quantifying a few key tensions first:
- The gap between peak and average. Provision for the peak and daily utilization stays low for long stretches; provision for the average and a single burst breaks through the ceiling. The difference between the two determines how much buffer capacity you need.
- Two constraints at once: compute and memory. Even with spare FLOPS, memory may run out first because of KV Cache across concurrent sessions. Capacity assessment must cover both dimensions.
- Heterogeneous workloads. Pure chat traffic produces a smooth resource curve. Agent-style workloads trigger repeated model calls, tool calls, and state maintenance in sequence, and their per-user compute cost can be tens of times that of ordinary chat.
A rough way to estimate single-instance concurrency:
def estimate_max_concurrency(gpu_memory_gb, model_weights_gb,per_token_kv_cache_gb, max_seq_len, safety=0.85):"""Estimate the number of concurrent sessions a single GPU can holdunder the current memory budget.- gpu_memory_gb: memory capacity of a single GPU- model_weights_gb: memory consumed by the model weights- per_token_kv_cache_gb: KV Cache memory cost per token- max_seq_len: maximum context length reserved per session"""usable = gpu_memory_gb * safety - model_weights_gbper_session = per_token_kv_cache_gb * max_seq_lenif per_session <= 0:return 0return int(usable // per_session)
The calculation makes one fact clear: raising the maximum context length reduces the available concurrency proportionally. This is why providers usually set tiered quotas on context length and concurrent sessions rather than leaving them unlimited.
3. Resource Guarantees Under a Subscription Model
Once compute becomes tight, “suspend new-user onboarding and protect existing users” sounds counterintuitive, but it follows from the internal logic of a subscription business model.
A subscription is not built on a single transaction but on repeat renewal. Users renew because they believe the service will stay reliably available. If requests start failing and latency climbs while capacity is strained, the first casualty is precisely the trust of paying users — the foundation of renewal. “Stabilize the core first” is therefore the standard answer under a subscription model: spend limited compute defending the SLA you have already committed to, rather than diluting resources until nobody is satisfied.
From an engineering standpoint, this requires an explicit notion of tenant priority. Paying tenants’ requests should get higher scheduling weight and stricter availability guarantees, while new-user or free-tier requests can be downgraded or queued when capacity runs short.
4. Differentiated Quotas: Why Entitlements Should Be Split
The most valuable technical insight in the original discussion is this: different users have completely different compute consumption profiles, and selling them together in one bundle is neither fair nor sustainable. The point matters a great deal in engineering terms.
Consider two user classes:
- Chat users: moderate request rate, short contexts, low cost per request.
- Coding/Agent users: dense requests, long contexts, frequent tool calls and code generation — a single session can trigger a large amount of background GPU consumption.
Put both into one “unlimited” plan and you get a textbook tragedy of the commons: heavy users draw down shared resources without restraint, light users suffer as a side effect, cost cannot be attributed, and capacity cannot be predicted.
The sound approach is to split entitlements by consumption profile and give each one its own quota:
| Entitlement | Typical workload | Quota dimensions | Rate limiting strategy |
|---|---|---|---|
| Basic chat | Multi-turn Q&A | Requests per day, context length | Token bucket |
| Coding / Agent | Code generation, tool orchestration | Concurrent sessions, background tasks | Concurrency semaphore + priority queue |
Splitting entitlements is fundamentally about converting an uncontrollable shared cost into a measurable, cappable, priceable private quota — which is what restores fairness and sustainability.
5. Implementation: Admission Control and Priority Scheduling
Quota design only matters once it lands on schedulable mechanisms. The core is three layers: rate limiting → admission control → priority scheduling.
The first layer uses a token bucket to control entry rate and absorb bursts:
import timeclass TokenBucket:"""Token bucket rate limiting: refill tokens at a constant rate and allow bursts."""def __init__(self, capacity, refill_rate):self.capacity = capacity # bucket size, which bounds the allowed burstself.refill_rate = refill_rate # tokens added per secondself.tokens = capacityself.last_refill = time.monotonic()def acquire(self, tokens=1):now = time.monotonic()elapsed = now - self.last_refillself.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)self.last_refill = nowif self.tokens >= tokens:self.tokens -= tokensreturn Truereturn False
The second layer admits requests by priority within the compute budget, so paying tenants go first:
import heapq, itertoolsclass PriorityAdmission:"""Priority admission scheduling: admit by priority within budget; lower priority value wins."""def __init__(self):self._counter = itertools.count() # guarantees FIFO within the same priorityself._heap = []def submit(self, priority, tenant_id, est_cost):heapq.heappush(self._heap,(priority, next(self._counter), tenant_id, est_cost))def schedule(self, budget):admitted, used = [], 0while self._heap:priority, _, tenant, cost = self._heap[0]if used + cost > budget: # stop admitting once remaining compute is insufficientbreakheapq.heappop(self._heap)used += costadmitted.append(tenant)return admitted, budget - used
The third layer adds multi-tenant resource isolation — for example, dedicated instance groups or memory quotas per tenant — so that a single tenant cannot exhaust shared resources. On container platforms this is typically done with ResourceQuota and PriorityClass to enforce hard isolation and preemption policies.
6. Observability and Elastic Scaling
Capacity planning is not a one-off exercise but a continuous loop. A robust service wires the following metrics into logging, monitoring, and alerting:
- Utilization metrics: GPU compute utilization, memory occupancy, request queue depth.
- Service quality metrics: time to first token (TTFT), end-to-end P99 latency, rate limit rejection rate.
- Capacity forecasting metrics: request growth slope per unit time, remaining available quota.
On top of these, you can define multi-level alerts and elasticity policies:
- When memory utilization stays above a threshold, trigger horizontal scaling by adding inference instances.
- When queue depth exceeds its limit, automatically tighten the admission rate for lower-priority tiers.
- When scaling still cannot arrive in time, enter graceful degradation mode: shorten the maximum context, reduce generation length, queue instead of failing outright.
The value of this loop is turning “compute is tight” from a reactive, after-the-fact scramble into a controlled process that is observable ahead of time and schedulable in the moment.
7. Summary
Behind a single “we’ve suspended new sign-ups” notice sits either the absence or the presence of a whole capacity governance capability. For developers, four threads are enough to build a resilient foundation for an LLM service:
- Quantify the load: recognize that inference is rigidly constrained in both compute and memory, and make capacity decisions from data rather than intuition.
- Protect the core: under a subscription model, anchor on the SLA and renewal, and use tenant priority to keep existing users’ experience intact.
- Split the quotas: design separate entitlements and rate limits for chat and coding/Agent consumption profiles, restoring fairness and sustainability.
- Close the operational loop: combine monitoring, alerting, elastic scaling, and graceful degradation into a continuously tuned capacity cycle.
Compute scarcity is a long-term condition of the generative AI era. What actually differentiates services is not how many GPUs you can pile up for the peak, but the engineering ability to allocate resources in an orderly way, guarantee fairness, and degrade gracefully when resources run short.
