Service Governance and Capacity Planning Under Compute Supply Scarcity

1. The Technical Essence: Not an Outage, but a Capacity Governance Gap

Recently, an AI service halted new membership registrations due to compute scarcity following a new model release. While the public perceived this as a “happy problem” of rapid growth, an engineering perspective reveals a highly typical issue: a lack of an adjustable buffer mechanism between service capacity and user growth.

The fundamental difference between LLM inference services and traditional web services lies in resource rigidity. A standard HTTP request consumes millisecond-level CPU time slices, whereas a long-context inference request occupies the compute and VRAM of an entire accelerator card for durations ranging from hundreds of milliseconds to tens of seconds. When a new model’s capability leap triggers non-linear traffic growth, the resource pool expansion rate cannot keep up with the request arrival rate. Accelerator procurement, rack mounting, driver adaptation, model loading, and stress testing take weeks or even months, whereas a post-release traffic spike can peak in a matter of hours.

This creates a structural contradiction: extreme demand-side elasticity meets highly inelastic supply. Under these constraints, halting membership registrations is essentially a coarse-grained admission control method, brutally forcing request volumes back below the capacity watermark. It works, but at the cost of user experience and business credibility. A more mature approach dismantles this blanket ban into a tiered, observable, and auto-adjusting capacity governance system.

2. Capacity Evaluation: Calculating Requests Per Card

The first step in capacity governance is not scaling out, but quantification. Many teams instinctively add machines when traffic spikes, but without knowing the actual throughput limit of a single instance, capacity planning is mere guesswork.

Capacity evaluation for LLM inference requires distinguishing three key metrics:

• Theoretical Compute Ceiling: Determined by the accelerator model, VRAM bandwidth, model parameter count, and precision. It is typically measured in tokens processed per second (tokens/s).
• Effective Throughput: The actual throughput achieved while meeting SLAs (e.g., P99 Time To First Token < 2 seconds). This is usually significantly lower than the theoretical value.
• Concurrency Capacity: The number of simultaneous requests that can be accommodated within a given VRAM budget, leveraging techniques like batching and paged attention.

A simplified capacity estimation model looks like this:

def estimate_capacity(tokens_per_sec_per_card, card_count,
                      avg_output_tokens, target_p99_latency_s,
                      safety_margin=0.7):
    # Effective throughput per card under latency constraints
    effective_tps = tokens_per_sec_per_card * safety_margin
    # Total cluster throughput (tokens/s)
    cluster_tps = effective_tps * card_count
    # Average time per request (seconds)
    per_req_time = avg_output_tokens / effective_tps
    # Concurrency limit to meet latency constraints (Little's Law approximation)
    max_concurrency = int(cluster_tps * target_p99_latency_s / avg_output_tokens)
    # Serviceable requests per second
    qps = cluster_tps / avg_output_tokens
    return {
        "cluster_tokens_per_sec": cluster_tps,
        "max_concurrency": max_concurrency,
        "estimated_qps": round(qps, 2),
        "per_request_seconds": round(per_req_time, 3),
    }

It is crucial to emphasize that this model is just a starting point. True capacity must be calibrated via stress testing. This means generating traffic using a representative distribution of input lengths (rather than a single length) and observing the inflection point where latency spikes as concurrency increases. Concurrency before the inflection point is the safe zone; concurrency beyond it triggers a queuing avalanche. This is especially critical in long-context scenarios, as individual requests hold VRAM for extended periods, and queue backlogs can rapidly amplify tail latency.

3. Tiered Governance: Splitting the “Halt” into Four Gates

Blanket admission control is criticized because it treats all users—regardless of priority—equally, locking them out indiscriminately. The logical approach is to build a multi-layered control mechanism, ensuring limited compute prioritizes high-value, predictable requests.

3.1 Layer 1: Gateway Rate Limiting and Queuing

Implement token bucket-based rate limiting at the gateway layer, distinguishing quotas across three tiers: free users, paid members, and enterprise clients. Requests exceeding the quota are not immediately rejected; instead, they enter a bounded queue and return the queue position with an estimated wait time. The key to a bounded queue is the “bound” itself—the queue length must match the backend’s processing capacity. Otherwise, queuing merely shifts timeouts from the gateway to the backend.

# Pseudo-code: Tiered token bucket + bounded wait queue
class TieredRateLimiter:
    def __init__(self, tiers):
        self.buckets = {t: TokenBucket(rate=r, burst=b)
                        for t, (r, b) in tiers.items()}
    def admit(self, user_tier, queue):
        if self.buckets[user_tier].try_consume(1):
            return "admit"
        if queue.size() < queue.capacity and user_tier != "free":
            queue.push(user_tier)
            return "queued"
        return "reject_with_retry_after"

3.2 Layer 2: Request Scheduling and Prioritization

Once inside the inference cluster, the scheduler should support priority preemption: high-priority requests can jump ahead of low-priority ones. During resource scarcity, low-priority requests are degraded (e.g., by reducing the maximum output length or routing to a smaller model). This requires the inference framework to support continuous batching, allowing requests of varying priorities to be dynamically stitched within the same batch without waiting for an entire batch to complete.

3.3 Layer 3: Graceful Degradation and Fallback

When the capacity deficit cannot be bridged by scheduling alone, a clear degradation strategy must be in place. Temporarily throttle non-core functions (like long-document parsing or complex multi-turn reasoning) to concentrate resources on the core Q&A pipeline. Truncate or segment ultra-long context requests. Provide clear messaging and retry suggestions for requests that time out in the queue. Degradation is not a failure; it converts an uncontrollable systemic crash into a controllable, partial loss.

3.4 Layer 4: Dynamic Quota and Admission Adjustment

Whether to allow new memberships is fundamentally a quota issuance decision. Rather than suspending registrations in a panic after capacity runs out, establish a dynamic quota mechanism based on capacity watermarks: automatically release new quotas when cluster utilization falls below a threshold, and restrict issuance with advance notice when it exceeds the threshold. This requires capacity monitoring metrics (GPU utilization, VRAM usage, queue depth, P99 latency) to integrate with business-side provisioning APIs, forming a closed loop.

4. Elastic Scaling: Buying Time Instead of Buying Cards

Weak supply-side elasticity is a reality, but it is not insurmountable. Viable strategies include:

• Hybrid Deployment: Deploy inference services across both on-premises clusters and elastic cloud resources. Routine traffic is handled by the on-premises cluster, with peak overflow directed to the cloud. The challenge lies in model distribution and cold starts—mitigated by pre-warming images, pre-loading weights, and maintaining a minimal warm pool.
• Model Tiering: Prepare models of different scales for different SLA requests. Route lightweight requests to smaller models and complex requests to larger ones via a routing layer based on request characteristics. This significantly boosts overall throughput without adding hardware.
• Quantization and Distillation: Reduce the compute consumption per request without significantly compromising performance, effectively acting as capacity expansion. Ensure that precision loss from quantization is validated against business metrics via regression testing, rather than solely looking at perplexity.
• Capacity Reservation and Peak Shifting: Schedule deferrable batch processing tasks (like offline evaluations or data cleaning) during off-peak hours to flatten the peak pressure on online inference.

5. Observability: No Metrics, No Governance

All the mechanisms described above rely on accurate, real-time metrics. It is recommended to establish monitoring across at least the following dimensions:

Dimension	Key Metrics	Purpose
Resource	Accelerator utilization, VRAM usage, temperature, power	Determine if physical bottlenecks are reached
Service	QPS, concurrency, P99 TTFT (Time To First Token), P99 total latency	Determine if SLAs are breached
Queue	Queue depth, wait time distribution, rejection rate	Determine if rate limiting is too strict or too loose
Business	Request ratio per tier, quota consumption rate	Determine if quota issuance is reasonable

Beyond metrics, establish linkage rules between capacity watermarks and business actions. For example: “If cluster utilization exceeds 85% for 10 consecutive minutes, trigger quota tightening and an expansion ticket.” These rules should be documented in the operations manual and rehearsed periodically to avoid ad-hoc decision-making during traffic peaks.

6. Conclusion: Crystallizing Emergency Actions into Governance Capabilities

Returning to the original question: is halting membership registrations reasonable? From a purely technical standpoint, it is a valid emergency response under resource constraints. However, from an engineering maturity perspective, it indicates that the capacity governance system has not yet closed the loop. A mature AI service should absorb traffic shocks smoothly in the early stages of growth via rate limiting, queuing, degradation, and dynamic quotas—downgrading “halting registrations” from a first reaction to a last resort.

For developers, the takeaway from this scenario is concrete: when designing any service reliant on scarce compute, define the capacity model first, then design admission and scheduling strategies, and finally tie the three together with observability. Compute scarcity is a long-term reality, but scarcity does not mean losing control—the difference lies in having a measurable, adjustable, and predictable governance mechanism.