Large Language Model Rate Limiting Design: Capacity Planning, Circuit Breaking, and Compensation Strategies

    When deploying large language models, unexpected traffic surges and heavy inference loads often outpace GPU scaling capabilities. This article explores core strategies for implementing proactive rate limiting designs, covering pre-launch stress testing, dynamic scaling thresholds, rate limiting and circuit breaking parameters, unified return codes, and post-incident review templates to maintain overall system availability while minimizing user impact.

    After a new model is released, access volume and inference loads frequently exceed expectations simultaneously. The larger the model parameters and the longer the context, the more VRAM and computation time a single request consumes. Because GPU cluster scaling speeds cannot keep pace with traffic changes, rate limiting becomes almost inevitable.

    Rather than waiting to remedy rate limiting after it occurs, it is better to proactively define capacity, parameters, return codes, and compensation rules. Rate limiting is a form of degraded service with loss; it rejects a portion of requests to preserve overall availability. Teams must focus on two key tasks: minimizing the impact of degradation on users, and compensating affected users afterward using transparent rules.

    1. Pre-launch Stress Testing: Unified Standards for Reproducible Metrics

    Metrics without unified standards cannot be compared horizontally or used to make scaling decisions. This table should be updated for every model version change. The values below are placeholders and must be based on actual test results.

    Model Version Hardware Input Length Output Length Distribution Batch Config Concurrency TTFT P95/P99 (ms) Output Throughput (tokens/s) Peak VRAM (GB) Error Rate Pass Criteria
    TBD TBD 4K TBD TBD TBD TBD TBD TBD TBD See below
    TBD TBD 32K TBD TBD TBD TBD TBD TBD TBD See below
    TBD TBD 128K TBD TBD TBD TBD TBD TBD TBD See below

    Fill in the table according to the following standards:

    • Measurement Method: Time to First Token (TTFT) is measured from when the client sends the request to when it receives the first output token. Output throughput is calculated as the total tokens generated during the steady-state period divided by the duration.
    • Sampling Window: Run each concurrency tier for at least 30 minutes. Discard the first 5 minutes of warm-up data and only calculate statistics for the steady-state window.
    • Statistical Standard: P95 and P99 are calculated based on all successful requests within the window. Error requests are tracked separately and excluded from latency percentiles.
    • Pass Criteria: Derived from product SLOs (e.g., TTFT P99 must not exceed an agreed value, and the error rate must not exceed an agreed limit). Criteria should be confirmed jointly by product and R&D teams, not reverse-engineered from stress test results.
    • Testing Tools: The tool must simulate the real distribution of request lengths and use a fixed random seed to ensure fair comparisons between different versions. The same version must be tested under identical hardware and scheduling configurations; otherwise, results are not comparable.

    2. Scaling Thresholds: Example Values Requiring Business Baseline Calibration

    The following thresholds are examples only. Every team should determine their own values based on their baselines, traffic curves, and cost budgets.

    • Begin pre-scaling when the queue wait time P95 exceeds 50% of the target value.
    • Trigger the scaling process when GPU utilization remains above 75% for 5 consecutive minutes.
    • Calculate the scaling lead time based on the cold start duration. If bringing a new node online takes 15 minutes, the water level must leave at least 15 minutes of growth margin.
    • Reserve backup compute power proportionally to recent peaks. Large events require separate assessments to confirm whether the reserved cost is justified.

    The more capacity you reserve, the higher the idle cost. The less you reserve, the higher the risk of triggering rate limits. A common practice is to maintain ample reserves during peak hours, elastically scale down during off-peak hours, and set independent quotas for large events.

    Scaling can also fail, for example, due to insufficient resources or image pull timeouts. Teams need to set a scaling timeout. Upon timeout, the system should roll back or switch to a backup cluster and notify on-call personnel for manual confirmation. Alert thresholds, duration, and scaling processes must be bound together. If the same alert triggers multiple times, it should be escalated to manual handling to prevent infinite automatic scaling retries.

    3. Rate Limiting and Circuit Breaking: Translating Principles into Parameters

    The table below lists the main parameters. Values must be determined based on stress test results; examples are for reference only.

    Parameter Function Example Value Design Basis
    Tenant Token Bucket Rate Limits the average request rate per tenant Set by plan, e.g., 10 req/s Consistent with plan commitments
    Burst Capacity Allows a short-term excess of requests over the average rate 2 to 3 times the average rate Based on business burst duration and acceptable queue length
    Tenant Concurrency Limit Limits the number of concurrent requests undergoing inference per tenant Set by plan, e.g., 4 or 16 Based on per-GPU concurrency capacity and VRAM headroom
    Global Queue Limit Rejects new requests when the queue exceeds this length Reverse-calculated from the maximum acceptable wait time Prevents unbounded queue wait time growth
    Per-Tenant Queue Share Limits the queue slots occupied by a single tenant e.g., 30% of the global queue Ensures tenant fairness and prevents one tenant from maxing out the global queue
    Queue Timeout Returns a failure if a request waits longer than this duration e.g., 30 seconds Return before client timeout to avoid useless inference
    Circuit Break Trigger Error rate or queue wait time exceeds threshold for a set duration Error rate > 20% for 60 seconds Filters transient fluctuations and prevents false triggers
    Half-Open Probe Allows a small number of requests after circuit break to test recovery Allow 5% traffic, restore upon consecutive successes Gradual recovery to prevent instantly crashing the backend upon restoration

    4. Unified Return Codes

    Clients should handle rate limiting differently depending on the cause. Once return codes are unified, callers can make correct retry decisions.

    Scenario HTTP Status Code Business Error Code Response Header Client Recommendation
    Tenant exceeds quota or concurrency limit 429 RATE_TENANT_LIMIT Retry-After Back off per Retry-After, or reduce request rate
    Global queue is full 503 RATE_GLOBAL_QUEUE_FULL Retry-After Retry with exponential backoff and random jitter
    Queue timeout 503 RATE_QUEUE_TIMEOUT Retry-After Determine whether to retry based on business logic; avoid instant replays
    Circuit breaker open 503 RATE_CIRCUIT_OPEN Retry-After Wait for circuit breaker recovery; do not resubmit

    A 429 indicates that the caller has exceeded their own quota, while a 503 indicates overall server overload or temporary unavailability. Client retries should have a maximum limit and use exponential backoff with random jitter. Non-429 4xx errors should not be automatically retried. For non-idempotent requests, the caller must include an idempotency key to prevent duplicate inference or billing caused by retries.

    5. Post-Incident Review Template

    After each rate limiting or circuit breaking event, the team should complete a review within an agreed timeframe. The template should include at least the following fields:

    1. Incident ID and severity level.
    2. Impact window: Start time, end time, and the basis for determination (e.g., which alert fired or which metric crossed the threshold).
    3. Impact scope: Number of affected tenants, number of rejected requests, and the rejection percentage, broken down by tenant and model version.