Rate Limiting and Quota Ratio Design for LLM APIs

    This article explains how to keep short-window and long-window quotas for LLM APIs internally consistent. It covers proportional design for 5-hour, weekly, and monthly quotas; a dual-layer limiter implementation with rollback; and monitoring and autoscaling practices in cloud-native deployments. Readers will learn a lower-bound formula for weekly quota, monthly quota conversion considerations, dual-bucket implementation patterns, and pre-launch validation checks.

    For LLM API rate limits and plan quotas, the core issue is whether quotas across short and long windows are internally consistent. This article answers three questions: how to design 5-hour, weekly, and monthly quotas in proportion; how to implement dual-layer rate limiting in code; and how to implement monitoring and autoscaling in a cloud-native environment.

    1. Rate-Limiting Dimensions and Terminology

    Traditional cloud-native API rate limiting focuses mainly on QPS or concurrent connections to prevent system overload. LLM services must also account for token consumption because tokens map directly to GPU compute cost. As a result, most providers set quotas across multiple time windows. This article uses the following terms:

    • Short-window quota: the 5-hour quota, used to constrain dense calls within a short period and smooth out traffic spikes.
    • Long-window quota: the weekly total and monthly quota, used to control total consumption and long-term resource usage.

    The short-window quota smooths traffic, while the long-window quota controls cost. If the ratio between them is unreasonable, users may suddenly receive quota errors in the middle or latter part of the week.

    2. Deriving the Weekly Quota Lower Bound

    How many 5-hour windows can be fully used in a week?

    A day has 24 hours, and each 5-hour window occupies 5 hours. If windows are split by calendar day from midnight, each day can have at most 4 full windows; the remaining 4 hours are not enough to form another complete window. Therefore, 7 days can contain at most 28 full windows. If windows are not aligned to calendar days but roll continuously, 168 hours per week can accommodate at most 33 full windows (168 ÷ 5 ≈ 33.6).

    These numbers come from clock arithmetic, not from industry statistics.

    Formula

    Let the weekly quota be W, the 5-hour quota be H, the expected number of full windows per day be k, and the number of usage days per week be d. If normal usage should trigger the short-window quota first and the weekly quota should not run out before that, the following condition must hold:

    W ≥ k × d × H

    Here, k and d depend on business hours and SLAs. For example, nightly batch jobs may use only 1 window per day, while a continuously running coding assistant may approach full load.

    This is a lower bound. If a provider intentionally sets the weekly quota lower to control cost, the product documentation should clearly tell developers that the weekly limit will trigger before the 5-hour limit.

    Example calculation

    The following table provides a set of example parameters. The values are used only to illustrate the calculation process.

    Parameter Meaning Value
    H 5-hour quota 1 million tokens
    k Full windows per day 4
    d Usage days per week 5
    W lower bound k × d × H 20 million tokens
    M Calendar-month quota 90 million tokens
    Average weeks per month 365.25 ÷ 12 ÷ 7 about 4.35
    M converted to average weekly quota M ÷ 4.35 about 20.69 million tokens

    Under these parameters, a weekly quota of 20 million tokens satisfies the lower bound. Since continuous full-load usage is about 87 million tokens per month, below the monthly quota of 90 million tokens, the monthly limit will not trigger before the weekly limit.

    If the weekly quota is set to 12 million tokens, a user who runs full load 5 days per week and 4 windows per day will exhaust the quota after about 12 windows, meaning errors begin on the third workday. This is exactly the case where the weekly limit becomes the bottleneck early.

    3. Monthly Quota Conversion Basis

    A calendar month has 28 to 31 days. If it is simply calculated as 4 weeks, February and 31-day months will show noticeable deviations. The basis must be clarified during design:

    • Calendar month: resets at midnight on the first day of each month. The user experience is intuitive, but consumption may concentrate at the beginning of the month.
    • Rolling 30 days: quota is released 30 days after each consumption. This is smoother but more complex to implement.
    • Contract month: resets on the user’s subscription date. This suits paid plans, but each user has a different reset time and must be calculated separately.

    Whichever basis is used, the weekly quota and monthly quota should be compared under the same basis. The weekly quota W converted to monthly usage is W × 4.35. The monthly quota M should be no less than this value; otherwise, the monthly limit will take effect before the weekly limit, making the weekly limit meaningless. If M is far greater than W × 4.35, the monthly limit barely matters, and you should consider whether to keep this layer of restriction.

    4. Implementing Dual-Bucket Rate Limiting

    At the execution layer, a token bucket can implement the short-window limit, while a fixed-period counter can implement the long-window limit. The following code demonstrates the basic logic of dual-bucket deduction. The key point is to roll back deducted quota on failure to avoid incorrect deductions.

    1. import threading
    2. import time
    3. class TokenBucket:
    4. """Token bucket. capacity and tokens are in tokens; refill_rate is tokens added per second."""
    5. def __init__(self, capacity: float, refill_rate: float):
    6. self.capacity = capacity
    7. self.refill_rate = refill_rate
    8. self.tokens = capacity
    9. self._last_ts = time.monotonic()
    10. self._lock = threading.Lock()
    11. def _refill(self) -> None:
    12. now = time.monotonic()
    13. elapsed = now - self._last_ts
    14. self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
    15. self._last_ts = now
    16. def try_consume(self, amount: float) -> bool:
    17. with self._lock:
    18. self._refill()
    19. if self.tokens < amount:
    20. return False
    21. self.tokens -= amount
    22. return True
    23. def refund(self, amount: float) -> None:
    24. with self._lock:
    25. self.tokens = min(self.capacity, self.tokens + amount)
    26. class DualQuotaLimiter:
    27. """Dual-bucket rate limiting: short-window and long-window quotas must both be satisfied; if either fails, neither is deducted."""
    28. def __init__(self, short_bucket: TokenBucket, long_bucket: TokenBucket,
    29. base_short_rate: float, low_watermark: float = 0.1):
    30. self.short = short_bucket
    31. self.long = long_bucket
    32. self.base_short_rate = base_short_rate
    33. self.low_watermark = low_watermark
    34. self._lock = threading.Lock()
    35. def _throttle_factor(self) -> float:
    36. remaining_ratio = self.long.tokens / self.long.capacity
    37. if remaining_ratio >= self.low_watermark:
    38. return 1.0
    39. return max(0.1, remaining_ratio / self.low_watermark)
    40. def acquire(self, amount: float):
    41. """Return (HTTP status code, error code)."""
    42. if amount <= 0:
    43. return 400, "INVALID_AMOUNT"
    44. with self._lock:
    45. # When long-window balance is low, reduce short-window refill rate proportionally to slow consumption without hard blocking.
    46. self.short.refill_rate = self.base_short_rate * self._throttle_factor()
    47. if not self.short.try_consume(amount):
    48. return 429, "SHORT_QUOTA_EXCEEDED"
    49. if not self.long.try_consume(amount):
    50. self.short.refund(amount)
    51. return 429, "LONG_QUOTA_EXCEEDED"
    52. return 200, "OK"

    Pay attention to the following points when using this code.

    1. Parameter units. The short-window base_short_rate should be set to H ÷ 18,000, i.e., the number of tokens added per second, because 5 hours equals 18,000 seconds.
    2. The long-window bucket should not refill continuously. The weekly total quota should be reset wholesale at the period boundary. Therefore, set refill_rate to 0 when instantiating the long-window bucket, and reset tokens with a scheduled job at the period boundary.
    3. Concurrency. The locks above are effective only within a single process. In a multi-instance deployment, “check both buckets, deduct, and roll back on failure” should be written as a Redis Lua script and completed in one atomic call.
    4. Return values. A 429 indicates insufficient quota. It should also return a Retry-After response header, whose value can be calculated from the remaining quota and refill rate.

    5. Deployment in Cloud-Native Environments

    Rate-limiting logic needs to be distributed across multiple layers, with each layer taking on different responsibilities.

    Layer Purpose Key configuration Monitoring metrics Common errors
    API gateway Block over-limit requests at ingress Configure 5-hour and weekly limits by consumer and route 429 ratio, hit counts per rule Time zone mismatch between gateway and backend, causing incorrect reset time display
    Rate-limiting service Execute dual-bucket deduction and rollback Short-window capacity, long-window capacity, refill rate Balance distribution, rollback count Only deducting the short window, no rollback
    Logging and monitoring Observe consumption curves Sampling granularity, alert thresholds Ratio of weekly-limit consumption speed to 5-hour consumption Looking only at totals, not ratios
    Containers and autoscaling Increase backend supply capacity HPA metrics, instance minimum and maximum Inference queue length, instance count, unit cost Automatically loosening plan quotas after scaling

    Autoscaling requires special clarification. Kubernetes HPA increases inference instances, which changes backend supply capacity but does not change user plan quotas. To temporarily increase user quota during peak periods, the following conditions must all be met: the SLA allows it, the plan terms specify the rules for temporary increases, and a cost cap is set. Quota changes should be completed through configuration changes, with approval and audit records, and must not be triggered automatically by autoscaling events.

    Alerts should also be configured. When resource utilization exceeds a threshold, the number of scaling events increases abnormally within a short period, or the daily cost approaches the cap, operations personnel should be notified promptly.

    6. Pre-Launch Checklist

    1. Ratio validation: Confirm that the weekly quota is no less than k × d × H, that the monthly quota and the converted weekly quota match, and document the business assumptions.
    2. Dual-bucket rollback test: Simulate requests that exhaust the long-window quota, verify that the short-window quota is correctly refunded, and stress test concurrent deduction under multithreading or multiple instances.
    3. Consumption alert thresholds: Set alerts for the ratio of weekly-limit consumption speed to 5-hour consumption, and adjust quotas when the two cross or deviate from each other.

    After these three items are completed, the plan design can be considered validated. Quota ratios are essentially business decisions; code can only ensure that those decisions are executed accurately.