画板 1

    Designing LLM Usage Quotas: Fixed Windows and Sliding Limits

    5 hr ago0 views

    LLM inference cost scales with input and output length, so request-count limits alone cannot protect shared capacity. This article explains token-based quotas, compares fixed windows, sliding windows, token buckets, and leaky buckets, and covers metering, periodic resets, usage visibility, and implementation requirements.

    1. Why compute capacity needs quotas

    The cost of LLM inference depends directly on the length of the input and the output. Within the same service, a short question-and-answer request and a long-document analysis can consume tens of times more resources than each other. If traffic is limited only by request count, a small number of long requests can saturate the inference cluster and raise latency for everyone else. Providers therefore usually treat tokens as the basic unit of measurement and set a consumption cap for each account over a defined period.

    Quotas mainly address three problems:

    • Protect shared compute so one user cannot crowd out others.
    • Keep cost and usage predictable.
    • Draw clear service boundaries for different paid tiers.

    2. Common rate-limiting algorithms

    2.1 Fixed-window counting

    Time is divided into fixed-length windows, such as one window per hour. Usage is accumulated inside the window, and requests are rejected once the cap is exceeded. The counter resets when the window ends. This is simple to implement, but near a window boundary a user can consume close to twice the cap in a very short time and create a burst of load.

    2.2 Sliding window

    Usage is counted over a fixed duration ending at the current time. This removes the spike at window boundaries. The tradeoff is that finer-grained time slices or timestamps must be stored, which raises storage and compute cost.

    2.3 Token bucket

    Tokens are added to a bucket at a fixed rate. Each request consumes the corresponding number of tokens, and the bucket capacity sets the largest burst allowed. A token bucket constrains both the average rate and short bursts, which fits conversational interaction.

    2.4 Leaky bucket

    Requests leave at a constant rate. Excess requests are queued or dropped. Output is the smoothest of these approaches, but queuing increases wait time for long requests.

    The pseudocode below shows the core steps of a token-bucket decision:

    1. function allow(user, cost_tokens, now):
    2.     bucket = load_bucket(user)
    3.     elapsed = now - bucket.last_refill
    4.     bucket.tokens = min(bucket.capacity, bucket.tokens + elapsed * bucket.rate)
    5.     bucket.last_refill = now
    6.     if bucket.tokens >= cost_tokens:
    7.         bucket.tokens -= cost_tokens
    8.         save_bucket(user, bucket)
    9.         return ALLOW
    10.     save_bucket(user, bucket)
    11.     return REJECT

    In a real system, the read, calculation, and write-back must happen in a single atomic operation. Otherwise concurrent requests can all pass the check and actual consumption can exceed the quota.

    3. Difficulties of metering by tokens

    Estimating token consumption has two difficulties.

    First, input length can be estimated with a tokenizer before the request, but output length is known only after generation finishes. A common approach is to reserve quota for the maximum output length in advance, then refund the difference based on actual usage after generation.

    Second, the same text can be segmented differently by different models. Billing and quotas should therefore use the token count measured by the server. A client estimate is only a hint.

    Metering should be unified at the gateway, with business services calling a single deduction interface. This avoids inconsistent measurements across services and makes later rule changes easier.

    4. Periodic allowances and reset rules

    Many services reset allowances on a billing cycle, such as at the start of each month. The design needs to make these rules explicit:

    • Which time zone defines the cycle boundary.
    • Whether unused allowance carries over, and what the carryover cap is.
    • How remaining allowance is converted when a user upgrades or downgrades.
    • How in-flight requests are settled when the allowance resets.

    If nothing carries over, users may concentrate consumption at the end of the cycle and create a load spike. Unlimited carryover removes the constraint. A common compromise is a limited carryover ratio plus an expiration time for the carried amount.

    5. Usage visibility

    Users need to see their consumption at any time; otherwise the quota is a black box. A console should provide:

    • The share already used in the current window and the reset time.
    • Usage trends for the current cycle, broken down by day or by model.
    • Alerts as the cap approaches, for example one notification at 80% and another at 100%.

    Usage shown in the interface should come from the same records as billing data, so the display and the bill do not disagree.

    6. Implementation requirements

    1. Store quota state in a low-latency key-value store and set a reasonable expiration policy.
    2. Deductions must be atomic. If a request fails, deducted allowance must be refunded as compensation.
    3. Record a request identifier for every deduction so retries can be recognized and handled idempotently.
    4. Separate the rate-limit decision from business processing. The gateway should return a consistent error code and the remaining allowance so clients can back off and retry.
    5. Roll out quota-rule changes gradually. Before a new rule goes live, observe rejection rates and complaint volume on a subset of accounts.

    When migrating from an old rule, the new rule can first run in shadow mode and compute decisions in parallel. Compare the differences before switching over. This surfaces cases where normal users would be rejected by mistake and lowers launch risk.

    09040904

    Top articles