Designing Tiered Quotas for LLM Services: Rate Limiting and Over-Quota Handling

1. Why LLM Services Need Tiered Quotas

The compute cost of LLM inference is directly tied to input length, output length, and model size. If all requests share a single pool of resources, a small number of high-frequency users will saturate compute and degrade response times for everyone else. As a result, providers typically divide users into tiers such as free, basic, and professional, each with different limits, balancing cost, fairness, and experience.

For users, quota rules determine whether day-to-day usage runs smoothly. For designers, the rules must satisfy three requirements at once: they must be accurately metered, clearly explainable, and reliably enforced under high concurrency.

2. Breaking Quotas into Six Dimensions

2.1 Limiting Request Frequency with Time Windows and Token Buckets

The most common approach is to set a maximum number of requests per user within a time window. The window can be fixed, for example resetting at midnight each day, or sliding, always looking back over the most recent hours.

• Fixed windows are simple to implement, but near window boundaries instantaneous traffic can approach twice the limit.
• Sliding windows are smoother but require storing more timestamps and cost more to compute.
• Token buckets allow a certain amount of burst traffic and suit interactive chat. The bucket refills tokens at a fixed rate, and each request consumes one or more tokens.

In practice, the two are often combined: a token bucket controls QPS (requests per second), while fixed windows control RPM (requests per minute) and daily totals. The table below shows order-of-magnitude reference values as a design starting point; they do not represent the real parameters of any provider:

Tier	QPS	RPM	TPM (tokens per minute)	Typical use
Free	1	10	10,000	Trials and lightweight Q&A
Basic	5	60	100,000	Personal development and daily office work
Professional	20	600	1,000,000	Small teams and production workloads

These values need to be adjusted after load testing against model throughput and cost.

2.2 Metering High-Cost Capabilities Separately

Different capabilities vary widely in computational overhead. Deep reasoning modes generate large amounts of intermediate content, and a single call costs far more than ordinary chat. If both share one counter, ordinary users’ quotas will be overestimated in consumption, while high-cost capabilities become easy to abuse. The right approach is to give high-cost capabilities their own counting dimension and set limits at a level that matches their cost.

2.3 Handling Context Window Overflow

The context window is the maximum number of tokens a model can process in a single inference. Input content, conversation history, and model output all share this budget. When the limit is exceeded, there are three common approaches:

1. Truncate the earliest history;
2. Summarize the conversation history and replace the original text with the summary;
3. Split long documents into chunks, retrieve relevant passages on demand, and then inject them into the context.

There is no fixed conversion between tokens and characters. Taking the cl100k_base encoding in OpenAI’s tiktoken library as an example, in common Chinese text one Chinese character corresponds to roughly 1 to 2 tokens, depending on the characters, words, and punctuation. Different models’ tokenizers produce different results. Billing and limits should be counted with the tokenizer of the corresponding model; character-count estimates in documentation can only serve as a reference.

2.4 Designing Quotas for Image Generation

A single image generation involves far more computation than text generation, so most services limit by number of calls. For batch image scenarios such as e-commerce asset creation or advertising creative work, a fixed daily call cap quickly becomes a bottleneck. These users usually need pay-as-you-go billing or batch interfaces; simply raising the daily quota does not solve the problem.

2.5 Limiting File Uploads and Parsing

File processing includes parsing, chunking, embedding, and storage, and its cost grows with the number and size of files. Quotas can limit by file count or by total bytes; combining both is more robust. A per-file size cap is also needed.

Security risks must be guarded against as well. A compressed archive may be a decompression bomb: tiny in size but consuming large amounts of disk and memory once extracted. Therefore limit the compression ratio, the total bytes after extraction, and the nesting depth. Page counts and pixel dimensions of PDFs or images also need caps, to prevent large multimodal files from exhausting memory and making the service unavailable.

Parsing should go into an asynchronous queue. The upload endpoint only writes a task and returns a task ID, and workers consume the queue. Each task gets a timeout; on timeout it is terminated and marked as failed, so synchronous parsing does not block request threads.

2.6 Free Tier and Advertising Strategy

The free tier needs to cover part of the operating cost. Common measures include feature trimming, reduced limits, and displaying ads in the interface. Paid tiers usually offer ad removal, higher limits, and more features. The placement and frequency of ads directly affect the experience; users should be told clearly, and ads should not obscure core actions.

3. Engineering Implementation of Quota Deduction

Quota checks must happen on the request path, and the read, check, and increment steps must be atomic. If the balance is read first and deducted afterward, concurrent requests may all pass the check and cause overuse. Below is pseudocode independent of any tech stack:

function try_consume(user_id, dimension, cost):
    plan   = get_user_plan(user_id)
    rule   = plan.rules[dimension]
    now    = now_utc()
    window = current_window_start(rule.window_type, now)
    key    = build_key(user_id, dimension, window)
    result = counter.add_if_not_exceed(key, cost, rule.limit,
                                       expire_after = rule.window_length)
    if result.rejected:
        retry = seconds_until_window_end(rule.window_type, now)
        return deny(retry_after = retry, limit = rule.limit)
    return allow(remaining = rule.limit - result.total)
end function

In a multi-node cluster, counter.add_if_not_exceed is usually implemented as a Redis Lua script. Redis does not interleave other commands while executing a script, so the check and the increment cannot be interrupted:

-- KEYS[1]: counter key; ARGV[1]: cost of this call; ARGV[2]: limit; ARGV[3]: expiry seconds
local cost    = tonumber(ARGV[1])
local limit   = tonumber(ARGV[2])
local current = redis.call('INCRBY', KEYS[1], cost)
if current == cost then
  redis.call('EXPIRE', KEYS[1], ARGV[3])
end
if current > limit then
  redis.call('DECRBY', KEYS[1], cost)
  return {0, current - cost}
end
return {1, current}

The first value returned by the script is 1 for allowed and 0 for rejected. On rejection the script rolls back this increment, ensuring the counter is not inflated by failed requests.

4. Returning 429 and Retry-After on Over-Limit

When a quota is exceeded, the service should return HTTP status code 429 Too Many Requests. The Retry-After response header tells the client how many seconds to wait before retrying; its value can be an integer number of seconds or an HTTP date. After receiving a 429, clients should wait according to Retry-After and add random jitter, so that a large number of clients do not retry at the same instant.

function handle_request(request):
    decision = try_consume(request.user_id, request.dimension, request.cost)
    if decision.allowed == false:
        return response(
            status  = 429,
            headers = {
                "Retry-After": ceil(decision.retry_after),
                "X-RateLimit-Limit": decision.limit,
                "X-RateLimit-Remaining": 0
            },
            body = json({
                "error": "rate_limit_exceeded",
                "message": "Too many requests. Please try again later."
            })
        )
    return forward_to_model(request)
end function

The X-RateLimit family of headers is not a standard specification but is widely used. The IETF also has a draft for RateLimit headers, which teams can adopt according to their conventions.

5. Common Questions and Pitfalls

How do you guarantee quota atomicity across cluster nodes?
Put the read, check, and increment into a single Redis Lua script. If Redis sharding is used, make sure the counter key for the same user lands on the same shard; a hash tag can pin the shard location. With a relational database, you can use a conditional UPDATE or take a row lock inside a transaction.

What if Redis is unavailable?
Define a degradation strategy in advance. Allowing requests and logging them (fail-open) suits low-risk dimensions; rejecting outright (fail-closed) suits high-cost capabilities to prevent abuse.

Output tokens cannot be known in advance, so how do you deduct quota?
A common practice is to pre-deduct based on input and estimated output before the request, then settle against actual usage after the request completes, refunding or charging the difference. Settlement failures need a compensation job so that quotas are neither missed nor double-charged.

If a request fails after quota is deducted, should the quota be refunded?
Failures caused by non-user reasons, such as internal model service errors or timeouts, should generally be refunded. Whether to refund should be written into the rules at design time and the reason recorded in logs for easy reconciliation.