How Token Quotas Work: Quota Metrics, Usage Control, and Rate Limit Tuning for LLM APIs
Understanding how LLM free-tier quotas are calculated involves more than just counting tokens. This guide breaks down token estimation methods, clarifies four common quota models (per-model, per-project, RPM, and TPM), and explains why long-text requests often hit rate limits unexpectedly. It also covers practical migration steps for switching between API endpoints, failure handling and fallback logic, and a pre-launch checklist for usage monitoring, privacy protection, and key management.
1. What Are Tokens and How to Estimate Them
In large language models, tokens are the unit used to meter text input and output. A tokenizer splits input text into fragments, each counted as one token. Both input and output count toward usage. In multi-turn conversations, repeated historical messages are counted again with every request.
For Chinese text, one character usually approximates one token. For English, a word averages one to two tokens. Code and symbols can vary more significantly. The rough estimation function below is useful for pre-calculation:
def estimate_tokens_rough(text: str) -> int:"""Rough estimation: Chinese characters count as ~1 token each; other characters count as ~1 token per 4."""cjk = sum(1 for ch in text if "\u4e00" <= ch <= "\u9fff")other = len(text) - cjkreturn cjk + (other + 3) // 4
Using this function on three sample texts yields the following (approximate, not measured):
| Text | Estimated Tokens | Notes |
|---|---|---|
| 大模型免费额度怎么算 (10 Chinese characters) | 10 | Each Chinese character counts as 1 |
| Token quotas are counted per project. (37 characters) | 10 | Roughly 4 characters per token |
def add(a, b): plus return a + b with newline and indent (31 characters) |
8 | Higher variance with symbols |
Rough estimation is only useful for understanding order of magnitude. For capacity planning, calibrate against the usage field returned by the API. Most OpenAI-compatible endpoints return prompt_tokens (input usage) and completion_tokens (output usage), though field names and locations vary by platform — check the relevant provider documentation.
import statisticsratios: list[float] = []def record_ratio(prompt: str, usage: dict) -> None:"""Record the ratio of actual input usage to the rough estimate for one request."""est = estimate_tokens_rough(prompt)if est > 0 and usage.get("prompt_tokens"):ratios.append(usage["prompt_tokens"] / est)def calibrated_estimate(text: str) -> int:"""Adjust the rough estimate using the median of historical ratios; fall back to the rough estimate if sample size is small."""base = estimate_tokens_rough(text)if len(ratios) < 20:return basereturn round(base * statistics.median(ratios))
Note that prompt_tokens includes role markers and system prompts, so the ratio tends to be higher than 1. If your system prompt is long, include it in the text being estimated; otherwise the calibrated result will overshoot.
2. Understanding Four Quota Models
Services labeled “free” can be metered in completely different ways. The table below lists four common quota types. Since quota policies may change, always verify against the provider’s console and billing page.
| Quota Type | Meaning | Where to Check | Common Misconception |
|---|---|---|---|
| Per-model | Each model has its own quota | Model list or usage page in the console | Assuming all models share one quota, missing per-model caps |
| Per-project or per-account | All API keys under the same project share one quota | Project settings or quota page | Creating more keys to increase the total — keys only provide permission isolation |
| RPM limit | Max requests per minute | Rate limit docs or response headers | Focusing only on token usage, missing request count limits |
| TPM limit | Max total tokens per minute | Rate limit docs or 429 error messages | Testing only with short Q&A, then hitting limits on long documents |
The quota model determines how you can scale. If the quota is shared per project and you route summarization, classification, and long-document Q&A through the same model, the quota will be consumed quickly. Creating three more API keys will not help. To scale, you would need to switch to a different project or upgrade to a paid plan. If the platform meters per model, moving classification tasks to a lightweight model gives you access to its separate quota, increasing total available capacity.
Registering multiple accounts to stack free quotas typically violates the platform’s terms of service. This can lead to account suspension and potential data loss. The correct approach is to read the quota documentation, plan usage according to the rules, and upgrade to a paid tier when needed.
3. Migration Setup: Endpoints in Config, Not in Business Logic
Most chat API endpoints share a similar structure: you send a list of messages and receive an array containing the reply. When switching platforms, you typically only need to change three things: the endpoint URL, the API key, and the model name. Put these in environment variables and have the business layer call a single unified function — this keeps the change footprint minimal.
import osimport randomimport timeimport requests# Store only environment variable names, not the keys themselvesENDPOINTS = [{"name": "primary", "base_url_env": "PRIMARY_BASE_URL", "api_key_env": "PRIMARY_API_KEY", "model": "model-large"},{"name": "backup", "base_url_env": "BACKUP_BASE_URL", "api_key_env": "BACKUP_API_KEY", "model": "model-small"},]def resolve_endpoints() -> list[dict]:"""Read URLs and keys from environment variables; skip endpoints with missing config."""found = []for ep in ENDPOINTS:base_url = os.getenv(ep["base_url_env"])api_key = os.getenv(ep["api_key_env"])if base_url and api_key:found.append({**ep, "base_url": base_url.rstrip("/"), "api_key": api_key})return founddef chat(messages: list[dict], timeout: int = 30) -> tuple[str, dict]:"""Try each endpoint in order; return the reply text and usage field."""last_error = Nonefor ep in resolve_endpoints():for attempt in range(3):try:resp = requests.post(f"{ep['base_url']}/chat/completions",headers={"Authorization": f"Bearer {ep['api_key']}"},json={"model": ep["model"], "messages": messages},timeout=timeout,)except requests.Timeout as e:last_error = ebreak # Timeout: switch to next endpointif resp.status_code == 429:last_error = RuntimeError(f"{ep['name']} endpoint rate limited")time.sleep(2 ** attempt + random.random()) # Backoff: 1s, 2s, 4s, plus jittercontinueif resp.status_code in (500, 502, 503, 504):last_error = RuntimeError(f"{ep['name']} endpoint server error")breakresp.raise_for_status() # 400, 401, 403 etc.: raise for the caller to handledata = resp.json()return data["choices"][0]["message"]["content"], data.get("usage", {})raise RuntimeError("All endpoints failed") from last_error
Degradation rules:
| Situation | Detection | Handling |
|---|---|---|
| Timeout | No response after 30 seconds | Switch to next endpoint |
| Rate limit | HTTP 429 | Backoff and retry, up to 3 attempts; switch endpoint if still failing |
| Server error | HTTP 500, 502, 503, 504 | Switch endpoint |
| Input too long | HTTP 400 with length-related error message | Chunk the input; do not switch endpoints |
| Auth failure | HTTP 401 or 403 | Stop immediately, check the key; do not retry |
After failing over to a backup endpoint, note that the backup may have a different quota and model capability. If output quality drops, log which endpoint was used to aid troubleshooting.
4. RPM vs. TPM: Why Long Texts Get Stuck
RPM relates to request count; TPM relates to total token volume. Suppose a platform allows 60 requests per minute and 200,000 tokens per minute. Sending 100 short queries at 200 tokens each totals only 20,000 tokens but exceeds 60 requests — triggering the RPM limit. Sending 12 long documents at 20,000 tokens each is only 12 requests but totals 240,000 tokens — triggering the TPM limit. Long-text processing tends to stall because it exhausts the token quota first.
Troubleshooting steps:
- Check the status code and error message. Mentions of request count usually point to RPM; mentions of tokens point to TPM. Exact wording varies by platform.
- Track usage. Accumulate the
usage.total_tokensfrom each response into a rolling 60-second window to identify which limit is hit first. - If TPM is exceeded, reduce concurrency, chunk long documents, and trim conversation history to only recent turns.
- If RPM is exceeded, merge small requests and control send rate with a queue.
- If still insufficient, move tasks to a separate per-model quota or upgrade to a paid plan.
The rate limiter below checks request count and token count within a sliding window before sending. Token counts use the calibrated estimate.
import timefrom collections import dequeclass MinuteLimiter:"""Sliding-window rate limiter: tracks request count and token count in the last 60 seconds."""def __init__(self, max_requests: int, max_tokens: int):self.max_requests = max_requestsself.max_tokens = max_tokensself.events: deque[tuple[float, int]] = deque()def _prune(self, now: float) -> None:while self.events and now - self.events[0][0] > 60:self.events.popleft()def wait_for(self, tokens: int) -> None:"""Block until there is capacity, then register the request."""while True:now = time.time()self._prune(now)used = sum(t for _, t in self.events)if len(self.events) < self.max_requests and used + tokens <= self.max_tokens:self.events.append((now, tokens))returntime.sleep(1)
Long documents require chunking. The function below splits by paragraph, keeping each chunk under the given token cap:
def split_by_paragraph(text: str, max_tokens: int = 1500) -> list[str]:chunks, buf = [], ""for para in text.split("\n\n"):candidate = f"{buf}\n\n{para}" if buf else paraif buf and estimate_tokens_rough(candidate) > max_tokens:chunks.append(buf)buf = paraelse:buf = candidateif buf:chunks.append(buf)return chunks
If a single paragraph itself exceeds the limit, this function cannot split it further — you would need to split by character. After chunking, keep the response from each chunk and concatenate or summarize at the end to avoid losing content.
5. Privacy and Key Management Checklist
- Store API keys only in environment variables or a key management service. Never hardcode keys in code or commit them to Git repositories.
- Log only timestamps, model names, endpoint names, status codes, latency, and token counts. Do not log full prompts or responses.
- Before sending, remove sensitive fields such as phone numbers, national ID numbers, and email addresses, or replace them with placeholders.
- Create a separate API key for each small tool, with a usage cap, so that a leaked key has limited blast radius.
- Set expiration times for local caches and clean them up periodically. Never store raw keys in caches.
- Review the platform’s data usage policy before use. Confirm whether your input may be used for training, and decide what data you can send accordingly.
6. Pre-Launch Checklist
- Quota model confirmed: check the console to determine whether the quota is per-model, per-project, per-request, or per-token.
- Usage monitoring in place: record the
usagefield for every request, use it to calibrate estimates, and set up alerts when thresholds are exceeded. - Degradation and privacy reviewed: backup endpoints switch correctly, errors do not cause infinite loops, and logs and caches contain no keys or sensitive data.
