LLM API Cost Governance: A Guide to Retry, Rate Limiting, and Budget Alerts
This article provides a comprehensive guide to LLM API cost governance, covering business model selection, cost estimation models, and strategies for handling retries, rate limiting, and budget alerts. It includes Python code examples for cost calculation, token bucket rate limiting, usage metering, and a pre-launch checklist to ensure effective cost management and system stability.
1. Choose the Business Model: Subscription vs. Pay-as-you-go API
Large model capabilities are typically provided in three ways: free tier, subscription membership, and pay-as-you-go API. The billing units, permissions, and risks differ among them, and integrators need to account for them separately.
| Dimension | Free Tier | Subscription | Pay-as-you-go API |
|---|---|---|---|
| Billing Method | No charge | Fixed periodic charge | Settled by input and output tokens |
| Permissions & Quota | Quota may change | Usage limit per individual account | Controlled by API key and quota |
| Cost Predictability | Low | High | Depends on traffic |
| Suitable Scenarios | Personal trial, prototyping | Employee daily use | Business system integration, high traffic fluctuation |
| Integrate with Own System | Not recommended for production dependency | Check Terms of Service | Supported per API documentation |
| Main Risk | Quota exhaustion | Feature boundary changes | Cost overrun, request throttling |
Specific prices, quotas, and tiers are subject to the provider’s published rules. This article does not list specific numbers.
You can determine the selection based on the following rules:
- Personal use only, fixed features, cost calculated per capita: Choose subscription.
- Business system calls the model, traffic fluctuates significantly, feature-based metering required: Choose pay-as-you-go API.
- For hybrid usage, separate budgets for subscription and API. Subscription quotas cannot be used as API quotas, and API costs should not be counted towards subscriptions.
- Server-side calls should uniformly use the API and record usage at a unified gateway layer to avoid individual business lines making their own calls, which makes costs impossible to attribute.
2. Cost Model and Estimation
A token is the basic unit the model uses to process text. The same word count corresponds to different token numbers across different languages and splitting methods, so estimations should be based on the actual metering results returned by the provider, not directly converted by word count.
The cost of a single call consists of two parts: input tokens and output tokens. Some services also charge additional fees for caching, tool calls, etc., which need to be verified item by item in the billing description. The unit price for output is generally higher than for input, so allowing the model to generate verbose replies will significantly increase costs.
Daily cost can be estimated using the following formula:
Daily Cost = Daily Calls × (Average Input Tokens ÷ 1000 × Input Price + Average Output Tokens ÷ 1000 × Output Price)
| Variable | Meaning | Source |
|---|---|---|
| Daily Calls | Number of requests initiated per day | Gateway or application logs |
| Average Input Tokens | Average input amount per request | usage field returned by the provider |
| Average Output Tokens | Average output amount per request | usage field returned by the provider |
| Input Price, Output Price | Price per 1,000 tokens | Provider’s published pricing page, subject to billing |
When recalculating, follow these steps:
- Take the actual token count from the bill or
usagefield, do not estimate using word count. - Verify the currency and pricing caliber, confirm whether input and output are priced separately, and whether additional fees are included.
- Calculate daily and monthly costs based on average daily call volume.
- Reserve a buffer for price changes and traffic fluctuations; the buffer ratio should be set by the team based on historical fluctuations.
The following code implements the cost calculation for a single call. It requires Python 3.10 or higher and uses only the standard library.
from dataclasses import dataclass@dataclass(frozen=True)class Pricing:currency: str # Currency, must match provider billing, e.g., "CNY"input_per_1k: float # Price per 1000 input tokens, based on provider's published ratesoutput_per_1k: float # Price per 1000 output tokens, based on provider's published ratesdef estimate_cost(input_tokens: int, output_tokens: int, pricing: Pricing) -> float:"""Estimate single call cost based on per 1k token price, return value unit matches pricing.currency."""if input_tokens < 0 or output_tokens < 0:raise ValueError("Token count cannot be negative")if pricing.input_per_1k < 0 or pricing.output_per_1k < 0:raise ValueError("Unit price cannot be negative")return (input_tokens / 1000) * pricing.input_per_1k + (output_tokens / 1000) * pricing.output_per_1k
Parameter description:
input_tokensandoutput_tokensshould be taken from theusagefield returned by the provider, do not substitute with word count.- The unit prices in
Pricingmust match the billing caliber. If the unit price is incorrect, the estimation result will deviate entirely, so verify it against a real bill before going live. - The function raises an error directly for negative inputs to avoid misuse resulting in negative costs, which could mask issues.
3. Retry, Rate Limiting, Usage Metering, and Budget Alerts
3.1 Retry and Rate Limiting
Before retrying, first determine whether the error is worth retrying:
- 429 (Too Many Requests), 5xx (Server Error), and timeouts can be retried.
- 400, 401, 403, 404 indicate issues with the request itself or permissions; retrying is meaningless and should raise an error directly.
Callers can control behavior using the following parameters:
| Parameter | Meaning | Suggestion |
|---|---|---|
| max_retries | Maximum retry count | Usually 2 to 3 times; too many will amplify cost and latency |
| base_delay | Seconds to wait before the first retry | Start at 0.5 seconds, double each time |
| max_delay | Maximum seconds to wait per retry | e.g., 8 seconds, prevents infinite wait time |
| timeout | Timeout seconds for a single request | Set based on the longest acceptable response time; long text requests should be set separately |
| rate_per_sec | Number of tokens refilled per second | Set as a proportion of the provider’s quota, leave a buffer |
| capacity | Token bucket capacity, i.e., allowed burst requests | Slightly larger than a single batch request size |
There is a risk with retries: generation requests executed multiple times may be billed multiple times. The team should record a request ID for each request for reconciliation. If the business cannot accept duplicate results, do not auto-retry that request; instead, raise an error or queue it.
The following code implements token bucket rate limiting and exponential backoff retry.
import loggingimport randomimport threadingimport timelogger = logging.getLogger("llm_cost")class RetryableError(Exception):"""Retryable error. 429, 5xx responses should map to this exception; fill retry_after with Retry-After header for 429."""def __init__(self, status: int, retry_after: float | None = None):super().__init__(f"Retryable error status={status}")self.status = statusself.retry_after = retry_afterclass TokenBucket:"""Token bucket rate limiting. rate_per_sec is tokens refilled per second, capacity is bucket size."""def __init__(self, rate_per_sec: float, capacity: int):if rate_per_sec <= 0 or capacity <= 0:raise ValueError("rate_per_sec and capacity must be greater than 0")self.rate = rate_per_secself.capacity = capacityself.tokens = float(capacity)self.updated = time.monotonic()self.lock = threading.Lock()def acquire(self) -> None:"""Acquire a token, blocks and waits if no tokens are available."""while True:with self.lock:now = time.monotonic()self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)self.updated = nowif self.tokens >= 1:self.tokens -= 1returnwait = (1 - self.tokens) / self.ratetime.sleep(wait)def call_with_retry(send, payload, *, bucket: TokenBucket, max_retries: int = 3,base_delay: float = 0.5, max_delay: float = 8.0, timeout: float = 30.0):"""Exponential backoff retry. Returns result on success of send(payload, timeout=...);raises RetryableError or TimeoutError on failure. Other exceptions are raised directly without retry."""for attempt in range(max_retries + 1):bucket.acquire()try:return send(payload, timeout=timeout)except (RetryableError, TimeoutError) as exc:if attempt == max_retries:raiseif isinstance(exc, RetryableError) and exc.retry_after is not None:delay = exc.retry_afterelse:delay = min(max_delay, base_delay * (2 ** attempt))delay += random.uniform(0, delay * 0.1) # Add jitter to prevent multiple clients from retrying simultaneouslylogger.warning("Attempt %d failed, retrying in %.2f seconds: %s", attempt + 1, delay, exc)time.sleep(delay)def fake_send(payload, timeout: float):"""Fake request function for demo. In real environments, replace with HTTP client calls and map 429 and 5xx to RetryableError."""if random.random() < 0.3:raise RetryableError(429, retry_after=0.1)return {"text": "Example reply", "usage": {"input_tokens": 800, "output_tokens": 300}}
The behavior of this code is as follows:
- Before each send, it acquires a token from the token bucket, so the request rate will not exceed
rate_per_sec, and short bursts are capped atcapacity. - The wait time after a failure grows by
base_delay × 2^nand is limited bymax_delay. - If the provider provides a
Retry-Afterin the 429 response, the code prioritizes the wait time provided by the server. - Non-retryable errors are thrown directly outward, allowing callers to alert or correct the request accordingly.
3.2 Usage Metering and Budget Alerts
Usage metering should record input tokens, output tokens, and cost across three dimensions: feature, user, and model. This way, when billing anomalies occur, the team can quickly pinpoint which feature, type of user, or model caused it.
It is recommended to set two thresholds for budget alerts:
- Reaching 80% of the daily budget: Issue an alert to notify the person in charge to check traffic.
- Reaching 100% of the daily budget: Circuit break. After circuit breaking, the caller should handle degradation, such as switching to a smaller model, returning cached results, or queuing tasks for the next day.
The following code implements usage metering, threshold alerting, and circuit breaking.
from collections import defaultdictfrom datetime import dateclass UsageMeter:"""Track usage by feature, user, and model. Alerts when budget threshold is reached, circuit breaks at 100%."""ALERT_LEVELS = (0.8, 1.0)def __init__(self, daily_budget: float, pricing: Pricing):if daily_budget <= 0:raise ValueError("daily_budget must be greater than 0")self.daily_budget = daily_budgetself.pricing = pricingself._reset(date.today())def _reset(self, day: date) -> None:self.day = dayself.spent = 0.0self.totals = defaultdict(lambda: {"input": 0, "output": 0, "cost": 0.0})self.alerted = set()def _roll_day(self) -> None:"""Reset daily stats when the day changes."""today = date.today()if today != self.day:self._reset(today)def record(self, feature: str, user_id: str, model: str,input_tokens: int, output_tokens: int) -> float:"""Record usage for a single call, returns the cost of this call."""self._roll_day()cost = estimate_cost(input_tokens, output_tokens, self.pricing)item = self.totals[(feature, user_id, model)]item["input"] += input_tokensitem["output"] += output_tokensitem["cost"] += costself.spent += costself._check_alerts()return costdef _check_alerts(self) -> None:ratio = self.spent / self.daily_budgetfor level in self.ALERT_LEVELS:if ratio >= level and level not in self.alerted:self.alerted.add(level)logger.warning("Daily budget usage reached %.0f%% (%.2f / %.2f %s)",ratio * 100, self.spent, self.daily_budget, self.pricing.currency)def allow(self) -> bool:"""Circuit break check. Returns False when budget is exhausted, caller should execute degradation strategy."""self._roll_day()return self.spent < self.daily_budgetdef report(self):"""Returns usage details sorted by cost descending, can be used for cost dashboards."""return sorted(self.totals.items(), key=lambda kv: kv[1]["cost"], reverse=True)
Parameter description:
daily_budgetis the daily budget, with the same unit aspricing.currency.ALERT_LEVELSdefines alert thresholds. Teams can add thresholds as needed, such as 50%.allow()is called before each request. It only checks for circuit breaking and does not handle degradation; the degradation logic should be implemented by the caller.- The details returned by
report()can be integrated into a dashboard for daily review.
3.3 Integration Example and Key Management
The following example strings together the previous three parts. The runtime requires Python 3.10 or higher. Save all code as llm_cost.py and run python llm_cost.py. The unit prices used in the example are for demonstration and are unrelated to any provider’s actual pricing.
import osdef redact(text: str, secret: str) -> str:"""Replace secret key before logging to prevent it from being written to logs."""return text.replace(secret, "***") if secret else textif __name__ == "__main__":logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")# Keys are read only from environment variables or secret management services, never hardcoded in code or repos.api_key = os.environ.get("LLM_API_KEY", "")if not api_key:raise SystemExit("LLM_API_KEY not configured")pricing = Pricing(currency="CNY", input_per_1k=1.0, output_per_1k=1.0) # Demo values, unrelated to real pricesmeter = UsageMeter(daily_budget=0.5, pricing=pricing)bucket = TokenBucket(rate_per_sec=5, capacity=5)for _ in range(3):if not meter.allow():logger.error("Budget exhausted, this request is degraded")breaktry:resp = call_with_retry(fake_send, {"prompt": "Please summarize this text"}, bucket=bucket)except RetryableError as exc:logger.error("Retries exhausted, request failed: %s", redact(str(exc), api_key))continueusage = resp["usage"]meter.record("summary", "user_001", "demo-model",usage["input_tokens"], usage["output_tokens"])for key, item in meter.report():print(key, item)
Expected results: The first request usually succeeds, the cost exceeds the demo budget, and the logs output two alerts for 80% and 100%. When the second loop starts, allow() returns False, and the program outputs a degradation prompt and exits the loop.
Key management rules:
- Read keys only from environment variables or secret management services.
- Never write keys into code, config files, logs, or tickets.
- Redact keys and user privacy information before outputting logs and exceptions, as shown in the
redactexample. - If a key is leaked, immediately revoke and reissue it in the provider’s console, then check usage during the leak period.
4. Pre-launch Checklist and Post-mortem
Confirm the following items before going live:
Quota & Rate Limiting
- Confirmed the provider’s QPS, concurrency, and daily quotas, and set
rate_per_secandcapacityaccordingly. - Long text requests and standard requests use different timeout settings.
Retry & Timeout
- 429, 5xx, and timeouts are retried; 400, 401, 403, 404 are not.
- A request ID is recorded for each request to reconcile duplicate billing.
- Auto-retry is disabled for requests that are not suitable for duplicate execution.
Keys & Logs
- Keys come from environment variables or secret management services, and no plaintext keys are found in the code repository.
- Keys and user privacy information in logs and alerts have been redacted.
Budget & Alerts
