LLM API Cost Governance: A Guide to Retry, Rate Limiting, and Budget Alerts

    This article provides a comprehensive guide to LLM API cost governance, covering business model selection, cost estimation models, and strategies for handling retries, rate limiting, and budget alerts. It includes Python code examples for cost calculation, token bucket rate limiting, usage metering, and a pre-launch checklist to ensure effective cost management and system stability.

    1. Choose the Business Model: Subscription vs. Pay-as-you-go API

    Large model capabilities are typically provided in three ways: free tier, subscription membership, and pay-as-you-go API. The billing units, permissions, and risks differ among them, and integrators need to account for them separately.

    Dimension Free Tier Subscription Pay-as-you-go API
    Billing Method No charge Fixed periodic charge Settled by input and output tokens
    Permissions & Quota Quota may change Usage limit per individual account Controlled by API key and quota
    Cost Predictability Low High Depends on traffic
    Suitable Scenarios Personal trial, prototyping Employee daily use Business system integration, high traffic fluctuation
    Integrate with Own System Not recommended for production dependency Check Terms of Service Supported per API documentation
    Main Risk Quota exhaustion Feature boundary changes Cost overrun, request throttling

    Specific prices, quotas, and tiers are subject to the provider’s published rules. This article does not list specific numbers.

    You can determine the selection based on the following rules:

    • Personal use only, fixed features, cost calculated per capita: Choose subscription.
    • Business system calls the model, traffic fluctuates significantly, feature-based metering required: Choose pay-as-you-go API.
    • For hybrid usage, separate budgets for subscription and API. Subscription quotas cannot be used as API quotas, and API costs should not be counted towards subscriptions.
    • Server-side calls should uniformly use the API and record usage at a unified gateway layer to avoid individual business lines making their own calls, which makes costs impossible to attribute.

    2. Cost Model and Estimation

    A token is the basic unit the model uses to process text. The same word count corresponds to different token numbers across different languages and splitting methods, so estimations should be based on the actual metering results returned by the provider, not directly converted by word count.

    The cost of a single call consists of two parts: input tokens and output tokens. Some services also charge additional fees for caching, tool calls, etc., which need to be verified item by item in the billing description. The unit price for output is generally higher than for input, so allowing the model to generate verbose replies will significantly increase costs.

    Daily cost can be estimated using the following formula:

    Daily Cost = Daily Calls × (Average Input Tokens ÷ 1000 × Input Price + Average Output Tokens ÷ 1000 × Output Price)

    Variable Meaning Source
    Daily Calls Number of requests initiated per day Gateway or application logs
    Average Input Tokens Average input amount per request usage field returned by the provider
    Average Output Tokens Average output amount per request usage field returned by the provider
    Input Price, Output Price Price per 1,000 tokens Provider’s published pricing page, subject to billing

    When recalculating, follow these steps:

    1. Take the actual token count from the bill or usage field, do not estimate using word count.
    2. Verify the currency and pricing caliber, confirm whether input and output are priced separately, and whether additional fees are included.
    3. Calculate daily and monthly costs based on average daily call volume.
    4. Reserve a buffer for price changes and traffic fluctuations; the buffer ratio should be set by the team based on historical fluctuations.

    The following code implements the cost calculation for a single call. It requires Python 3.10 or higher and uses only the standard library.

    1. from dataclasses import dataclass
    2. @dataclass(frozen=True)
    3. class Pricing:
    4. currency: str # Currency, must match provider billing, e.g., "CNY"
    5. input_per_1k: float # Price per 1000 input tokens, based on provider's published rates
    6. output_per_1k: float # Price per 1000 output tokens, based on provider's published rates
    7. def estimate_cost(input_tokens: int, output_tokens: int, pricing: Pricing) -> float:
    8. """Estimate single call cost based on per 1k token price, return value unit matches pricing.currency."""
    9. if input_tokens < 0 or output_tokens < 0:
    10. raise ValueError("Token count cannot be negative")
    11. if pricing.input_per_1k < 0 or pricing.output_per_1k < 0:
    12. raise ValueError("Unit price cannot be negative")
    13. return (input_tokens / 1000) * pricing.input_per_1k + (output_tokens / 1000) * pricing.output_per_1k

    Parameter description:

    • input_tokens and output_tokens should be taken from the usage field returned by the provider, do not substitute with word count.
    • The unit prices in Pricing must match the billing caliber. If the unit price is incorrect, the estimation result will deviate entirely, so verify it against a real bill before going live.
    • The function raises an error directly for negative inputs to avoid misuse resulting in negative costs, which could mask issues.

    3. Retry, Rate Limiting, Usage Metering, and Budget Alerts

    3.1 Retry and Rate Limiting

    Before retrying, first determine whether the error is worth retrying:

    • 429 (Too Many Requests), 5xx (Server Error), and timeouts can be retried.
    • 400, 401, 403, 404 indicate issues with the request itself or permissions; retrying is meaningless and should raise an error directly.

    Callers can control behavior using the following parameters:

    Parameter Meaning Suggestion
    max_retries Maximum retry count Usually 2 to 3 times; too many will amplify cost and latency
    base_delay Seconds to wait before the first retry Start at 0.5 seconds, double each time
    max_delay Maximum seconds to wait per retry e.g., 8 seconds, prevents infinite wait time
    timeout Timeout seconds for a single request Set based on the longest acceptable response time; long text requests should be set separately
    rate_per_sec Number of tokens refilled per second Set as a proportion of the provider’s quota, leave a buffer
    capacity Token bucket capacity, i.e., allowed burst requests Slightly larger than a single batch request size

    There is a risk with retries: generation requests executed multiple times may be billed multiple times. The team should record a request ID for each request for reconciliation. If the business cannot accept duplicate results, do not auto-retry that request; instead, raise an error or queue it.

    The following code implements token bucket rate limiting and exponential backoff retry.

    1. import logging
    2. import random
    3. import threading
    4. import time
    5. logger = logging.getLogger("llm_cost")
    6. class RetryableError(Exception):
    7. """Retryable error. 429, 5xx responses should map to this exception; fill retry_after with Retry-After header for 429."""
    8. def __init__(self, status: int, retry_after: float | None = None):
    9. super().__init__(f"Retryable error status={status}")
    10. self.status = status
    11. self.retry_after = retry_after
    12. class TokenBucket:
    13. """Token bucket rate limiting. rate_per_sec is tokens refilled per second, capacity is bucket size."""
    14. def __init__(self, rate_per_sec: float, capacity: int):
    15. if rate_per_sec <= 0 or capacity <= 0:
    16. raise ValueError("rate_per_sec and capacity must be greater than 0")
    17. self.rate = rate_per_sec
    18. self.capacity = capacity
    19. self.tokens = float(capacity)
    20. self.updated = time.monotonic()
    21. self.lock = threading.Lock()
    22. def acquire(self) -> None:
    23. """Acquire a token, blocks and waits if no tokens are available."""
    24. while True:
    25. with self.lock:
    26. now = time.monotonic()
    27. self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
    28. self.updated = now
    29. if self.tokens >= 1:
    30. self.tokens -= 1
    31. return
    32. wait = (1 - self.tokens) / self.rate
    33. time.sleep(wait)
    34. def call_with_retry(send, payload, *, bucket: TokenBucket, max_retries: int = 3,
    35. base_delay: float = 0.5, max_delay: float = 8.0, timeout: float = 30.0):
    36. """Exponential backoff retry. Returns result on success of send(payload, timeout=...);
    37. raises RetryableError or TimeoutError on failure. Other exceptions are raised directly without retry."""
    38. for attempt in range(max_retries + 1):
    39. bucket.acquire()
    40. try:
    41. return send(payload, timeout=timeout)
    42. except (RetryableError, TimeoutError) as exc:
    43. if attempt == max_retries:
    44. raise
    45. if isinstance(exc, RetryableError) and exc.retry_after is not None:
    46. delay = exc.retry_after
    47. else:
    48. delay = min(max_delay, base_delay * (2 ** attempt))
    49. delay += random.uniform(0, delay * 0.1) # Add jitter to prevent multiple clients from retrying simultaneously
    50. logger.warning("Attempt %d failed, retrying in %.2f seconds: %s", attempt + 1, delay, exc)
    51. time.sleep(delay)
    52. def fake_send(payload, timeout: float):
    53. """Fake request function for demo. In real environments, replace with HTTP client calls and map 429 and 5xx to RetryableError."""
    54. if random.random() < 0.3:
    55. raise RetryableError(429, retry_after=0.1)
    56. return {"text": "Example reply", "usage": {"input_tokens": 800, "output_tokens": 300}}

    The behavior of this code is as follows:

    • Before each send, it acquires a token from the token bucket, so the request rate will not exceed rate_per_sec, and short bursts are capped at capacity.
    • The wait time after a failure grows by base_delay × 2^n and is limited by max_delay.
    • If the provider provides a Retry-After in the 429 response, the code prioritizes the wait time provided by the server.
    • Non-retryable errors are thrown directly outward, allowing callers to alert or correct the request accordingly.

    3.2 Usage Metering and Budget Alerts

    Usage metering should record input tokens, output tokens, and cost across three dimensions: feature, user, and model. This way, when billing anomalies occur, the team can quickly pinpoint which feature, type of user, or model caused it.

    It is recommended to set two thresholds for budget alerts:

    • Reaching 80% of the daily budget: Issue an alert to notify the person in charge to check traffic.
    • Reaching 100% of the daily budget: Circuit break. After circuit breaking, the caller should handle degradation, such as switching to a smaller model, returning cached results, or queuing tasks for the next day.

    The following code implements usage metering, threshold alerting, and circuit breaking.

    1. from collections import defaultdict
    2. from datetime import date
    3. class UsageMeter:
    4. """Track usage by feature, user, and model. Alerts when budget threshold is reached, circuit breaks at 100%."""
    5. ALERT_LEVELS = (0.8, 1.0)
    6. def __init__(self, daily_budget: float, pricing: Pricing):
    7. if daily_budget <= 0:
    8. raise ValueError("daily_budget must be greater than 0")
    9. self.daily_budget = daily_budget
    10. self.pricing = pricing
    11. self._reset(date.today())
    12. def _reset(self, day: date) -> None:
    13. self.day = day
    14. self.spent = 0.0
    15. self.totals = defaultdict(lambda: {"input": 0, "output": 0, "cost": 0.0})
    16. self.alerted = set()
    17. def _roll_day(self) -> None:
    18. """Reset daily stats when the day changes."""
    19. today = date.today()
    20. if today != self.day:
    21. self._reset(today)
    22. def record(self, feature: str, user_id: str, model: str,
    23. input_tokens: int, output_tokens: int) -> float:
    24. """Record usage for a single call, returns the cost of this call."""
    25. self._roll_day()
    26. cost = estimate_cost(input_tokens, output_tokens, self.pricing)
    27. item = self.totals[(feature, user_id, model)]
    28. item["input"] += input_tokens
    29. item["output"] += output_tokens
    30. item["cost"] += cost
    31. self.spent += cost
    32. self._check_alerts()
    33. return cost
    34. def _check_alerts(self) -> None:
    35. ratio = self.spent / self.daily_budget
    36. for level in self.ALERT_LEVELS:
    37. if ratio >= level and level not in self.alerted:
    38. self.alerted.add(level)
    39. logger.warning("Daily budget usage reached %.0f%% (%.2f / %.2f %s)",
    40. ratio * 100, self.spent, self.daily_budget, self.pricing.currency)
    41. def allow(self) -> bool:
    42. """Circuit break check. Returns False when budget is exhausted, caller should execute degradation strategy."""
    43. self._roll_day()
    44. return self.spent < self.daily_budget
    45. def report(self):
    46. """Returns usage details sorted by cost descending, can be used for cost dashboards."""
    47. return sorted(self.totals.items(), key=lambda kv: kv[1]["cost"], reverse=True)

    Parameter description:

    • daily_budget is the daily budget, with the same unit as pricing.currency.
    • ALERT_LEVELS defines alert thresholds. Teams can add thresholds as needed, such as 50%.
    • allow() is called before each request. It only checks for circuit breaking and does not handle degradation; the degradation logic should be implemented by the caller.
    • The details returned by report() can be integrated into a dashboard for daily review.

    3.3 Integration Example and Key Management

    The following example strings together the previous three parts. The runtime requires Python 3.10 or higher. Save all code as llm_cost.py and run python llm_cost.py. The unit prices used in the example are for demonstration and are unrelated to any provider’s actual pricing.

    1. import os
    2. def redact(text: str, secret: str) -> str:
    3. """Replace secret key before logging to prevent it from being written to logs."""
    4. return text.replace(secret, "***") if secret else text
    5. if __name__ == "__main__":
    6. logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
    7. # Keys are read only from environment variables or secret management services, never hardcoded in code or repos.
    8. api_key = os.environ.get("LLM_API_KEY", "")
    9. if not api_key:
    10. raise SystemExit("LLM_API_KEY not configured")
    11. pricing = Pricing(currency="CNY", input_per_1k=1.0, output_per_1k=1.0) # Demo values, unrelated to real prices
    12. meter = UsageMeter(daily_budget=0.5, pricing=pricing)
    13. bucket = TokenBucket(rate_per_sec=5, capacity=5)
    14. for _ in range(3):
    15. if not meter.allow():
    16. logger.error("Budget exhausted, this request is degraded")
    17. break
    18. try:
    19. resp = call_with_retry(fake_send, {"prompt": "Please summarize this text"}, bucket=bucket)
    20. except RetryableError as exc:
    21. logger.error("Retries exhausted, request failed: %s", redact(str(exc), api_key))
    22. continue
    23. usage = resp["usage"]
    24. meter.record("summary", "user_001", "demo-model",
    25. usage["input_tokens"], usage["output_tokens"])
    26. for key, item in meter.report():
    27. print(key, item)

    Expected results: The first request usually succeeds, the cost exceeds the demo budget, and the logs output two alerts for 80% and 100%. When the second loop starts, allow() returns False, and the program outputs a degradation prompt and exits the loop.

    Key management rules:

    • Read keys only from environment variables or secret management services.
    • Never write keys into code, config files, logs, or tickets.
    • Redact keys and user privacy information before outputting logs and exceptions, as shown in the redact example.
    • If a key is leaked, immediately revoke and reissue it in the provider’s console, then check usage during the leak period.

    4. Pre-launch Checklist and Post-mortem

    Confirm the following items before going live:

    Quota & Rate Limiting

    • Confirmed the provider’s QPS, concurrency, and daily quotas, and set rate_per_sec and capacity accordingly.
    • Long text requests and standard requests use different timeout settings.

    Retry & Timeout

    • 429, 5xx, and timeouts are retried; 400, 401, 403, 404 are not.
    • A request ID is recorded for each request to reconcile duplicate billing.
    • Auto-retry is disabled for requests that are not suitable for duplicate execution.

    Keys & Logs

    • Keys come from environment variables or secret management services, and no plaintext keys are found in the code repository.
    • Keys and user privacy information in logs and alerts have been redacted.

    Budget & Alerts