A Practical Guide to LLM API Cost Estimation and Optimization

    This article explores how to estimate, monitor, and optimize the costs of large language model (LLM) API calls. It covers the fundamentals of token-based pricing, building a cost model, scaling to monthly budgets, implementing practical cost-reduction strategies, and setting up effective monitoring and alerting systems for long-term budget control.

    1. Why Token Cost Should Be Part of Engineering Design

    Most LLM APIs are metered by tokens. A token is the basic unit a model uses to split text. The same paragraph will yield a different number of tokens depending on the tokenizer used. A single Chinese character usually corresponds to about one to two tokens, while a single English word corresponds to roughly one to two tokens. Therefore, the cost of the same feature can vary several times depending on the language and text format.

    Billing is generally divided into two parts: input tokens and output tokens. The unit prices for these are usually different, with output typically priced higher. Some services also offer cached pricing for repeated input prefixes, where the unit price for the cache-hit portion is lower than the cache-miss portion. These rules are common in the industry, but specific prices and rules are subject to each service’s official pricing page. In engineering implementation, these values should not be hardcoded.

    Cost issues may not be obvious during the prototyping phase, but they amplify rapidly at scale. A prompt that consumes a few hundred extra tokens per call can result in significant overhead with tens of thousands of daily calls. Therefore, cost should be treated as an engineering metric, included in design reviews alongside latency and accuracy.

    2. Breaking Down the Cost Model

    The basic cost of a single request can be expressed as:

    Single Cost = Input Token Count × Input Unit Price + Output Token Count × Output Unit Price

    If the service distinguishes between cache hits and misses, the input portion must be calculated separately:

    Input Cost = Missed Token Count × Miss Unit Price + Cached Token Count × Cache Unit Price

    The code below encapsulates this model into a function. Prices are passed in via configuration, making it easy to update only the configuration when official prices change without modifying the logic.

    1. from dataclasses import dataclass
    2. @dataclass(frozen=True)
    3. class PriceConfig:
    4. input_per_million: float # Price per million input tokens (cache miss)
    5. cached_input_per_million: float # Price per million input tokens (cache hit)
    6. output_per_million: float # Price per million output tokens
    7. currency: str = "USD"
    8. def estimate_request_cost(
    9. prompt_tokens: int,
    10. cached_tokens: int,
    11. completion_tokens: int,
    12. price: PriceConfig,
    13. ) -> float:
    14. if cached_tokens > prompt_tokens:
    15. raise ValueError("Cached token count cannot exceed total input token count")
    16. miss_tokens = prompt_tokens - cached_tokens
    17. cost = (
    18. miss_tokens * price.input_per_million
    19. + cached_tokens * price.cached_input_per_million
    20. + completion_tokens * price.output_per_million
    21. ) / 1_000_000
    22. return round(cost, 6)

    Most compatible interfaces return input and output token statistics in the response, while the cache hit count may be provided in a separate field. The exact field names should be referenced from the specific API documentation and should not be assumed based on experience.

    If your team accounts in a local currency (e.g., CNY), you also need to handle exchange rates. It is recommended to keep the original currency in the configuration and convert it using a unified standard during financial settlement or reporting, recording the exchange rate date used to avoid inconsistencies in historical data caused by exchange rate fluctuations.

    3. From Single Requests to Monthly Budgets

    Single-request cost is only the starting point; budget management requires placing it within the context of business scale. The following four steps are recommended:

    First, sample real traffic. Do not estimate using ideal values from design documents. Instead, extract a period of requests from production logs and calculate the mean, median, and 95th percentile of input and output tokens. Long-tail requests often account for the majority of costs, so looking only at the mean will underestimate risks.

    Second, estimate request volume. Break down average daily calls by business scenario, distinguish between peak and off-peak hours, and leave a margin for retries and failed requests. Retries are a commonly overlooked cost source; if a failed request automatically retries three times, actual consumption could be several times the original.

    Third, aggregate daily and monthly costs. Daily cost equals the sum of daily request volumes for each scenario multiplied by the corresponding average single-request cost. Monthly cost can be calculated based on actual operating days.

    Fourth, set budget thresholds. Set daily and monthly budgets for each business line or feature, and trigger alerts when a certain percentage is reached.

    Example: A feature has about 10,000 daily calls, with an average of 2,000 input tokens and 500 output tokens, and no cache hits. Plugging these values into the function from the previous section yields the single-day cost, which can then be multiplied by the number of days to get the monthly cost. When using different parameters, simply replace the input values; the model itself remains unchanged.

    1. def monthly_budget(daily_requests: int, avg_prompt: int, avg_cached: int,
    2. avg_completion: int, price: PriceConfig, days: int = 30) -> float:
    3. per_request = estimate_request_cost(avg_prompt, avg_cached, avg_completion, price)
    4. return round(per_request * daily_requests * days, 2)

    4. Cost Reduction Strategies

    Cost reduction should start with engineering structure rather than simply cutting features. The following categories of strategies are common in practice.

    1. Streamline Prompts

    Redundant instructions, repetitive descriptions, and overly long examples in prompts directly increase input tokens. When streamlining, review section by section: which rules can be merged, which examples can be replaced by a single more typical one, and whether the entire conversation history needs to be retained. The general practice is to keep the task definition and output constraints, and summarize or truncate historical content using a sliding window. After streamlining, the effect must be validated using the same batch of test samples to ensure accuracy hasn’t dropped significantly.

    2. Fix Prefixes to Improve Cache Hit Rates

    Caching usually takes effect based on identical prefixes. Therefore, immutable content—such as system instructions, role settings, tool definitions, and fixed reference materials—should be placed at the very beginning of the prompt, while variable content—like user questions and real-time data—should be placed at the end. This keeps the prefix consistent across multiple requests, increasing the probability of a cache hit. Conversely, inserting a timestamp or random ID at the beginning makes caching almost impossible.

    3. Control Output Length

    Output tokens are typically priced higher than input tokens, so controlling output often yields greater savings. You can limit the maximum output length via API parameters while explicitly requiring brevity in the instructions, such as “Return JSON only, no explanations.” For classification and extraction tasks, structured output reduces token consumption, facilitates programmatic parsing, and minimizes post-processing costs.

    4. Tiered Routing Based on Task Difficulty

    Not all tasks require the most powerful model. Business requests can be categorized into simple, medium, and complex: simple tasks like keyword classification and format conversion can be handled by smaller models, while complex tasks like multi-step reasoning and long document analysis can be assigned to more capable models. The prerequisite for tiered routing is establishing an evaluation set to compare the quality and cost of different models across various tasks. Downgrading should only occur when the quality gap is acceptable.

    5. Context Control in Retrieval-Augmented Generation (RAG)

    In RAG scenarios, the number of retrieved documents and the chunk size directly determine the input length. You should determine the appropriate number of retrieved items based on evaluation results to avoid stuffing the context with a large amount of weakly related content. Removing duplicate paragraphs and truncating after sorting by relevance can also reduce tokens without compromising effectiveness.

    6. Application-Layer Result Caching

    For identical or highly similar requests, results can be cached at the application layer with a reasonable expiration time. This type of caching bypasses model calls entirely, offering the most direct savings, but attention must be paid to data timeliness to avoid returning outdated information to users.

    7. Batch and Asynchronous Processing

    For non-real-time tasks like report generation and data labeling, requests can be merged or placed in queues to be processed during off-peak hours. This not only facilitates concurrency control but also avoids cost amplification from retries caused by instantaneous traffic spikes.

    5. Monitoring and Alerting

    Cost control without monitoring relies entirely on post-hoc reconciliation. It is recommended to log the following fields for each request: model name, input token count, output token count, cache hit token count, latency, business tag, and request status. This data can be written to a logging service and then used to generate dashboards via a monitoring and alerting system.

    Dashboards should include at least three perspectives: cost trends by business line, unit cost comparisons by model, and abnormal fluctuations by feature. Alert rules can be configured as follows: send a reminder when daily cost exceeds 80% of the daily budget, log anomalous samples when a single request cost exceeds a multiple of the historical 95th percentile, and notify the owner when the retry rate remains above the baseline for several consecutive hours.

    The purpose of alerting is to allow the team to intervene early, not to assign blame after the fact, so thresholds should be repeatedly adjusted based on actual business conditions.

    6. Common Pitfalls and Implementation Recommendations

    The first pitfall is comparing only unit prices. Different services have varying unit prices, output lengths, caching strategies, and response qualities. Comparing unit prices directly may lead to incorrect conclusions. A more reliable approach is to run multiple solutions on the same test set, calculate the “average cost per completed task” and “task success rate,” and make a comprehensive judgment.

    The second pitfall is ignoring retries and failed requests. Timeouts, rate limits, and format parsing failures all generate extra calls. You should set retry limits, distinguish between retryable and non-retryable errors, and ensure idempotency of retries to prevent the same task from being executed repeatedly.

    The third pitfall is using the most powerful model for all requests. This causes costs to accumulate continuously in areas with no quality benefit. Tiered routing and small-model preprocessing can usually significantly reduce expenditures while keeping quality virtually unchanged.

    When implementing, it is recommended to proceed in three steps. First, establish logging and cost calculation to make current spending visible. Second, optimize the one or two features with the highest cost proportions and validate the results using a test set. Third, incorporate cost metrics into launch reviews. New features must provide an estimated single-request cost and monthly budget during the canary phase.

    Cost optimization is not a one-time job. As the business grows, prompts iterate, and models update, the cost structure will continue to change. Only by forming a closed loop of estimation, monitoring, and optimization can budgets remain controllable in the long run.