LLM API Cost Calculation: Breaking Down Billing Dimensions and How to Cut Costs

    Most teams estimate LLM API spend from a single number — the price per million tokens — and are then surprised by the bill. Two workloads generating tens of millions of requests per day against the same model can differ by several multiples in cost, because billing is a compound outcome of metering unit, input/output structure, cache reuse, batch form, and request time window. This vendor-neutral article explains each layer of cost, how the layers couple, and the engineering levers available: prompt and output trimming, prefix cache design including hit pricing, storage fees, TTL and write premiums, batch discounts, off-peak pricing, and the hard rule that some discounts cannot be stacked. It closes with a structured estimation function that makes cost trade-offs explicit, plus a practical cost-reduction checklist.

    1. Why API Costs Are Routinely Underestimated

    When estimating the cost of adopting an LLM API, many teams look at one number only: price per million tokens. The real bill often lands far from that estimate. Even at the same request volume — say, tens of millions of requests per day — two workloads running the same model can differ severalfold in cost.

    The reason is that API billing is not a single-price problem. It is a composite result of multiple dimensions acting together: metering unit, input/output structure, cache reuse, batch form, and the time window of each request.

    Understanding these dimensions is the prerequisite for capacity planning and cost governance. This article does not target any specific vendor’s price list. Instead, it starts from a general billing model and explains how each layer of cost arises, how the layers couple to one another, and how engineering practices can push the total down.

    2. Token Metering: The Baseline

    LLM APIs meter usage in tokens rather than characters or words. Roughly, one token corresponds to 1.5–2 Chinese characters, or about 0.75 English words, though actual tokenization is decided by the model’s own tokenizer. Two conclusions follow directly:

    1. Prompt engineering has a direct cost impact. Redundant context, repeated system instructions, and pointless conversation history all convert into real input-token charges. Trimming the prompt is the plainest and often most effective cost lever.
    2. Input and output are priced separately. Most platforms set a distinct unit price for each, and output is usually the more expensive side.

    Why input and output are asymmetric

    The input phase is a “read” operation: the model encodes the whole context in one parallel pass. The output phase is a “write” operation: tokens are generated autoregressively one by one, and producing each new token passes through the network again. Computational cost accumulates roughly linearly with output length, so a higher output price than input price is an industry norm, commonly with a ratio in the 2–6× range.

    The practical implication: controlling output length pays off far more than controlling input length. Where the task allows it, compressing the output through structured constraints — limiting fields, limiting item counts — often saves more than deleting input.

    3. Caching: The Most Overlooked Cost Lever

    For workloads with a highly repeated prefix — a fixed system prompt, unchanged few-shot examples, repeated Q&A over a long document — prompt caching is the highest-return optimization available. Billing typically splits into three parts.

    3.1 Cache hits vs. misses

    The first request must write the prefix into the cache and is charged at the miss price. Later requests that match the same prefix are charged at a far lower cache-hit price for that portion of the input. The gap is usually steep: the hit price can be as little as 10–20% of the normal input price. For a stable system prompt, once the hit rate is saturated, input cost can drop by an order of magnitude.

    The key engineering requirement is prefix stability. Matching is prefix-based, so any change at any position invalidates everything after it. Stable content — system instructions, tool definitions, fixed examples — belongs at the front; dynamic content — user input, timestamps, session IDs — belongs at the end.

    3.2 Cache storage fees and TTL

    The cache is not free. Most platforms bill for cache residency as capacity × time (for example, per million tokens per hour). This means:

    • The higher the hit rate, the better the economics. A low-hit-rate cache becomes pure overhead.
    • Caches have a time-to-live (TTL). If the interval between two requests exceeds the TTL, the cache expires, must be written again, and the hit benefit drops to zero.
    • A long TTL suits stable, low-frequency reuse; a short TTL suits bursty hotspots.

    3.3 Cache write costs

    Some platforms charge separately for cache writes, tiered by duration: maintaining a shorter TTL after the write is one price, maintaining a longer TTL is a higher price (because storage resources are occupied for longer). When calculating the return on caching, the write premium must go into the denominator:

    1. Net cache savings = hits × (miss price − hit price) × cached tokens
    2. − cache write premium
    3. − cache storage fee (capacity price × resident tokens × hours)

    Only when the net figure is positive does enabling the cache actually save money.

    4. Batching and Time-Based Pricing

    4.1 Batch discounts

    For latency-insensitive work — offline summarization, bulk translation, data cleaning, log labeling — using the batch interface typically earns a meaningful discount (commonly 50–60% of list price). The cost is latency: response times stretch from seconds to minutes or even hours. Whether to enable it depends on whether the workload can be made asynchronous.

    4.2 Time-based pricing

    Some platforms price peak and off-peak windows differently. Peak usually covers weekday working hours; off-peak covers nights, weekends, and public holidays. The two unit prices can differ by up to 2×. Billing is typically determined by the time window of the request completion moment, not the moment the request was submitted.

    This rule creates scheduling room for deferrable work: move offline batches to nights or weekends and you effectively get half price.

    4.3 Discount stacking limits (the easiest trap)

    Batch discounts and cache discounts often cannot both apply. This is a hard constraint that is easy to have written into the rules and just as easy to miss. Before optimizing, work out which combination is actually cheaper: when a request could both hit the cache and run through batch, estimate the effective unit price of the “cache only” and “batch only” paths separately, then take the lower one rather than assuming they stack.

    5. Building a Cost Estimation Model

    Turning these dimensions into a single estimation function is the key step in cost governance. The sketch below shows how to accumulate each component structurally (all prices are illustrative; use the official pricing page of the platform you choose):

    1. def estimate_cost(
    2. input_tokens: int,
    3. output_tokens: int,
    4. cache_hit_tokens: int = 0,
    5. use_batch: bool = False,
    6. is_peak: bool = True,
    7. cache_hours: float = 0.0,
    8. ):
    9. # Illustrative unit prices (per 1M tokens); set these per your platform
    10. price_in, price_out = 4.0, 16.0
    11. price_cache_hit = 0.4
    12. price_cache_store = 0.02 # per 1M tokens per hour
    13. batch_factor = 0.5 # batch = 50% of list price
    14. peak_factor = 1.0 if is_peak else 0.5 # off-peak = half price
    15. miss_tokens = input_tokens - cache_hit_tokens
    16. cost_in = miss_tokens * price_in / 1e6
    17. cost_hit = cache_hit_tokens * price_cache_hit / 1e6
    18. cost_out = output_tokens * price_out / 1e6
    19. cost_store = cache_hit_tokens * price_cache_store * cache_hours / 1e6
    20. base = cost_in + cost_hit + cost_out
    21. if use_batch: # note: batch and cache treated as non-stackable here
    22. base = max(base - cost_hit, 0) * batch_factor + cost_hit
    23. total = base * peak_factor + cost_store
    24. return total

    The value of such a model is not precision but making decisions explicit: change one parameter — turn caching off, shift to overnight — and the cost change is visible immediately, which prevents optimization by guesswork.

    6. Cost Reduction Checklist

    • Trim prompts and outputs. The highest return on effort: it cuts both input and output cost at the same time.
    • Maximize prefix caching. Stable content first, dynamic content last; evaluate hit rate and net savings before enabling.
    • Use batching and off-peak scheduling. Push offline work into asynchronous batches and idle windows.
    • Route by model tier. Send simple tasks to lightweight, cheap models and reserve high-capability, high-price models for complex ones, instead of using a sledgehammer to crack a nut.
    • Do the stacking math. When cache and batch discounts are mutually exclusive, compare prices case by case and pick the better one.
    • Make usage observable. Track token consumption and cache hit rates through monitoring, alerting, and log services so that cost is attributable and can be forecast with warnings.

    7. Conclusion

    The essence of API cost is the multiplicative effect of many billing rules, not a single unit price. Token metering determines the payoff from prompt optimization; input/output asymmetry explains why compressing output is the better deal; caching, batching, and time-based pricing provide three adjustable levers; and the non-stacking constraints between them are a reminder that every scenario must be priced out individually. Codify this logic into an estimation model and a routing policy, and you can keep quality where it matters while making every token you pay for earn its keep.