LLM API Billing Explained: Why Prices Multiply and How to Optimize Costs

    This article breaks down how LLM API pricing actually works and how developers can control costs. It explains why models bill by token, why input and output tokens are priced differently, how cache-hit pricing can substantially lower bills, and how subscription quotas are misaligned with pay-as-you-go API billing. It also provides practical cost-accounting methods and actionable optimization strategies, including improving cache hit rates, compressing input, controlling output length, layered model routing, and choosing the right billing model.

    Large language model APIs have seen frequent pricing adjustments as newer generations of models are released. When developers upgrade to a new-generation model, they often notice a clear jump in unit prices compared with the previous generation: the price per million input tokens rises, output is priced even higher, and cache hits are listed separately at a price far below the regular one. If you fixate on the surface impression that “the unit price has multiplied several times,” you tend to draw one-sided or even incorrect conclusions. To properly assess how costs have changed, you have to go back to the underlying billing logic and understand how tokens are counted, why input and output are priced differently, how cache hits affect the bill, and how subscription-based quotas are misaligned with usage-based (pay-as-you-go) billing.

    1. First, understand why LLMs bill by token

    Large language models are essentially autoregressive models built on the Transformer architecture. They split text into tokens (in Chinese text, a single character often maps to one or more tokens). The model runs a forward pass over the input sequence and then generates output token by token. Because each new output token requires re-running attention over the existing context, the compute involved is directly tied to sequence length.

    For this reason, virtually all vendors use “billing by token” as the basis for pricing rather than per-request or per-word charges. This approach has two key dimensions:

    • Input tokens (prompt): everything the user submits — prompts, system instructions, context, conversation history, and so on.
    • Output tokens (completion): the content the model actually generates and returns.

    Understanding this is the prerequisite for all the analysis that follows, because input and output do not carry equal workloads during computation, and that asymmetry is exactly what drives the difference in their prices.

    2. Why input and output are priced differently

    A common pattern is that, within the same model generation, the output unit price is usually several times the input unit price (a multiplier commonly in the range of 2x to 4x, depending on vendor strategy). This isn’t arbitrary — it has clear computational causes:

    1. Asymmetric compute cost: processing input allows tokens to be computed in parallel (a single forward pass in a teacher-forcing manner), whereas generating output is autoregressive — tokens must be produced one at a time, sequentially, with each step depending on the previous one. Serial generation places a far heavier demand on inference compute than parallel input processing.
    2. Memory and scheduling overhead: the output phase must continuously maintain the KV Cache and perform sampling and decoding at each step, so the inference engine’s scheduling overhead concentrates in the generation phase.
    3. Duration of resource occupancy: a long output keeps an inference node occupied for a long time, whereas input processing typically completes in a very short window.

    So pricing output at several times the input price is essentially making the price reflect real compute-resource consumption. When estimating costs, developers can’t simply use “word count × unit price”; they need to count input and output tokens separately.

    3. Cache-hit pricing: the most easily overlooked cost lever

    Alongside input and output, the price lists of newer models often break out a separate “cache hit” price, and this number is usually far below the regular input price. This mechanism deserves its own discussion, because its impact on cost can exceed the rise or fall in unit price itself.

    How it works: during inference, a model computes and caches the key/value tensors (the KV Cache) for tokens it has already processed. When multiple requests share the same context prefix — for example, the same system prompt, the same long document, or the same code-base context — and the server can reuse the cache for that prefix, it doesn’t have to recompute it, substantially reducing compute. Billing “input tokens that hit the cache” at a discounted rate is the direct result of this optimization.

    What it means in practice: for typical application scenarios, cache hits can often bring meaningful savings:

    • Fixed system prompts that get carried repeatedly throughout a long conversation;
    • Retrieved document snippets cited multiple times in RAG scenarios;
    • Large amounts of repeated code-base context in code-completion and AI-assisted programming scenarios.

    The following illustrative code shows the idea behind cost accounting (the numbers are examples only; actual pricing follows the vendor’s price list):

    1. def estimate_cost(input_tokens, output_tokens, cache_hit_tokens,
    2. price_in, price_out, price_cache):
    3. """
    4. input_tokens: total number of input tokens
    5. output_tokens: total number of output tokens
    6. cache_hit_tokens: input tokens that hit the cache (a subset of input_tokens)
    7. """
    8. billed_input = input_tokens - cache_hit_tokens
    9. cost_in = billed_input * price_in
    10. cost_out = output_tokens * price_out
    11. cost_cache = cache_hit_tokens * price_cache
    12. return cost_in + cost_out + cost_cache

    The key point is that cache_hit_tokens is a subset of the input tokens: the portion that hits the cache is settled at the lower cache price, and only the portion that misses is charged at the regular input price. So there is no simple linear relationship between “input unit price rising” and “the final bill rising” — the higher the cache hit rate, the smaller the real impact of a unit-price increase.

    4. The mismatch between subscription quotas and API billing

    More worth watching than unit price is the mismatch between subscription-based quotas (such as various coding subscriptions) and usage-based API billing. Developers often conflate these two systems when comparing costs:

    • Usage-based API billing: what you see is what you get — every token consumed is settled at the price list, with transparent pricing.
    • Subscription quotas: a monthly or weekly fee bundles a certain amount of usage, subject to rate limits, and the quota is often reset on a cycle.

    This mismatch creates a few real-world problems:

    1. List price vs. perceived cost: the “list price” on the price sheet is only a theoretical ceiling. What subscription users actually get is a capped, resettable bundle of quota, so real perceived cost can quickly converge with — or even reverse against — the API list price.
    2. Reset policies vary widely: different products differ in how often they reset periodic quotas, their concurrency caps, and their peak limits; some reset weekly, others refresh even more frequently. If developers look only at unit price, they can’t accurately predict availability.
    3. Multiple quota pools in parallel: some products bill the chat feature and the coding feature from separate, independent quota pools, adding further complexity to cost accounting.

    The conclusion: to judge whether a model is “expensive,” you have to look at the list price, cache discounts, rate-limit policy, and quota-reset mechanism together, not compare a single isolated unit-price number.

    5. A practical approach to cost accounting

    To move cost control from guesswork to something measurable, developers should establish a costing process:

    1. Instrument token tracking: at the application layer, record the input and output token counts of each request, along with cache-hit counts. Most SDK responses include a usage field, so collect it uniformly and store it.
    2. Model per scenario: account for different use cases (chat, summarization, code completion, RAG Q&A) separately, because their input-to-output ratios vary widely and a single average will be badly distorted.
    3. Define unit-cost metrics: for example, “cost per thousand requests,” “cost per user per month,” or “cost per completed task,” so that technical cost aligns with business metrics.
    4. Continuously benchmark and regress: after a model version upgrade, re-measure actual token consumption rather than reusing old data.

    6. Cost-optimization strategies for developers

    Once you understand the billing mechanics, you can optimize with intent:

    • Increase cache hit rate: keep stable, unchanging content (system instructions, tool definitions, long documents) fixed at the front of the context prefix, and place volatile content later, to maximize prefix reuse.
    • Compress input: trim prompts, remove redundant context, and summarize conversation history rather than carrying it in full.
    • Control output length: use explicit format constraints and length limits to avoid redundant model output, directly reducing the expensive output tokens.
    • Layered routing: use lightweight, low-cost models for simple tasks and reserve high-capability models for complex ones, avoiding overkill.
    • Choose the billing model sensibly: for stable, predictable usage, prefer subscriptions; for fluctuating usage or scenarios that need elastic concurrency, prefer pay-as-you-go, and weigh each against its rate-limit and reset policies.
    • Leverage general cloud capabilities: you can combine object storage for context persistence, message queues for request peak-shaving, and monitoring, alerting, and logging services for usage visibility, embedding cost governance into daily operations. Cloud platforms such as Baidu AI Cloud generally provide these kinds of general-purpose infrastructure capabilities, making it easier to build an observable, controllable invocation pipeline.

    Conclusion

    Headlines like “model X raised prices severalfold” easily draw people into a surface-level “is it expensive or not” debate. But for developers, the valuable judgment comes from breaking down the billing structure: why input and output are priced differently, how cache hits dilute cost, how subscription quotas and API billing are misaligned, and how rate limits and resets affect availability. Once you grasp this logic, you can quickly estimate real costs after any version upgrade, optimize your invocation strategy, and spend every bit of compute budget where it matters — which is far more meaningful than arguing over a single unit-price number.