LLM API Token Pricing Explained: Tiered Billing and Cost Optimization Practices

    This article explains why large language model (LLM) APIs bill per token across three tiers—uncached input, cached input, and output—and how the widening price gaps between them shape real costs. It walks through the structure of a typical price sheet, what shifting price ratios mean for developers, and practical levers for reducing spend: maximizing cache hit rates, constraining output length, trimming inputs, and designing a unified abstraction layer so you can retune quickly when pricing changes. The goal is to help teams build token-level cost awareness and keep the balance between experience and expense.

    1. Why token billing is tiered

    LLM APIs charge per token, not per request or per concurrent connection. The resources consumed by a single request are not evenly distributed: the model first runs prefill on the input, then generates output token by token during decode. These two phases differ in compute cost by orders of magnitude—generation repeatedly reads all content produced so far, which is far more expensive than reading the input.

    For this reason, mainstream LLM services split their pricing into three tiers: uncached input, cached input hits, and output, rather than quoting a single “price per thousand tokens.” Understanding these three tiers is the prerequisite for reading any price sheet, and the foundation for the cost optimization that follows.

    2. Reading the three-tier price structure

    Using an illustrative example of how a model service’s pricing shifted across two recent generations (the numbers only illustrate the structure and are not a specific product’s actual pricing):

    • Uncached input: roughly 6.5 → 20 (per million tokens), up about 3x;
    • Cached input hit: roughly 1.1 → 2 (per million tokens), also up around 3x, and the mildest of the three tiers;
    • Output: roughly 27 → 100 (per million tokens), up close to 4x, the steepest of the three.

    What is more revealing is the shift in the ratios between tiers:

    • Uncached input / cached input hit widened from about 6x in the previous generation to about 10x;
    • Output / cached input hit widened from about 25x to about 50x.

    This shows that the new generation is not a uniform proportional increase across the board. Instead, it structurally raises the output price while widening the discount available for cache hits. The fact that the cache-hit tier rose the least is essentially the provider using price as a lever to encourage callers to reuse prefix context, thereby reducing its own repeated computation load.

    3. What the ratio changes mean for developers

    The widening ratios lead to two direct conclusions:

    1. Output is the main cost driver. When output reaches 50x the cached input hit, the bill for any long-form generation scenario (code completion, document generation, long-document summarization) will be dominated by the output side. Controlling the length and necessity of output is more effective than obsessing over the input unit price.
    2. Cache reuse is the biggest lever. The 10x gap between uncached and cached input means that the same piece of context costs an order of magnitude more to recompute than to hit from cache. Stable system prompts, tool definitions, and few-shot examples are all natural candidates for cache hits.

    4. Core cost optimization levers

    1. Maximize the cache hit rate

    Placing the least-changing content at the very front of the prompt and the per-request user input at the end is the general practice for improving hit rates. Example:

    1. # Recommended: stable prefix + dynamic tail
    2. messages = [
    3. {"role": "system", "content": SYSTEM_PROMPT}, # Stable, easy to cache
    4. {"role": "system", "content": TOOL_DEFINITIONS}, # Stable, easy to cache
    5. {"role": "system", "content": FEW_SHOT_EXAMPLES}, # Stable, easy to cache
    6. {"role": "user", "content": current_user_input}, # Changes every time, put last
    7. ]

    Note two things: first, the prefix must be byte-for-byte identical (including spaces and newlines) to qualify for a cache hit; second, caches usually have a time-to-live (TTL), so in low-frequency scenarios the prerequisite for a hit is that requests are not spaced too far apart.

    2. Compress output length

    Because output carries the highest unit price, actively constrain the generation size:

    1. # Use stop sequences, a max_tokens cap, and structured output to bound length
    2. resp = call_model(
    3. messages=messages,
    4. max_tokens=512, # Explicit cap to avoid verbose output
    5. stop=["\n\nEND"], # Stop early with a delimiter in structured scenarios
    6. )

    For tasks that can return structured results (extracting fields, generating JSON), asking the model to produce a concise structure directly rather than a lengthy explanation can significantly reduce output tokens.

    3. Slim down and denoise the input

    Pre-trim or summarize long retrieved documents, and feed only the relevant snippets into the context; trim tool definitions to what is needed; keep few-shot examples to “just enough” rather than as many as possible. The input unit price is low, but under high call volume it still adds up.

    4. Amortize cost with call strategy

    • Schedule batch offline jobs during low-price windows or via asynchronous queues rather than real-time synchronous paths;
    • Streaming output improves the experience while making it easier to decide mid-stream whether generation can end early;
    • Set up token consumption monitoring (using general-purpose logging and alerting capabilities), break down the bill by business line, and identify high-consumption call points.

    5. Selection and architecture recommendations

    Facing a “structural price increase” driven by a generational upgrade, deciding whether a new model is worth switching to requires more than looking at a single tier’s price. Build an estimation model tailored to your own call profile:

    • Measure the actual distribution of input tokens, output tokens, and cache hit ratio in your workload;
    • Recompute the cost per request against the new price sheet using this distribution, rather than linearly extrapolating from old prices;
    • Pay particular attention to whether your scenario is “output-heavy”—if so, you are highly sensitive to price increases; if it is “input-heavy with a stable prefix,” the cache hit rate becomes the decisive factor.

    From an engineering standpoint, wrap model calls in a unified abstraction layer that centrally manages cache strategy, token limits, and fallback routing (splitting traffic between expensive and lightweight models by task complexity). That way, when prices change again, you can retune quickly without touching business code.

    Conclusion

    Tiered billing is not arbitrary pricing by providers; it is a faithful mapping of the underlying cost structure of model inference: output is the most expensive, uncached input is next, and cached input is the cheapest. Once you understand this, the direction for cost optimization becomes clear—use a stable prefix to earn cache hits, use output constraints to save on unit price, and use call monitoring to keep everything under control. As model capabilities keep improving and price structures keep shifting, teams with token-level cost awareness and architectural isolation will be the ones that truly balance experience against expense.