LLM API Pricing Explained: How Cache Hit Rate Determines Your Real Token Cost

    List prices for a new LLM generation can rise 3–4x while your actual bill barely moves. This article breaks down three-tier token pricing — input on a cache miss, input on a cache hit, and output — and explains the two price ratios that define the new structure. It shows why disaggregated inference and prefix caching make cached input cheap and generated tokens expensive, then puts numbers to it with a Python cost calculator and two workload scenarios: an enterprise agent with a 90% cache hit rate versus an indie developer at 20%. You will come away with the formula for effective unit price and four practical ways to cut cost through how you call the API.

    1. List Price Nearly Quadrupled. Your Bill Didn’t.

    When a model vendor ships a new generation, most people read only the list price. Output went from 27 to 100 per million tokens — roughly 4x. Input went from 6.5 (cache miss) / 1.1 (cache hit) to 20 / 2 — roughly 3x on both tiers. Taken at face value, the whole price sheet looks like it moved up across the board.

    Stopping there produces a seriously misleading conclusion. LLM billing is never a single unit price multiplied by a usage count. It is a weighted result of three price tiers (input on miss, input on hit, output) combined with how you actually call the model. What determines the bill is not the list price but how well your usage pattern matches the structure of that price sheet.

    2. Anatomy of the Pricing Model: Three Tiers, Two Ratios

    Mainstream LLM APIs bill across three tiers (list prices per million tokens):

    • Input (cache miss): the full prompt is recomputed. Highest input price.
    • Input (cache hit): the prefix of the prompt hits the cache and skips recomputation. Lowest price of the three.
    • Output: tokens generated one at a time by the model. Usually the most expensive tier.

    Two ratios matter more than the absolute numbers:

    1. Cache-miss input price / cache-hit input price: about 6x in the previous generation, stretched to about 10x in the new one (20 vs. 2). Whether a cache hit lands now has a much larger effect on cost.
    2. Output price / cache-hit input price: about 25x in the previous generation, stretched to about 50x (100 vs. 2). The signal is clear: the vendor is using price to push you into putting every reusable prefix into the cache.

    So the smallest increase went to the cache-hit tier and the largest to the output tier. That non-uniform increase is itself a pricing strategy. It does not penalize every user equally — it rewards high-reuse, long-context calling patterns.

    3. Why the Tiers Diverge

    The logic traces back to the inference stack. Two techniques are now standard: disaggregated inference and prefix caching (KV Cache reuse). When many requests share the same system prompt, the same few-shot examples, or the same long document prefix, the KV Cache for that content can be reused and the repeated computation skipped, which cuts compute cost for the cached portion sharply.

    Cost structure gets reallocated accordingly. The cache-hit portion is cheap to compute, so it can be priced low. The output portion cannot be reused and must be forwarded token by token, so its cost is rigid — hence a high price. The vendor hands the discount to the reusable part and passes the cost on to the part that cannot be reused. That is why two users on the same model can end up with bills several times apart.

    4. Two Workloads, Two Very Different Bills

    Theory needs numbers. Assume a general pricing model (per million tokens, figures illustrative):

    • Input, cache miss: 20
    • Input, cache hit: 2
    • Output: 100

    Effective input price:

    1. effective_input = hit_price × cache_hit_rate + miss_price × (1 - cache_hit_rate)
    2. combined = (effective_input × input_tokens + output_price × output_tokens) / total_tokens

    Run it in code:

    1. def effective_cost(input_hit, input_miss, output_price,
    2. cache_hit_rate, in_out_ratio):
    3. """Combined cost per million tokens.
    4. input_hit: input price on a cache hit
    5. input_miss: input price on a cache miss
    6. output_price: output price
    7. cache_hit_rate: cache hit rate (0-1)
    8. in_out_ratio: input-to-output token ratio (20 means 20:1)
    9. """
    10. eff_input = input_hit * cache_hit_rate + \
    11. input_miss * (1 - cache_hit_rate)
    12. in_tokens, out_tokens = in_out_ratio, 1
    13. combined = (eff_input * in_tokens + output_price * out_tokens) \
    14. / (in_tokens + out_tokens)
    15. return eff_input, combined
    16. # Scenario A: enterprise agent - long context, high hit rate
    17. eff_in_a, cost_a = effective_cost(2, 20, 100, 0.9, 20)
    18. # Scenario B: indie developer - short input, low hit rate
    19. eff_in_b, cost_b = effective_cost(2, 20, 100, 0.2, 5)
    20. print(f"Enterprise: effective input {eff_in_a}, combined {cost_a:.2f}/1M tokens")
    21. print(f"Indie dev: effective input {eff_in_b}, combined {cost_b:.2f}/1M tokens")
    22. print(f"Cost gap: {cost_b / cost_a:.1f}x")

    Scenario A (enterprise agent): input-to-output ratio around 20:1, which is typical for long-context coding and document agents, with a 90% cache hit rate.

    • Effective input price ≈ 2 × 0.9 + 20 × 0.1 = 3.8
    • Combined price ≈ (20 × 3.8 + 1 × 100) / 21 ≈ 8.4 per million tokens

    Scenario B (indie developer): occasional calls, a different prompt every time, 20% hit rate, 5:1 input-to-output ratio.

    • Effective input price ≈ 2 × 0.2 + 20 × 0.8 = 16.4
    • Combined price ≈ (5 × 16.4 + 1 × 100) / 6 ≈ 30.3 per million tokens

    Same model. The first user’s unit cost is roughly 1/3.6 of the second’s. The conclusion is straightforward: the so-called across-the-board increase is actually a discounted price for enterprise customers who reuse the same prefixes over the long run, while developers who swap out their prompt every call pay list price or a premium.

    5. Cost Optimization Practices for Developers

    Since the bill is decided by “behavior × structure,” the controllable half is how you call the API:

    1. Keep the system prompt and tool definitions stable. Put the system prompt, tool descriptions, and few-shot examples at the very front of the request and keep them byte-identical. Dynamic fields such as timestamps and random IDs break prefix matching.
    2. Lengthen the reusable prefix. Front-load large blocks such as long documents and repository context and keep them fixed, so the cached KV Cache covers as many tokens as possible.
    3. Control the input-to-output ratio and output length. The output tier is the most expensive, so avoid redundant generation. Structured output and a capped max_tokens both cut cost directly.
    4. Monitor cost with attribution. Log input and output tokens, cache hit rate, and cost per request at the gateway layer, then break the spend down by scenario. Only then can you find the real cost hotspots.

    6. Conclusion

    Evaluating a pricing change means more than counting how many times the list price went up. What matters is the effective unit price — a joint function of cache hit rate, input-to-output ratio, and the three list prices. The real driver of the cost gap is usage pattern: high-reuse, long-context, high-hit-rate workloads are the beneficiaries of this pricing structure, while fragmented, low-hit-rate calls are the ones under pressure. Understand that, and you can spend your budget where it is actually needed instead of being led by the headline numbers.