LLM API Price Hikes: Why Cache Hit Rate Determines Your Real Cost

    As a new generation of large language models raises output and uncached-input prices while keeping cache-hit input cheap, calling the change a simple "price hike" misses the bigger picture. This article explains how LLM API pricing is shifting from coarse-grained toward fine-grained, workload-based pricing, and shows with formulas and two worked scenarios (high-reuse long-context vs. low-reuse ad-hoc) how cache hit rate, input-to-output ratio, and context reuse determine real cost. It covers building your own cost model, practical optimization tactics (stabilizing prompts, post-posing dynamic content, controlling output length, retrieval, model routing, monitoring), platform cache limits and boundary conditions, and an FAQ — concluding that the right question isn't "did prices go up?" but "does my workload suit caching?"

    When a new generation of large language models raises its output price to about ¥100 per million tokens, pushes uncached input to about ¥20 per million tokens, and keeps cache-hit input at roughly ¥1–2 per million tokens, many developers’ first reaction is “prices went up.” But that judgment usually only looks at a single list price and ignores three variables that matter far more in real workloads: the input-to-output ratio, the cache hit rate, and context reuse.

    A note upfront: this article uses illustrative example parameters for its calculations and does not reference any specific vendor’s pricing. Actual prices, caching rules, hit-rate accounting, and billing methods should always be confirmed against the official documentation and your bill.

    The same model service can cost several times more or less depending on how you call it. On the surface, an across-the-board price adjustment looks like “everything got more expensive,” but from an engineering-cost perspective it is more accurately a shift in price structure — from coarse-grained billing toward fine-grained, workload-based pricing. Understanding this matters more than comparing any single tier’s unit price.

    1. It Looks Like a Price Hike, but the Price Structure Is Changing

    In the traditional view, users focus on just two prices: input and output. But in the new generation of LLM services, the input side is typically split into tiers:

    1. Uncached input: the context in the request is not reused from a cache and must be processed in full;
    2. Cached input: shared prefixes, system prompts, tool definitions, and similar content can be reused from cache, so only the new portion needs processing;
    3. Output price: the cost of the tokens the model generates, which usually gets no caching discount.

    Using the example parameters, the structure can be abstracted as:

    Price tier Previous generation (example) New generation (example) Relative change
    Uncached input ~¥6.5 ~¥20 ~3x
    Cached input ~¥1.1 ~¥1–2 Relatively modest
    Output ~¥27 ~¥100 Nearly 4x

    In absolute terms, the output side rose the most, uncached input rose noticeably, and the cache-hit tier changed too but by a smaller relative amount.

    More telling is how the ratios between tiers changed:

    • In the previous generation, uncached input was roughly 6x the cached input price;
    • In the new generation, if cache hits are estimated at ¥2, uncached input is 10x the cache-hit price;
    • The gap between output and cache-hit prices also widened. At ¥2 cache hits, the output-to-cache-hit ratio is about 50x; at ¥1, it is about 100x.

    This amplifies the role of cache hits in multi-tier pricing. A request with a high cache hit rate may enjoy the low tier; one that rebuilds a completely different context every time bears costs near the uncached price — or higher.

    2. The Real Cost, in Formula Form

    LLM inference cost is not determined by token count alone — it is determined by whether recomputation is needed.

    Assume:

    • Uncached input price: input_price_miss
    • Cache-hit price: cache_hit_price
    • Cache hit rate: hit_rate
    • Output price: output_price
    • Input-to-output ratio: input_output_ratio (input tokens ÷ output tokens)

    Then the effective input unit price is:

    1. effective input price = cache_hit_price × hit_rate + input_price_miss × (1 - hit_rate)

    The blended average price can be written as:

    1. blended cost = (input_output_ratio × effective input price + output_price) / (input_output_ratio + 1)

    Note: this formula weights prices in per-million-tokens terms, the hit rate is an input-token-weighted hit rate, and all prices are illustrative — replace them with your platform’s actual billing basis.

    In Python-style pseudocode:

    1. def effective_input_price(input_price_miss, cache_hit_price, hit_rate):
    2. return cache_hit_price * hit_rate + input_price_miss * (1 - hit_rate)
    3. def blended_cost(
    4. input_price_miss,
    5. cache_hit_price,
    6. hit_rate,
    7. output_price,
    8. input_output_ratio
    9. ):
    10. input_price = effective_input_price(
    11. input_price_miss,
    12. cache_hit_price,
    13. hit_rate
    14. )
    15. return (
    16. input_output_ratio * input_price + output_price
    17. ) / (input_output_ratio + 1)

    To see the impact of the hit rate, start with a sensitivity table. Assume an input-to-output ratio of 20:1, uncached input at ¥20 per million tokens, cache hits at an estimated ¥2 per million tokens, and output at ¥100 per million tokens.

    Cache hit rate Effective input price Blended cost
    0% ¥20.0 ~¥23.8
    50% ¥15.0 ~¥19.0
    90% ¥3.8 ~¥8.4
    99% ¥2.18 ~¥6.8

    As the table shows, when input tokens far outnumber output tokens, the cache hit rate significantly changes the effective input price and, in turn, the blended cost.

    3. Two Typical Workloads

    To make this concrete, here are two typical scenarios.

    Scenario 1: High-reuse, long-context workload

    Assume a long-context coding assistant, knowledge-processing system, or long-running agent with these characteristics:

    • Input-to-output ratio of about 20:1;
    • Highly stable system prompts, tool descriptions, and business rules;
    • Many requests sharing the same prefix;
    • Cache hit rate of 90%;
    • Uncached input at ¥20 per million tokens;
    • Cache hits at an estimated ¥2 per million tokens;
    • Output at ¥100 per million tokens.

    The effective input price is then:

    1. 2 × 0.9 + 20 × 0.1 = ¥3.8 per million tokens

    The blended average cost is about:

    1. (20 × 3.8 + 100) / 21 ≈ ¥8.4 per million tokens

    If the cache-hit price is ¥1, the effective input price becomes:

    1. 1 × 0.9 + 20 × 0.1 = ¥2.9 per million tokens

    And the blended average cost is about:

    1. (20 × 2.9 + 100) / 21 ≈ ¥7.5 per million tokens

    In other words, a high cache hit rate drives the input-side cost down sharply. Even with a high output price, because input volume dwarfs output volume, the overall average cost can still be pulled down significantly.

    Scenario 2: Low-reuse, ad-hoc workload

    Now assume a low-reuse developer scenario:

    • Occasional single calls to the model;
    • A different prompt every time;
    • No fixed system prompt, or a system prompt that changes frequently;
    • Input-to-output ratio of about 5:1;
    • Cache hit rate of only 20%;
    • Uncached input at ¥20 per million tokens;
    • Cache hits at an estimated ¥2 per million tokens;
    • Output at ¥100 per million tokens.

    The effective input price is then:

    1. 2 × 0.2 + 20 × 0.8 = ¥16.4 per million tokens

    The blended average cost is about:

    1. (5 × 16.4 + 100) / 6 ≈ ¥30.3 per million tokens

    Same model, same nominal prices — yet the blended cost across these two workloads differs by roughly 3.6x. When the high-reuse workload also gets a lower cache-hit price, the gap can widen to about 4x. This difference is not the model charging different users different rates; it is the combined result of request structure, cache hit rate, and input-to-output ratio.

    4. High-Reuse vs. Low-Reuse Workloads

    Rather than debating “who gets the discount and who pays full price,” it is more useful to look at workload characteristics directly. The table below puts high-reuse and low-reuse workloads side by side:

    Dimension High-reuse workload Low-reuse workload
    Typical scenario Long-running agents, fixed templates, knowledge-base Q&A, batch jobs Ad-hoc calls, short chats, frequently edited prompts
    System prompt Stable, versioned Frequently changed or missing
    Context structure Long, stable shared prefix Short, discontinuous context
    Cache hit rate Higher Lower
    Input-to-output ratio Input-heavy, output-light Input-light or output-heavy
    Effective input price Near the cache-hit tier Near the uncached tier
    Cost outcome Closer to a structural discount Closer to full price, or higher under load

    The takeaway: what determines real cost is not a single list price but the combination of cache hit rate, input-to-output ratio, context stability, and request reuse.

    1. Cache hit rate

    The cache hit rate is the amplifier of the cost structure. A high hit rate pushes the input side toward the low tier; a low hit rate pushes it toward the uncached tier.

    2. Input-to-output ratio

    When input tokens are numerous and output tokens are few, input-side cost dominates and the value of caching becomes more obvious. Conversely, when output is long and input is short, the output price dominates and the impact of caching diminishes.

    3. Context stability

    Caching usually depends on a stable prefix or shared portion. If you dynamically modify the system prompt, tool definitions, or task instructions on every request, the hit rate can drop.

    4. Request reuse

    Batch jobs, fixed templates, and long-running agents tend to reuse context more readily than random, exploratory calls — and are more likely to benefit from caching.

    So the “did prices go up or down” question cannot be answered from a price table alone; it has to be evaluated against real business workloads.

    5. How to Judge Your Own Real Cost

    If you are evaluating an LLM service, do not just look at “how much is input, how much is output.” Build your own cost model. At minimum, track these metrics:

    1. Average input tokens — including system prompts, context, retrieved content, and message history.
    2. Average output tokens — including final answers, structured results, code, and summaries.
    3. Input-to-output ratio — input tokens ÷ output tokens.
    4. Cache hit rate — if the platform provides hit statistics, use them; otherwise estimate from request structure. A common estimate is stable-prefix tokens ÷ total input tokens. Also record hit-rate fluctuations from TTL, version invalidation, and retries.
    5. Average request interval and reuse rate — whether the same system prompt persists over time, whether the knowledge base is fixed, and whether tool definitions are stable.
    6. Failure/retry rate — retries amplify token consumption, especially input-side cost.
    7. Total cost per task — do not look only at per-million-token prices; look at how many tokens a complete business task actually consumes.

    A simplified evaluation script might look like:

    1. input_price_miss = 20
    2. cache_hit_price = 2
    3. output_price = 100
    4. scenarios = [
    5. {
    6. "name": "high_reuse",
    7. "hit_rate": 0.90,
    8. "input_output_ratio": 20
    9. },
    10. {
    11. "name": "low_reuse",
    12. "hit_rate": 0.20,
    13. "input_output_ratio": 5
    14. }
    15. ]
    16. for scenario in scenarios:
    17. cost = blended_cost(
    18. input_price_miss,
    19. cache_hit_price,
    20. scenario["hit_rate"],
    21. output_price,
    22. scenario["input_output_ratio"]
    23. )
    24. print(scenario["name"], cost)

    The value of this code is turning “it feels more expensive” into a quantifiable judgment.

    6. The Key to Cutting Costs: Raise the Cache Hit Rate, Don’t Just Shrink Context

    When developers think about cost optimization, the first instinct is often to reduce input tokens. That works, but if the cache hit rate is low, shrinking context can actually hurt task quality. A more systematic approach is to raise the reusable proportion of your request structure.

    1. Stabilize system prompts and task templates

    If the system prompt, role definition, output specification, and tool descriptions change frequently, caching is hard. Enterprise applications should version and templatize these and avoid ad-hoc concatenation on every request.

    Recommended practices:

    • Put fixed rules in the system prompt;
    • Place dynamic data later;
    • Avoid inserting timestamps, random IDs, or temporary variables into the shared prefix;
    • Use stable prompt template versions.

    2. Put dynamic content at the end

    When the shared prefix is identical and only later content differs, caching works better. In general, earlier content is more likely to be reused, so keep invariant content up front and variable content at the end.

    For example, a request can be organized as:

    1. [System role]
    2. [Fixed business rules]
    3. [Tool definitions]
    4. [Knowledge-base snippets]
    5. [History summary]
    6. [Current user question]

    If the first four items are stable, they have a chance to form a high-hit cache.

    3. Control output length

    When the output side is expensive, controlling output length is more direct than shrinking input.

    Options include:

    • Specify the output format;
    • Cap the maximum output tokens;
    • Use summary-style answers;
    • Use structured fields instead of lengthy prose;
    • Avoid asking the model for overly long explanations on simple tasks.

    For example, state clearly in the system prompt:

    1. Output JSON only, with no extra explanation.
    2. Include these fields: conclusion, reason, risk, recommendation.
    3. Keep the total under 200 words.
    4. Max output tokens: 800.
    5. Retry limit on failure: 2.

    This reduces output token consumption and also cuts the extra cost of ineffective retries.

    4. Use retrieval instead of stuffing in the full context

    If the knowledge base is large, do not stuff everything into the request. A more sensible approach is to retrieve first, then inject only the snippets most relevant to the current question.

    This both reduces input tokens and improves context relevance. But note: if retrieval results change frequently, they can also hurt the cache hit rate. So stable knowledge versions and snippet-ordering strategies matter.

    5. Route requests by tier

    Not every task needs a high-capability model. For simple classification, summarization, formatting, and rule matching, route to a lighter, cheaper model; reserve high-capability models for complex reasoning, code generation, and long-context analysis.

    Example logic:

    1. def choose_model(task):
    2. if task.requires_deep_reasoning:
    3. return "high_capability_model"
    4. if task.max_input_tokens > 100000:
    5. return "long_context_model"
    6. return "cost_efficient_model"

    Here, high_capability_model, long_context_model, and cost_efficient_model are generic labels and do not correspond to any specific product.

    6. Build cost monitoring and alerts

    If you cannot observe cost, you cannot optimize it. At minimum, record:

    • Input tokens per request;
    • Output tokens per request;
    • Cache hit status;
    • Average latency;
    • Cost per task;
    • Daily total spend;
    • Abnormally high-frequency calls;
    • Failed-retry counts.

    When cost grows abnormally, you should be able to quickly pinpoint whether it is runaway input, overly long output, a drop in cache hit rate, or a change to some task template.

    7. Platform Cache Limits and Boundary Conditions

    Cache hits can substantially lower cost, but they have boundaries. Caching mechanisms vary widely across platforms, and “cache hit” should not be understood as a universal discount everywhere.

    1. Minimum cacheable prefix

    Some platforms require cached content to reach a minimum length, or only cache a prefix once it meets a certain token count. If the system prompt or shared context is too short, an effective cache may not form.

    2. Cache lifetime

    Caches usually have a TTL (time to live). If the interval between requests is too long, the cache may have expired; if requests are frequent and structurally stable, the cache is more likely to stay hit.

    3. Exact-prefix matching

    Caching often depends on an exact prefix match. If the system prompt, tool definitions, knowledge-base snippets, or template version change in any way, a cache that would have hit may be invalidated.

    4. Cache write or storage fees

    Some platforms may charge for cache writes, cache storage, or special caching features, and may only offer an input-side hit-price discount. Confirm the platform’s billing basis before integrating.

    5. Caching usually affects only the input side

    In common billing models, caching mainly affects the cost of computing input tokens; the output side is usually still billed at the output price. Whether it affects output at all depends on the platform’s rules.

    6. Dynamic requests are not well suited to heavy caching

    If the system prompt is completely different on every request, or if you must inject large amounts of temporary data each time, the cache hit rate will be low.

    7. Cache invalidation can cause cost spikes

    When business rules, tool definitions, or knowledge-base versions change, a previously stable cache can be invalidated. Without a gradual-rollout (canary) mechanism, cost can rise noticeably in a short time.

    8. High-quality tasks shouldn’t chase only low cost

    For complex reasoning, critical decisions, code review, compliance analysis, and similar tasks, over-compressing context can damage output quality. Cost optimization must rest on an acceptable quality baseline.

    9. Caching involves isolation and security concerns

    If a cache or logs involve multi-tenant data, ensure data isolation, permission isolation, and lifecycle management across customers. Before enterprise integration, check whether the platform has the corresponding security and compliance capabilities.

    8. FAQ

    What if my cache hit rate is low?

    First check whether your request structure is stable. Fix the system prompt, tool definitions, and business rules, and push dynamic content to the end. Also check for timestamps, random IDs, temporary variables, or other factors that frequently change the prefix. If the platform provides hit-rate statistics, compare hit rates across template versions.

    How do I cut costs when output is too long?

    First cap the maximum output tokens, then require structured output — for example, JSON, summaries, tables, or fixed fields only. For simple tasks, avoid asking the model for long explanations. When output makes up a large share, controlling output length is often more direct than compressing input.

    Is caching charged separately?

    It depends on the platform’s rules. Some platforms only lower the input hit price; others may charge separately for cache writes or storage. When evaluating, do not look only at the cache-hit unit price — look at the full bill and billing documentation.

    How do I avoid cost spikes from cache invalidation?

    Version your prompt templates, knowledge-base versions, and tool definitions, and roll out changes gradually (canary). At the same time, monitor the cache hit rate, input tokens, and cost per task. If the hit rate drops noticeably, roll back the template promptly or restore the stable cache structure.

    9. Conclusion: Don’t Just Ask “Did It Go Up?” — Ask “Does My Workload Suit Caching?”

    Changes in the price tiers of the new generation of LLM services do affect the cost expectations of developers and enterprises. But if you only stare at list prices like “output ¥100, input ¥20, cache hit ¥1,” you will easily reach a one-sided conclusion.

    A more rigorous approach:

    1. Check whether the input side distinguishes cache hits from misses;
    2. Check whether the cache-hit tier is priced well below the output tier;
    3. Check whether your input-to-output ratio suits cache optimization;
    4. Check whether your business requests have stable prefixes and high reuse;
    5. Check whether monitoring, templating, retrieval augmentation, and model routing can lower your real cost.

    For long-running, context-stable, input-heavy, high-reuse scenarios, caching can turn a nominal price increase into an actual cost decrease. For low-reuse, short-request, output-heavy scenarios, the same pricing system can look expensive.

    So the essence of the price change is not a simple “increase” or “decrease” — it is that LLM service pricing is shifting from a single token price toward finer-grained, workload-structure-based pricing. What developers truly need to build is not price-table awareness, but cost-structure awareness and engineering optimization skills.