LLM Pricing and Cache Hit Rates: How to Calculate Your Real Token Costs

    This article breaks down how the latest generation of flagship LLM pricing shifts cost pressure from reusable inputs to non-reusable outputs, and why cache hit rate becomes the single most important cost variable. Using an illustrative three-tier price sheet, it introduces an effective unit price model based on prefix cache hit rate and input/output ratio, then works through concrete scenarios—from long-context enterprise agents to sporadic individual calls—to show that real price increases vary widely (roughly 2.9x to 3.4x) rather than the headline "4-5x." It explains why decode-stage output tokens are priced far higher than input, covers practical engineering steps for maximizing cache hits (byte-stable prefixes, tail-loaded dynamic content, token and cost logging), and offers a decision workflow for judging whether the new pricing is a gain or a loss for your own workload.

    LLM Pricing and Cache Hit Rates: How to Calculate Your Real Token Costs

    1. First, find out which tier the “price increase” actually landed on

    Once the newest flagship models launched, claims like “output pricing is nearly 4x the previous generation” and “input pricing shifted up across the board” spread quickly through the technical community. But if you only look at the list price, it’s easy to jump to the intuition that “everything went up massively and it’s unacceptable.” The reality needs to be unpacked.

    Take the three-tier pricing model common across the industry (the numbers below are illustrative price tags for easier calculation, in $/million tokens, and do not represent any specific vendor’s actual pricing):

    Tier Previous gen New gen Multiplier
    Input · cache miss 6.5 20 ≈3.1x
    Input · cache hit 1.1 2 ≈1.8x
    Output 27 100 ≈3.7x

    Three observations:

    1. The cache-hit tier saw the mildest increase (under 2x), cache miss came next, and output saw the steepest increase (close to 4x). In other words, the weight of the price increase is clearly loaded onto the “non-reusable” portion.
    2. The relative ratios between tiers were stretched apart. In the previous generation, “miss/hit” was about 6x and “output/hit” about 25x; in the new generation they became 10x and 50x respectively. The price structure itself has become a strong incentive signal.
    3. Therefore, a description like “5x the previous generation” only holds for specific usage profiles. For high-reuse workloads, the actual increase is far below the list-price increase.

    2. Don’t do the math with list prices—do it with “effective unit price”

    Total cost is never a single-tier price; it’s a weighted value determined by both the hit rate and the input/output ratio. Define:

    • r: prefix cache hit rate (0–1)
    • k: input/output ratio (input tokens / output tokens per interaction)

    The effective input unit price and the blended unit price are:

    1. effective_in = P_hit · r + P_miss · (1 - r)
    2. blended_price = (k · effective_in + P_out) / (1 + k)

    Scenario A: Enterprise long-context agent (r=0.9, k=20)

    Fixed system prompt + fixed tool definitions, hundreds of thousands of tokens consumed per task:

    1. effective_in = 2×0.9 + 20×0.1 = 3.8
    2. blended_price = (20×3.8 + 1×100) / 21 ≈ 8.4 $/million tokens

    Compare with the same profile on the previous generation (miss 6.5, hit 1.1, output 27): input side 1.64, blended about 2.85. The actual increase is about 2.9x, not 3–4x—a high hit rate absorbs most of the increase.

    Scenario B: Individual developer with sporadic calls (r=0.2, k=5)

    Different requests each time, highly variable prompts:

    1. effective_in = 2×0.2 + 20×0.8 = 16.4
    2. blended_price = (5×16.4 + 1×100) / 6 ≈ 30.3 $/million tokens

    Compare with the same profile on the previous generation (5.42 and 27): blended about 9.0, an actual increase of about 3.4x.

    Scenario C: One-off long task, very low reuse (r=0, k=10)

    1. blended_price = (10×20 + 100) / 11 ≈ 27.3 $/million tokens

    Put the three scenarios in one table and the conclusion is far clearer than “how many times did it go up”:

    Scenario Hit rate Input/output ratio Blended price (new gen) vs. previous gen
    A Enterprise long-context agent 90% 20:1 ≈8.4 ≈2.9x
    B Sporadic individual calls 20% 5:1 ≈30.3 ≈3.4x
    C One-off long task 0% 10:1 ≈27.3 —

    The absolute cost gap between A and B is about 3.6x. The list prices rose uniformly, but the effective unit prices they translate into are diverging: users with fixed prefixes, long sessions, and batch offline tasks get a discount, while users who rewrite their prompt every time pay the full price or even a premium.

    3. In multi-turn conversations, the value of caching is amplified further

    Single-turn RAG scenarios usually don’t show much difference; multi-turn conversations are where the dividing line appears. Assume:

    • Fixed prefix (system prompt + tool schema) = 8,000 tokens
    • ~1,500 new tokens added per turn (user input + tool return results)
    • 20 turns total, each carrying the full history

    Without caching, the input for turn t is about 8000 + 1500×(t-1), and the cumulative total over 20 turns is about 445,000 input tokens, all billed as cache miss → 8.9.

    With prefix caching enabled, each turn only pays the miss price for the newly added 1,500 tokens, while the historical portion goes through the hit price: cumulative hits are about 408,500 tokens and misses about 36,500 tokens → 1.55.

    Same conversation content, a 5.7x cost difference, with an actual hit rate as high as 91.8%—landing right in the “over 90%” range vendors claim. This also explains why the new pricing dares to push the miss price so high: the architecture side can already push the hit rate stably high, so the remaining problem becomes “did you write your request correctly?”

    4. Why output is so much more expensive than input: the physical difference between the two-stage costs

    LLM inference has two phases, and their hardware bottlenecks are completely different.

    Prefill (processing input): The sequence is fed into the model in parallel all at once, the matrix multiplication is large-scale, arithmetic intensity is high, and the bottleneck is compute—GPU utilization is easy to saturate. The cost is basically linear with input length.

    Decode (generating output): Every generated token requires reading the entire set of active weights once, arithmetic intensity is extremely low, and the bottleneck is memory bandwidth rather than compute. Worse, each step must also read the entire KV Cache of the current session, so the per-token cost grows with context length—this isn’t a linear add-on; it’s “the longer the context, the more expensive each subsequent character becomes.”

    This directly explains the shape of the price sheet: output is priced at 5x the input miss price and 50x the hit price precisely because the decode phase’s bandwidth consumption and KV storage pressure are inherently “heavy resources.” Conversely, what a cache hit saves is exactly that reusable computation and KV construction in prefill, which is why the hit price can be as low as close to storage cost.

    Also, sparse MoE architectures (trillions of total parameters, activating only a few experts per token) don’t simply equate to “cheap”:

    • Resident memory holds all expert weights, independent of the number activated, which caps the concurrency a single card can carry;
    • Expert routing introduces all-to-all communication, and in cross-node deployments the network becomes an extra cost item;
    • Different requests within a batch activate different experts, reducing effective batch utilization;
    • New structures like hybrid/linear attention depend on custom operators, and both kernel efficiency and numerical stability require additional investment.

    These costs won’t show up in the phrase “only a few parameters are activated,” but they will definitely show up in output-side pricing.

    5. Engineering practice: make cache hit rate a first-class metric

    Since the pricing structure has already folded hit rate into a cost difference of 2x or more, caching strategy shouldn’t just be the SDK’s default behavior—it should be governed as a first-class metric for both performance and cost.

    1) The prefix must be byte-for-byte stable. Timestamps, random IDs, usernames, the insertion position of retrieval results, field order in tool schemas, JSON serialization order, whitespace and line breaks—any jitter anywhere invalidates all subsequent caching. Once the longest common prefix used for cache matching breaks, the further back the content sits, the harder it is to hit.

    2) Load all dynamic content at the tail. Move anything that “changes every turn” to the end of the message sequence, keeping the stable prefix as long as possible:

    1. import json
    2. SYSTEM_PROMPT = """You are a coding assistant. You must follow these constraints:
    3. 1. Output only the requested changes;
    4. 2. Ask first when uncertain;
    5. 3. Preserve the existing code style.
    6. """ # Stable; no timestamps, tenant names, or random IDs appended
    7. TOOL_SCHEMAS = [...] # Use a fixed-order list; never generate dynamically from a dict/set
    8. def build_messages(history, now, retrieved):
    9. # Put all dynamic fragments in the final user message
    10. dynamic = {
    11. "current_time": now.isoformat(timespec="seconds"),
    12. "retrieved_snippets": retrieved, # Retrieval results appended at the tail, not at the start of the prompt
    13. }
    14. return [
    15. {"role": "system", "content": SYSTEM_PROMPT},
    16. *history,
    17. {"role": "user", "content": "Context:\n" + json.dumps(dynamic, ensure_ascii=False)},
    18. ]

    3) Calculate the effective unit price yourself—don’t go by feel. Parameterize the price sheet and plug it into cost accounting:

    1. CACHE_HIT, CACHE_MISS, OUT = 2.0, 20.0, 100.0 # $ / million tokens
    2. def blended_price(r: float, k: float) -> float:
    3. """r: prefix cache hit rate; k: input/output token ratio"""
    4. eff_in = CACHE_HIT * r + CACHE_MISS * (1 - r)
    5. return (k * eff_in + OUT) / (1 + k)
    6. # How the blended price changes as hit rate goes from 0.5 to 0.9
    7. for r in (0.0, 0.5, 0.8, 0.9):
    8. print(f"hit={r:.2f} k=20 -> {blended_price(r, 20):.2f} $/million tokens")

    4) Request logs must include token and cache-hit fields. At minimum, log prompt_tokens, completion_tokens, cache_hit_tokens, session ID, and prompt version, aggregated by session/tenant/tool. Pair this with a logging service for long-term retention, and use monitoring and alerting to set rules for “hit rate dropping below a threshold” and “abnormal per-task token consumption.” Without these fields, you can’t do post-hoc attribution, nor prove that prefix jitter is the cause of rising costs.

    5) Structural cost-reduction levers, ranked by payoff:

    • Increase reuse: fix the system prompt and tool definitions, inject retrieval results at the tail, and keep a session on the same routing instance;
    • Cut output tokens: the output side has the highest unit price, so cap max generation length, require structured and concise output, and switch long-form generation to streaming + segmented validation—the payoff often exceeds optimizing input;
    • Control context length: periodically compress or summarize sessions, turning long history into a short summary before it enters the context—this reduces both input volume and the per-token KV read cost in the decode phase;
    • Tiered routing: send simple intents to a small model or rule-based path, and use the flagship model only for complex tasks; batch offline tasks together to improve cache hit rate;
    • Inference-serving side: use a container platform for elastic scaling and batch scheduling, a message queue to smooth peaks and guarantee queuing semantics for long tasks, and object storage to hold session snapshots and intermediate artifacts, avoiding recomputation of long tasks.

    6. How to tell whether the new pricing is a gain or a loss for you

    An actionable decision process:

    1. Sample a week of real traffic and compute the actual hit rate r and input/output ratio k (don’t use hand-estimated values);
    2. Plug them into the blended_price formula above and compare with the result for the same profile on the previous generation to get your own multiplier;
    3. If the increase is significantly higher than the list-price increase, there’s room to optimize the hit rate or prefix stability—prioritize the position of dynamic content and tool-schema jitter;
    4. If output tokens make up a large share (long-form generation, multi-step reasoning output), shift the optimization focus to the output side rather than the input side;
    5. Put blended_price and hit rate on a daily dashboard as inputs to capacity planning and budget approval.

    7. Conclusion

    The price adjustment of the new flagship models essentially shifts cost pressure from “reusable inputs” to “non-reusable outputs,” and uses the 10x and 50x spreads between tiers to turn cache reuse into an explicit first-class metric. For developers, what really determines the bill isn’t the price sheet—it’s the two variables you can control: hit rate and input/output ratio.

    Long-context, fixed-prefix, batch-run agent workloads remain the beneficiaries of this pricing; sporadic calls that rewrite the prompt every time will bear an actual increase close to or above the list price. So in this round of repricing, the engineering action with the highest return on investment isn’t switching models, but: freeze the system prompt and tool definitions, move dynamic content to the tail, log tokens and cache hits, and calculate the effective unit price. Get these right, and the unit cost for the same amount of work can usually be brought down substantially.