LLM Pricing Explained: How Cache Hits Determine Your Real Agent Costs

    When a model vendor announces new pricing, most people fixate on the headline that "output prices rose several-fold" and miss the fact that billing is split into three independent tiers: uncached input, cached input, and output. These tiers can differ by an order of magnitude, so your actual bill is driven less by the sticker price and more by how your requests distribute across them. This article explains why cached input is an order of magnitude cheaper, walks through an "effective unit price" model (with reusable Python code) for estimating real costs under two typical scenarios, describes why the agent era fundamentally reshapes the cost structure, and lists engineering practices—stable prefixes, putting volatile content last, controlling context growth, cache-aware routing, and observability—that raise your effective cache hit rate and lower your effective unit price.

    What a “Price Hike” Actually Changes

    When a model vendor announces pricing for a new generation, most people walk away with one conclusion—“output prices went up several times”—while overlooking the fact that billing actually splits into three independent tiers:

    • Uncached input price: The part of a user’s prompt that cannot reuse prior computation and must undergo a full prefill pass from scratch.
    • Cached input price: Tokens in the prompt whose prefix matches a previous request exactly, so the previously computed intermediate state (KV Cache) can be reused directly.
    • Output price: The tokens the model generates one at a time (decoding).

    The pricing for these three tiers often differs by an order of magnitude. A useful rule of thumb: the output unit price is typically several times the input unit price, and uncached input is in turn several times cached input. For illustration (the numbers below only demonstrate the billing logic and are not any real product’s pricing): suppose a generation prices output at $100 per million tokens, uncached input at $20, and cached input at $2—then the ratio of output to cached input reaches 50×, while the ratio of uncached to cached input is 10×.

    The key takeaway: “Prices went up” is a vague impression. What actually determines your bill is not the sticker price, but how your requests distribute across these three tiers.

    Cache Hits: The Real Cost Divide

    Why can cached input be an order of magnitude cheaper? The answer lies in the two phases of inference.

    • Prefill phase: The model reads the entire input sequence at once, computes the Key/Value tensors for every token, and writes them to memory. This compute cost grows with context length.
    • Decode phase: The model generates new tokens one by one, based on the existing KV Cache.

    If two requests share an identical prefix (for example, the same system prompt, the same few-shot examples, or the same long document context), the KV Cache computed by the first request can be reused directly by the second, eliminating redundant prefill work. The disaggregated inference architectures common in the industry—which schedule prefill and decode onto separate resource pools—exist precisely to manage and reuse this cache more efficiently, pushing hit rates very high in scenarios with high prefix reuse.

    This leads to the central claim of this article: In long-context scenarios with a fixed system prompt, the cache hit rate—not the sticker price—is the first variable that determines your true cost.

    The “Effective Unit Price” Model

    Rather than staring at sticker prices, compute an “effective unit price” that reflects real usage. Here is a reusable piece of calculation logic:

    1. def effective_cost_per_mtok(
    2. price_input_miss: float, # Uncached input price, per million tokens
    3. price_input_hit: float, # Cached input price
    4. price_output: float, # Output price
    5. io_ratio: float, # Input-to-output token ratio, e.g. 20 means 20:1
    6. cache_hit_rate: float, # Cache hit rate, in the range 0-1
    7. ) -> float:
    8. # Effective input price = hit_rate*hit_price + (1-hit_rate)*miss_price
    9. eff_input = (
    10. cache_hit_rate * price_input_hit
    11. + (1 - cache_hit_rate) * price_input_miss
    12. )
    13. # Treat "1 output unit + io_ratio input units" as one billing block
    14. # and take the weighted average.
    15. return (io_ratio * eff_input + price_output) / (io_ratio + 1)

    Using the same illustrative pricing above (output 100, uncached 20, cached 2), plug in two typical scenarios:

    Scenario A: Enterprise-grade long-context agent

    • Input-output ratio around 20:1, cache hit rate 90%
    • Effective input price ≈ 2×0.9 + 20×0.1 = 3.8
    • Blended price ≈ (20×3.8 + 100) / 21 ≈ $8.4 per million tokens

    Scenario B: A casual developer who calls occasionally with a different prompt every time

    • Input-output ratio around 5:1, cache hit rate only 20%
    • Effective input price ≈ 2×0.2 + 20×0.8 = 16.4
    • Blended price ≈ (5×16.4 + 100) / 6 ≈ $30.3 per million tokens

    With the same sticker price, the two scenarios differ in unit cost by roughly 3.6×. This is why people say that “prices rise across the board” while, in practice, high-reuse long-term customers effectively get a discount and casual developers who switch prompts every time pay the list price or even a premium.

    Why the Agent Era Rewrites the Cost Structure

    In the chat era, a single interaction is often only a few dozen to a few thousand tokens, and users are not sensitive to unit price. But an agent’s load profile is completely different:

    1. Very long context: Agents must carry tool definitions, prior trajectories, and retrieved documents—easily tens of thousands or even hundreds of thousands of tokens per request.
    2. Multi-step tool calls: A complex task may involve thousands of reasoning steps and tool round-trips, with token consumption accumulating several-fold.
    3. Long-running parallel execution: Hundreds of agents running autonomously in parallel for days—cache hit rate and context management directly determine the total bill.
    4. Underlying resources grow non-linearly with context: The memory footprint of the KV Cache, storage overhead, and the memory-bandwidth consumption during decoding all rise linearly or even super-linearly with context length.

    Under this kind of load, if the inference side cannot reuse the cache effectively or manage the KV Cache carefully, running agent tasks on a “chat-era” pricing model can easily send costs out of control. Splitting the billing into “uncached/cached” tiers is essentially passing the difference in underlying resource cost through to the user, forcing them to optimize how they make calls.

    Engineering Practices to Cut Real Costs

    Once you understand the billing model, you can target it directly. Each of the following strategies raises the effective cache hit rate and lowers the effective unit price:

    • Fix and front-load the system prompt: Put the stable, unchanging role definition, tool definitions, and output specifications at the very front of the prompt, and keep them byte-for-byte identical. Caching typically matches on the longest common prefix, so any variation in the prefix will degrade all subsequent content to “uncached.”
    • Put volatile content last: Keep frequently changing information—timestamps, random IDs, real-time user input—toward the end of the prompt so it does not pollute the prefix.
    • Control context growth: Apply sliding windows and summary compression to long conversations, and rerank and trim retrieved results to keep irrelevant content out of the prefix.
    • Cache-aware routing and scheduling: In a multi-instance deployment, route requests that share the same prefix to the same instance as much as possible to improve local cache reuse.
    • Build in observability: Using common cloud capabilities such as log services, monitoring and alerting, and object storage, measure the input-output ratio, hit rate, and token consumption per business dimension, and keep re-running the effective unit price model above to pinpoint where the cost goes.

    Conclusion

    To judge a model’s true cost, don’t look only at how much it lists—look at which billing tier your requests fall into. The cache hit rate is the lever on input-side cost, while the input-output ratio and context length determine how much that lever can move. For enterprise-grade agents, a stable prefix and careful context management are not optional optimizations—they are the core means of turning a “high price” into a “discounted price.” For casual developers, understanding this avoids paying bills far higher than expected on long-context tasks. As agent workloads become mainstream, whoever masters the engineering of cache and context holds the upper hand in inference cost.