The Real Logic Behind Pay-Per-Token LLM Pricing: How Cache Hit Rate Reshapes Costs

1. What the Price Table Actually Tells You

Let’s start with a comparison of pay-per-token pricing tiers (using a representative generational model as a reference, in USD per million tokens):

Pricing Tier	Previous Optimized Version	New Generation	Multiplier
Output	27	100	≈3.7x
Input (cache miss)	6.5	20	≈3.1x
Input (cache hit)	1.1	2	≈1.8x

At first glance, the obvious takeaway is “everything went up.” But what deserves closer attention isn’t the absolute multipliers — it’s how the ratios between tiers have shifted:

• Cache miss input / cache hit input: widened from roughly 6x to 10x
• Output / cache hit input: widened from roughly 25x to 50x

In other words, the cache-hit tier saw the mildest increase of all tiers, while output pricing rose most aggressively. This structural change isn’t arbitrary. It reflects a concrete technical reality: in large-scale inference systems, cache reuse is becoming the core variable that determines unit cost.

2. Why Cache-Hit Pricing Can Stay Low

To understand these ratios, you first need to understand how modern inference architectures handle repeated prefixes.

In LLM inference, a user request typically consists of three parts: the system prompt, conversation history or context, and the current turn’s new input. For coding agents in particular, a single session often contains large amounts of identical system prompts and heavily overlapping code context — every tool call and every self-correction round sends this content back through the model.

If attention computation had to be redone from scratch every time, costs would grow linearly or even superlinearly with context length. The industry’s standard approach is to split already-computed KV caches by prefix and persist them, then read directly from cache when the same prefix appears again, skipping redundant prefill computation.

There’s an often-overlooked engineering constraint here: a cache hit depends on the prefix being identical token by token. Change one punctuation mark in the system prompt, insert a timestamp, or reorder the tool descriptions, and the entire suffix becomes invalid. This is why a high cache hit rate isn’t a model capability — it’s the result of architecture design combined with caller-side engineering practices.

Some cloud providers use disaggregated inference architectures that split prefill and decode into separate resource pools, with a dedicated storage layer for KV caches. This allows long-context cache reuse rates to exceed 90% in specific scenarios. That number has a realistic basis in high-reuse scenarios like coding agents, but it’s scenario-dependent and shouldn’t be extrapolated to all workloads.

3. Turning List Prices into Real Costs: Two Scenarios Compared

List prices apply to individual token tiers, but real costs must be weighted by actual usage mix. Here are two typical scenarios.

Scenario A: Enterprise Coding Agent

• Input-to-output ratio: around 20:1 (common for long-context, multi-turn tool calls)
• Cache hit rate: 90%

Effective input price = 2 × 0.9 + 20 × 0.1 = 3.8 per million tokens

Blended cost per million tokens (at 20 parts input to 1 part output):

(20 × 3.8 + 1 × 100) / 21 ≈ 8.4

Scenario B: Low-Frequency Individual Developer

• Input-to-output ratio: around 5:1
• Cache hit rate: 20% (every prompt is different)

Effective input price = 2 × 0.2 + 20 × 0.8 = 16.4 per million tokens

Blended cost per million tokens:

(5 × 16.4 + 1 × 100) / 6 ≈ 30.3

Same model. Scenario A’s unit cost is roughly 1/3.6 that of Scenario B.

This gap explains the seemingly contradictory experience of “list prices went up 4x, but my bill barely moved.” It doesn’t come from accidental differences in billing methodology — it comes from structural differences in how the model is called.

4. Cache-Friendliness Is an Engineering Capability

If hit rate determines cost, then it’s not a matter of luck — it can be designed. The following practices have proven effective in real-world use:

1. Fix the prefix order of your system prompt

Put the most stable content first: role definition → tool list → output specification. Move variable content (current time, user ID, temporary context) to the end. Once the prefix is stable, cache reuse has a foundation.

2. Avoid inserting meaningless dynamic content at the front

A common anti-pattern: prepending Current time: 2025-XX-XX XXXX to the start of a system prompt. That single line invalidates the cache for everything after it. The right approach is to put such information in the user message or inject it as a separate trailing section.

3. Reuse context within a session instead of rebuilding it each turn

In multi-turn agent calls, pass the full conversation history back rather than reconstructing it from scratch every round. As long as the prefix remains continuous, subsequent content stays cacheable.

4. Use templated tool descriptions instead of free-form assembly

When tool schemas are generated dynamically by code each time, key ordering, indentation, and default values for optional fields can all vary, causing prefix mismatches. A fixed serialization approach (such as stable field ordering) can significantly improve hit rates.

Here’s a simplified code comparison showing cache-friendly vs. cache-unfriendly construction:

# Cache-unfriendly: dynamic content at the front, prefix changes every time
def build_prompt_bad(user_input, now):
    return f"Current time: {now}\nYou are a coding assistant...\n{user_input}"
# Cache-friendly: stable prefix + dynamic content at the end
STABLE_PREFIX = """You are a coding assistant.
Available tools: [read_file, write_file, run_tests]
Output format: explain your approach first, then provide the code."""
def build_prompt_good(user_input, now):
    return [
        {"role": "system", "content": STABLE_PREFIX},
        {"role": "user", "content": f"(Reference time: {now})\n{user_input}"},
    ]

The first approach creates a completely new prefix on every request, driving the hit rate toward zero. The second approach keeps the first segment reusable as long as STABLE_PREFIX stays unchanged.

5. Why Output Is the More Important Optimization Target

Recall the ratio from the beginning: output costs 50x more than cached input. This means saving one output token equals saving fifty cached input tokens. The priority for cost optimization is therefore clear:

1. Control generation length. Set a reasonable max_tokens, explicitly request concise output in your prompt, and avoid letting the model produce lengthy explanatory text.
2. Use structured output instead of natural language. JSON or field-based results consume fewer tokens than prose and are easier to parse.
3. Push deterministic reasoning downstream. For high-certainty subtasks (code formatting, type checking), don’t route every instance through model generation.
4. Reduce ineffective retries. Failed retries often repeat the full input and output, which is the most direct waste of output quota.

6. Practical Guidance for Model Selection and Budgeting

For technical decision-makers, evaluating whether a model is “expensive” shouldn’t rely on per-token list prices alone. Follow this sequence:

• Step 1: Measure your hit rate. Run a round with real production traffic and count the actual cache hit ratio instead of assuming an optimistic number.
• Step 2: Calculate your input-to-output ratio. Long-context agents often run well above 10:1, which significantly amplifies the impact of hit rate.
• Step 3: Convert to blended unit cost. Use the formula: (input volume × effective input price + output volume × output price) / total tokens.
• Step 4: Evaluate migration cost. If your current calling patterns have low cache-friendliness, fix the prompt structure first before deciding whether to switch models.

One important caveat: the benefits of a high hit rate come with preconditions. They require your workload to have stable prefix reuse patterns. For scenarios where every prompt is completely different (open-ended creative generation, one-off Q&A), hit rates are naturally low and actual costs will be closer to list prices. Treating an architecture-driven discount as a universal bonus is one of the easiest mistakes to make in budget planning.

Conclusion

The pricing adjustment in new-generation models is, on the surface, a shift toward higher numbers. In substance, it’s a migration of the billing dimension from “token count” to “effective compute”: reused prefixes are cheap, content that needs recomputation is expensive, and generation is the most expensive of all.

Understanding this structure is more valuable than memorizing any specific set of numbers. It means the center of gravity in cost control is shifting from “calling less” to “calling smarter” — stable prefixes, continuous context, compressed output, and fewer retries. For teams running long-lived agent workloads, the cost difference produced by these engineering habits can far exceed what you’d gain from comparing prices across a handful of models.