LLM Tiered Pricing Explained: How Cache Hit Rate and Input/Output Ratios Determine Real Costs

1. The Surface vs. the Substance of Price Changes

A major cloud provider recently launched a new generation of its large language model with noticeably higher API prices compared to the previous version. On paper, the numbers look stark: output pricing rose from roughly ¥27 per million tokens to ¥100—nearly a 4× increase—while input pricing jumped from ¥6.5 (cache miss) and ¥1.1 (cache hit) to ¥20 and ¥2, respectively, a roughly 3× increase.

A linear comparison would suggest a simple conclusion: everything got more expensive, and it’s unaffordable. But this view overlooks a critical variable—the actual weight of each pricing tier varies enormously across different usage patterns. To understand real cost changes, you have to put the price list back into concrete usage scenarios.

Another structural shift deserves attention. The price ratio between cache-miss and cache-hit input widened from about 6× to 10×, while the output-to-cache-hit ratio stretched from roughly 25× to 50×. These ratio changes are themselves a signal: they steer users toward maximizing cache hit rates and minimizing output token consumption.

2. Cache Hit Rate: The Overlooked Cost Lever

Understanding this pricing strategy starts with understanding inference-side caching. In mainstream LLM inference architectures, caching typically refers to reusing key-value states for context prefixes that have already been processed. When multiple requests share the same or highly similar system prompt, few-shot examples, or long document prefixes, the inference engine can skip redundant computation and reuse cached states directly.

One common technical approach in the industry uses a disaggregated inference architecture that decouples the prefill and decoding stages and caches prefill results in a distributed fashion. In long-context, high-reuse scenarios, this can push cache hit rates above 90%. The core idea is straightforward: turn expensive prefill computation into a one-time, reusable asset, so subsequent requests only need incremental processing.

This fundamentally reshapes the cost structure. For agent-style applications that frequently reuse the same system prompt, the vast majority of input tokens qualify for the cache-hit tier, pulling the effective input unit price down significantly.

3. Two-Scenario Cost Modeling: Same Model, 3.6× Cost Difference

To make the practical impact of tiered pricing concrete, let’s model two typical scenarios.

Scenario A: Enterprise Long-Context Agent

Assume an enterprise agent scenario with these characteristics:

• Input/output ratio of roughly 20:1 (common in long-context coding assistance, code review, document Q&A)
• Cache hit rate of 90%
• Per million-token batch: ~20M input tokens, ~1M output tokens

Effective weighted input price:

2 × 0.9 + 20 × 0.1 = 3.8 per million tokens

(¥2 is the cache-hit price; ¥20 is the cache-miss price.)

Output price: ¥100 per million tokens.

Blended cost per million tokens:

(20 × 3.8 + 1 × 100) / 21 ≈ 8.4

Scenario B: Infrequent Individual Developer

Now consider a different scenario:

• Input/output ratio of roughly 5:1 (short conversations, one-off Q&A)
• Cache hit rate of only 20% (each prompt differs significantly, little prefix reuse)

Effective weighted input price:

2 × 0.2 + 20 × 0.8 = 16.4 per million tokens

Output price remains ¥100 per million tokens.

Blended cost per million tokens:

(5 × 16.4 + 1 × 100) / 6 ≈ 30.3

The unit cost between these two scenarios differs by about 3.6×. Same model, same price list, wildly different actual bills.

4. The Commercial Logic Behind Tiered Pricing

This isn’t a simple “price hike.” It’s a user behavior steering mechanism. The core logic:

1. Reward high-reuse scenarios: Cache-hit tier prices rose relatively modestly. For long-running, stable-system-prompt, high-reuse enterprise applications, the actual cost increase is far smaller than the headline numbers suggest.
2. Discourage low-reuse scenarios: Cache-miss input and output tiers saw steep hikes, putting significant cost pressure on ad-hoc calls where every prompt is different and output volume is high.
3. Drive architectural optimization: The shifting price ratios send a clear message to developers—reduce redundant inputs, maximize cache reuse, and control output length are the three key levers for cost reduction.

From a business perspective, this helps prioritize limited inference capacity for high-value, high-reuse enterprise workloads while using price signals to filter out inefficient calls.

5. Developer Practices for Cost Optimization

Based on this analysis, developers can optimize LLM call costs in several directions:

1. Maximize Prefix Reuse

Place stable system prompts, role definitions, example dialogues, and knowledge-base prefixes at the front of requests so different calls share the longest possible common prefix. Many inference platforms match prefixes token by token starting from the first token—the more stable the prefix, the higher the hit rate.

# Poor practice: changing the system prompt on every request
messages = [
    {"role": "system", "content": f"You are an assistant. Current time is {now()}, user ID is {user_id}..."},
    ...
]
# Better practice: fixed prefix, dynamic info in later turns
SYSTEM_PROMPT = "You are a professional coding assistant. Follow these guidelines..."
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"Current time {now()}, please answer..."},
    ...
]

2. Control Output Token Consumption

The output tier is the most expensive component. Where business requirements allow, limit max output length, craft prompts that guide concise responses, and use structured output formats to reduce output-side costs.

3. Design Input/Output Ratios Wisely

For input-heavy scenarios like document Q&A or code review, improving cache hit rate has the biggest impact on total cost. For output-heavy scenarios like content generation, focus on output efficiency.

4. Monitor and Allocate

Set up cost monitoring by business line and call type. Identify high-consumption, low-yield patterns and make targeted architectural adjustments.

6. Conclusion

LLM pricing is shifting from flat-rate to tiered models—a reflection of both compute cost pressure and a push for more efficient usage. The headline price increases across the board mask a nuanced reality: for high-reuse, long-context, cache-friendly enterprise workloads, actual cost growth is limited; for individual developers making one-off calls with unique prompts every time, the cost increase is real.

Understanding the ratio relationships in the price table and modeling actual costs based on your own input/output ratios and cache hit rates is essential for sound technical decision-making. The bottom line is simple: cache hit rate and input/output ratio are the two most critical variables that determine your real LLM usage cost.