LLM Tiered Pricing Explained: From Cache Hits to Agent Costs

Introduction: List Prices Alone Can Mislead

Recent pricing changes across major LLM services have sparked considerable debate. New-generation model output prices reached several times the previous generation, input prices rose in tandem, and only the cache-hit tier saw a relatively moderate increase. Many developers reacted with the same instinct: everything is getting more expensive, and soon it will be unaffordable.

But focusing only on the price list makes it easy to misjudge actual spending. Mainstream LLM services generally adopt tiered pricing based on input, output, and cache-hit tokens, as seen on cloud platforms including Baidu AI Cloud. These three tiers correspond to three fundamentally different resource-consumption patterns. To understand this pricing model, it helps to look at the underlying inference system.

1. The Inference Cost Structure Behind Tiered Pricing

1.1 Output: The Most Expensive Tokens

Text generation in large language models is an autoregressive process. Every generated token requires a full forward pass, and the process is inherently sequential. It cannot be parallelized the way input processing can.

During decoding, the main bottleneck is memory bandwidth. For every generated token, the system must read model weights together with the continuously growing KV Cache. As a result, the marginal compute and bandwidth cost of output tokens is the highest, which is why output tokens carry the highest price.

1.2 Input: Parallelizable Prefill

Input tokens are processed in a single pass during the prefill stage. Matrix operations can be heavily parallelized, GPU utilization is high, and the per-token cost is significantly lower than for output tokens. This is the fundamental reason why input pricing is lower than output pricing across nearly all LLM services.

1.3 Cache Hits: The Lowest Marginal Cost

When multiple requests share the same prefix, such as a system prompt, tool definitions, few-shot examples, or a long document context, the KV Cache for that segment can be stored and reused in subsequent requests. This eliminates the repeated prefill computation for the shared portion.

The industry widely uses prefix caching and disaggregated inference architectures to improve reuse rates. By decoupling cache storage from compute nodes, hit rates for long-prefix scenarios can become very high. Public technical discussions have reported cases exceeding 90%. The marginal cost of a cache-hit call is extremely low, so this tier has the lowest price.

1.4 What the Widening Price Gap Signals

Comparing price lists across two generations of models reveals a structural trend. The ratio between non-cached input and cached input widened from roughly 6x to 10x. The ratio between output and cache-hit widened from roughly 25x to 50x.

Conceptually, this is a signal that pricing is moving from coarse-grained averages toward precise mapping of resource consumption. The closer a tier is to its true marginal cost, the more differentiated its price becomes. Cache hits are the call pattern service providers most want to see, while output is the most expensive form of generation. The widening gap is effectively a more faithful projection of the cost curve onto the price list.

2. Turning List Prices into Real Costs: A Reusable Model

Looking at any single price tier in isolation is not meaningful. Real cost is determined by four variables: cache-hit rate (h), input-output token ratio (r), and the three tier prices.

Let P_miss be the non-cached input price, P_hit the cached input price, and P_out the output price. Then:

• Effective input price: P_in_eff = P_hit × h + P_miss × (1 − h)
• Blended cost per million tokens: C = (r × P_in_eff + P_out) / (r + 1)

The following example code uses hypothetical prices to demonstrate the calculation method:

def blended_cost(p_hit, p_miss, p_out, hit_rate, in_out_ratio):
    """Calculate blended cost per million tokens."""
    p_in_eff = p_hit * hit_rate + p_miss * (1 - hit_rate)
    return (in_out_ratio * p_in_eff + p_out) / (in_out_ratio + 1)
# Scenario A: Enterprise agent with highly reused prefixes
# 90% hit rate, 20:1 input-output ratio (typical long-context coding workload)
cost_a = blended_cost(p_hit=2, p_miss=20, p_out=100,
                      hit_rate=0.9, in_out_ratio=20)   # roughly 8.4
# Scenario B: Individual developer with nearly fresh context per request
# 20% hit rate, 5:1 input-output ratio
cost_b = blended_cost(p_hit=2, p_miss=20, p_out=100,
                      hit_rate=0.2, in_out_ratio=5)    # roughly 30.3
print(cost_b / cost_a)  # roughly 3.6x

With identical list prices and the same model, two types of users can face a blended unit cost that differs by more than 3x. This suggests that what is being called a broad price increase is better understood as a structural repricing:

• Enterprise customers running stable system prompts with highly reused prefixes benefit from high hit rates and high input-output ratios. The low cache-hit price is fully amplified, and they effectively receive a discounted rate.
• Light users whose request context changes every time will have nearly all input billed at the non-cached rate. Since output also dominates their total cost, they effectively pay full price or even a premium.

The pricing lever has shifted from a unified average toward differentiation by call pattern. A caller’s own usage behavior directly determines where they land on the price curve.

3. The Agent Era: Why the Cost Structure Is Changing Dramatically

3.1 Call Volume Jumps by Orders of Magnitude

In the chatbot era, a single interaction typically involved tens to thousands of tokens. Users were not highly sensitive to unit price, and total spending stayed manageable. Agent scenarios are entirely different. Published industry practices show single tasks consuming thousands of tool-call steps and hundreds of thousands of tokens. Some deployments run hundreds of agents in parallel, operating autonomously for days. Each jump in call volume raises pricing sensitivity by an order of magnitude.

3.2 Resource Consumption of Long Contexts

KV Cache memory usage grows linearly with context length. During decoding, every generated token requires reading this cache, which increases bandwidth pressure in parallel. Attention computation itself also introduces superlinear overhead in long-context scenarios. Without cache reuse and tiered storage, long-context, high-concurrency agent workloads become extremely expensive on the inference side. This is the core motivation behind sustained investment in disaggregated inference architectures and cache tiering.

3.3 Heavier Model Architectures

Current flagship models are generally trillion-parameter sparse Mixture-of-Experts architectures. Although the number of activated parameters per token is controlled, all expert weights must remain resident in memory. Expert routing adds communication overhead, and newer mechanisms such as hybrid attention raise the bar for operator optimization. These costs do not appear in a single forward pass. They show up in cluster size, GPU count, and data center operations, and they ultimately feed through to pricing.

Taken together: under agent workloads, continuing with the old pricing system could push service providers close to losses on long-context, high-reuse calls. The pricing adjustment is essentially a remapping of the new cost curve onto the price list.

4. Cost-Oriented Engineering Practices

1. Prioritize prefix stability: Keep stable content such as system prompts, tool schemas, and knowledge summaries at the front, and ensure byte-for-byte consistency. Even a single space or timestamp difference can invalidate the cache and cause the entire prefix to be billed at the non-cached rate.
2. Manage context carefully: Truncate or summarize multi-turn history to avoid accumulating useless tokens. For large document scenarios, prefer retrieval on demand rather than injecting the full text. Control input size at the source.
3. Route requests by tier: Use smaller models for simple steps and upgrade to flagship models only for complex reasoning. Use asynchronous interfaces for latency-insensitive batch tasks, which usually come with lower unit prices.
4. Establish cost baselines: Tag every agent task with cost data. Continuously monitor hit rate, input-output ratio, and per-task cost. Treat cost per completed task as a core metric alongside accuracy.
5. Evaluate on blended cost, not list price: When assessing an LLM service, plug your own traffic profile, including hit rate and input-output ratio, into the formula above. Do not compare a single number from a price list.

Conclusion

LLM pricing is moving away from the era of one average price for everything and toward a tiered model that more precisely reflects inference resource consumption. The ratios among input, output, and cache-hit pricing are effectively a public explanation of the underlying cost structure.

Teams that understand this can align their architecture with the pricing incentives. Higher prefix reuse, more disciplined context management, and sensible model routing can all make the same model significantly cheaper per unit of work. Teams that ignore the pricing structure will eventually pay a substantial premium for low-reuse call patterns as agent workloads scale.

A price list is not just a billing document. It is also a system design recommendation written by the service provider for every caller.