LLM API Price Hikes Explained: Cache Pricing and the Real Cost Structure

1. The Phenomenon: Prices Rose Across the Board, but Not Evenly

When the latest generation of flagship models published their prices, “price hike” became the hot topic in developer communities. Take one recent adjustment that sparked widespread discussion: output pricing reached nearly 4x the comparable tier of the previous generation, uncached input roughly 3x, while the cache-hit tier rose only modestly. Even more interesting is how the ratios between tiers shifted: the ratio of uncached input to cache-hit pricing widened from about 6x to about 10x, and output to cache-hit from about 25x to about 50x.

If you only read the “prices are up across the board” headlines, it is easy to conclude that models are getting too expensive to use. But once you split the three tiers apart, the increases turn out to be highly uneven: the cache-hit tier is the most restrained, the output tier the most aggressive. This structural difference is no pricing mistake—it faithfully reflects the cost structure of inference. To read the price list correctly, you first need to understand what each of the three tiers actually pays for.

2. The Three-Tier Ledger: The Cost Gradient Determines the Price Gradient

A single model request is decomposed, at the engineering level, into three parts with completely different cost profiles:

1. Uncached input (prefill compute). The model runs a forward pass over all input tokens to build the KV cache. Prefill is compute-intensive but highly parallelizable, so GPU utilization is high and the marginal cost is mostly amortized compute—hence a mid-range unit price.

2. Cached input (prefix reuse). If the request’s prefix exactly matches a previous request, the KV cache can be reused directly, skipping most of the prefill computation. The provider only bears the cost of cache storage and read bandwidth—far cheaper than recomputing—so the hit price can be pushed down to roughly one-tenth of the miss price. But caching is not free: high-performance memory footprint, cache lifecycle management, and eviction and replication strategies all cost real money.

3. Output (decoding). This is the most expensive tier. Autoregressive generation must proceed token by token; every generated token requires a full pass over the activated weights and the ever-growing KV cache. The bottleneck is memory bandwidth rather than compute, and it can barely be amortized through parallelism. The longer the context, the larger the KV cache that must be read and written at each decoding step, so cost grows linearly—or worse—with length.

This explains the industry-standard price gradient: cache hit < uncached input < output. And the widening ratios between generations show that vendors are tilting their pricing structure toward cacheable, reusable workloads.

There is also a positive feedback loop that is easy to miss: only when the engineering is strong enough and cache hit rates are high enough can a provider afford to set the hit price extremely low. Techniques such as disaggregated inference and cache pooling can push hit rates for long-prefix reuse scenarios above 90%—not an aggressive assumption for coding-agent workloads once the engineering is in place. The higher the cache reuse rate, the lower the unit inference cost, and the more room there is to discount. A low hit price is, in essence, engineering capability expressed as a price.

3. A Quick Bit of Arithmetic: Your Real Bill Depends on Hit Rate × Input/Output Ratio

The price list has three tiers; your bill is one-dimensional. The blended cost per million tokens depends on two workload characteristics: the cache hit rate h and the input/output ratio R:

• Effective input price = hit price × h + miss price × (1 − h)
• Blended price = (R × effective input price + 1 × output price) / (R + 1)

Here is a worked example (assume a miss price of 20, a hit price of 2, and an output price of 100—all example prices per million tokens):

def effective_input_price(hit_price, miss_price, hit_rate):
    return hit_price * hit_rate + miss_price * (1 - hit_rate)
def blended_price(hit_price, miss_price, out_price, hit_rate, io_ratio):
    eff_in = effective_input_price(hit_price, miss_price, hit_rate)
    return (io_ratio * eff_in + out_price) / (io_ratio + 1)
# Scenario A: Enterprise agent with a stable system prompt and tool definitions
# Input/output ratio 20:1, cache hit rate 90%
print(blended_price(2, 20, 100, 0.9, 20))   # ≈ 8.4 per million tokens
# Scenario B: Low-frequency hobbyist developer with a different prompt every time
# Input/output ratio 5:1, cache hit rate 20%
print(blended_price(2, 20, 100, 0.2, 5))    # ≈ 30.3 per million tokens

Both scenarios run the same model against the same price list, yet the blended cost differs by roughly 3.6x. Why? Enterprise agent workloads are naturally “input-heavy with stable prefixes”—a large system prompt, tool schemas, and long conversation history push input volume far above output, and repeated calls keep the prefix nearly identical, so a 90% hit rate pulls the effective input price down from 20 to 3.8. The low-frequency hobbyist, by contrast, captures almost no cache benefit: an effective input price of 16.4, plus the most expensive tier applied to a larger share of tokens.

So the seemingly uniform “across-the-board increase” is in effect differentiated pricing by workload profile: enterprise customers running the same system prompt with highly reusable prefixes get an implicit discount, while developers who change prompts every time and call sporadically pay close to list price—or a premium. The price list favors no one, but the workload decides who captures the savings.

4. The Cost Structure Behind the Hikes: Agent Workloads and Trillion-Scale Sparse MoE

Agent workloads have rewritten the cost model. In the chatbot era, a single interaction was tens to a few thousand tokens, users were insensitive to unit price, and total spend was small. The agent era is entirely different: in long-horizon autonomous tasks, a single task can involve thousands of tool calls and consume hundreds of thousands of tokens, and multi-agent parallelism or continuous multi-day runs push the scale higher still. Under such workloads, memory footprint, KV cache storage, and decoding-stage bandwidth consumption all grow linearly or worse with context length. At chatbot-era prices, a single long task could be a loss-making transaction. The first logic behind the price increase is a return from “conversation subsidy pricing” to pricing based on real marginal cost.

The hidden costs of trillion-scale sparse MoE. The new generation of flagship models generally uses trillion-parameter sparse MoE architectures: the total number of experts is enormous, but each forward pass activates only a tiny fraction of the parameters, keeping the active parameter count well under control. Sparse does not mean cheap, though: all expert weights must remain resident in memory—a static cost multiplied several times over relative to active parameters; expert routing adds communication and scheduling overhead; and kernel-level optimization for newer structures such as hybrid attention and MoE itself demands substantial compute and engineering investment. These costs eventually show up in the price.

5. What Developers Can Do: Convert “List Price” into “Effective Unit Price”

Faced with a price increase, agonizing over list prices is pointless; understanding what your bill is made of is what matters. Here are a few actionable strategies:

1. Design for prefix stability. Put invariant content—system prompt, tool definitions, few-shot examples—at the front of the prompt and dynamic content (user input, retrieval results) at the end, maximizing the chance of prefix reuse; within a session, keep the prefix byte-identical across turns.
2. Tiered model routing. Send simple classification, format conversion, and short summarization tasks to small or cheaper models, and reserve the flagship model for high-value steps such as planning, complex reasoning, and code generation. Mainstream agent frameworks generally support per-node model configuration.
3. Control output length. Output is the most expensive tier. Use structured outputs, explicit maximum-length limits, and concise instructions to keep the model from producing redundant content.
4. Monitor and set budget alerts. Track token consumption per application and per agent, wire up logging and alerting, and set spending caps. Cost anomalies are often a signal of degrading prompt design or runaway call loops.
5. Evaluate caching capability when choosing a provider. Treat “does it support context caching and prefix-reuse billing, how is the hit rate reported, how long does the cache live” as core selection criteria. Major model service platforms (including Baidu AI Cloud) now generally offer context caching with separate hit/miss pricing; batch inference discounts and off-peak pricing are also worth factoring into the comparison.

One-sentence summary: on a premium flagship model, what determines your bill is not the price list but how you construct your workload.

6. Closing Thoughts

This round of price increases is less “models got expensive” and more “pricing has finally aligned with the real structure of inference costs”: cache hits are cheap because reuse genuinely saves money; output is expensive because decoding genuinely costs money. For vendors, the pricing structure is a screening mechanism that directs subsidies toward reusable, high-retention workloads. For developers, understanding the cost logic behind the three tiers—and proactively designing cacheable prefixes, routing tasks sensibly, and keeping output length in check—is the right way to control costs. Cost optimization in the agent era is fine-grained engineering about the composition of your bill, not an emotional debate about list prices.