Large Model API Price Hikes: The Real Math Behind Cache Hit Rates

1. The Facts First: The Three Tiers Didn’t Move in Sync

Consider the new-generation flagship model API pricing published by a major model vendor (in CNY per million tokens):

Billing tier	Previous generation (coding version)	New generation	Nominal increase
Output	27	100	~3.7x
Input · cache miss	6.5	20	~3.1x
Input · cache hit	1.1	2	~1.8x

Three key facts stand out from this table:

1. The increases are not synchronized. Public commentary loosely says “prices went up several times over,” but the three tiers diverge sharply: output rose the most aggressively, cache-hit rose the most gently, and the nominal gap between the two is roughly 2x.
2. The spread between tiers has been systematically widened. The previous-generation miss/hit ratio was about 6x; the new generation pushes it to 10x. The previous-generation output/hit ratio was about 25x; the new generation doubles it to 50x.
3. In one sentence: Under the new pricing structure, the act of “hitting the cache” has been assigned higher value by the pricing system—high-hit-rate traffic is protected, while low-hit-rate traffic absorbs most of the increase.

2. Compute Economics: Why a Cache Hit Deserves Its Own Low Price

To understand this asymmetric design, you need to look at the compute structure on the inference side.

1. Prefill and decode costs are inherently asymmetric. Large model inference has two phases: the prefill phase processes input in parallel with high compute utilization; the decode phase generates tokens autoregressively, limited by memory bandwidth, repeatedly reading the growing KV cache for every token produced. So “output costs several times more than input” is an industry-wide pattern. In this example, the output price is about 5x the cache-miss input price, which falls within the common range.

2. Prefill contains a large amount of repeated work. System prompts, domain knowledge, tool and function definitions, few-shot examples—these are byte-for-byte identical across a single session and even across thousands of requests, yet under traditional modes they are recomputed into KV cache over and over. Prefix caching (also called context caching) works by identifying and retaining already-computed KV cache, skipping the repeated prefill on a hit and computing only from the newly added content onward. What’s saved is real compute, so vendors naturally have room to push the hit-tier price extremely low—essentially passing the saved infrastructure cost back to users.

3. Disaggregated inference lets cache be reused at scale. Splitting prefill and decode onto heterogeneous resource pools, combined with a storage and scheduling layer centered on KV cache, allows cache to be shared across requests and across instances. According to the vendor’s public technical materials, in coding scenarios this kind of architecture can push cache hit rates above 90%—and the reason isn’t hard to see: the input for coding agents is almost entirely unchanging system prompts, tool definitions, and project context, with only a short trailing segment of new user instructions actually changing.

If the premise of “high hit rate” holds, the new pricing makes sense: the vendor places most of the increase on low-hit requests while protecting and subsidizing high-hit traffic.

3. Collapsing Three Tiers into One Number: Effective Unit Price and Blended Cost

Three separate prices aren’t intuitive for users, so they need to be collapsed. Define the following variables:

• h: input-side cache hit rate
• P_hit / P_miss: input hit / miss unit price
• P_out: output unit price
• r: input-to-output token ratio (r : 1)

This yields two core formulas:

• Effective input unit price: P_in = h × P_hit + (1 − h) × P_miss
• Blended unit cost: C = (r × P_in + P_out) / (r + 1)

Expressed as illustrative code:

def blended_cost(p_hit, p_miss, p_out, hit_rate, io_ratio):
    # Returns blended unit cost (CNY / million tokens)
    # hit_rate: input-side cache hit rate; io_ratio: input tokens / output tokens
    p_in = hit_rate * p_hit + (1 - hit_rate) * p_miss
    return (io_ratio * p_in + p_out) / (io_ratio + 1)
# Scenario A: enterprise coding agent, 90% hit rate, input:output = 20:1
print(blended_cost(2, 20, 100, 0.90, 20))   # ≈ 8.38
# Scenario B: low-frequency individual calls, 20% hit rate, input:output = 5:1
print(blended_cost(2, 20, 100, 0.20, 5))    # ≈ 30.33

Plugging in two typical scenarios (using new-generation prices P_hit=2, P_miss=20, P_out=100):

Scenario A: enterprise coding agent. In long-context agents, the input-to-output ratio generally sits around 20:1 (large amounts of project code and tool-call results as input, with only a small amount of decisions and code changes as output), and system prompts and project context are highly reused across requests. Take h=90%:

• Effective input unit price = 0.9 × 2 + 0.1 × 20 = 3.8 CNY
• Blended cost = (20 × 3.8 + 100) / 21 ≈ 8.4 CNY per million tokens

Scenario B: low-frequency calls from an individual developer. Every prompt is different, with almost no prefix reuse. Take h=20%, r=5

• Effective input unit price = 0.2 × 2 + 0.8 × 20 = 16.4 CNY
• Blended cost = (5 × 16.4 + 100) / 6 ≈ 30.3 CNY per million tokens

Same model, same price sheet, but the actual unit cost for these two user types differs by roughly 3.6x.

4. Back-Testing: The Increase Was Structurally Transferred

Now back-test the previous-generation prices (P_hit=1.1, P_miss=6.5, P_out=27) with the same formulas: Scenario A’s blended cost was about 2.85 CNY, and Scenario B’s about 9.0 CNY. That gives two sets of “actual increase” figures:

• Enterprise high-hit scenario: 2.85 → 8.4 CNY, roughly 2.9x, below every tier’s nominal increase (1.8–3.7x);
• Low-hit scenario: 9.0 → 30.3 CNY, roughly 3.4x, above the nominal input-miss increase (3.1x) and approaching the output tier (3.7x).

The conclusion is clear: nominally this is a across-the-board price hike, but in practice the increase has been structurally transferred—high-hit traffic is “looked after,” while low-hit traffic is “pressured.” Even for enterprise customers, absolute cost is still higher than the previous generation, but cache pricing significantly dilutes the nominal increase; meanwhile, low-reuse requests genuinely absorb an above-average share of the hike.

5. This Isn’t a Simple Price Hike—It’s User Segmentation

This design is structurally identical to classic pricing logic in the cloud computing industry: reserved instances versus on-demand, annual/monthly subscriptions versus spot instances—all follow the pattern of “those who commit to stable usage get lower prices, while elastic users pay a premium.” Applied to large model APIs, the carrier of “stability” is prefix reuse:

• Enterprise agents that run the same system prompt and the same set of tool definitions over the long term get a “discounted price” relative to the nominal rate;
• Individual developers who change prompts every time and call occasionally pay the “list price or even a premium.”

For the vendor, this pricing aligns price with its own cost structure: high-hit traffic means prefill compute is amortized, the same GPU can carry more decode throughput, and revenue per unit of hardware actually goes up. For users, it sends an important signal: your real cost ceiling isn’t on the price sheet—it’s in the shape of your own requests. Hit rate is a more important cost lever than unit price.

6. Engineering Practice: Treat Hit Rate as a First-Class Citizen

Since hit rate directly determines your actual unit price, it belongs on the core monitoring and optimization list alongside latency and error rate. A few actionable practices:

1. Prefix stability by design. Put all unchanging content (system prompt, domain knowledge, tool/function definitions, few-shot examples) up front and keep it byte-for-byte identical; put volatile content (user input, retrieval results, timestamps) at the tail of the prompt. Note that prefix matching is usually exact-match: a timestamp or random number that changes every time, placed at the head, can invalidate the entire cache—this is the most common “invisible waste” in practice.

2. Session affinity. Keep multi-turn conversations within the same session as much as possible to avoid frequent context rebuilds; in high-concurrency scenarios, watch whether load balancing breaks cache affinity (requests routed to nodes without the cache), and enable session stickiness or similar mechanisms when necessary.

3. Measure hit rate and manage sensitivity. Wire hit rate, effective input unit price, and blended unit cost into monitoring and alerting. Using this example’s prices for sensitivity analysis: every 10-percentage-point drop in hit rate raises the effective input unit price by about 1.8 CNY per million tokens; for a 20:1 scenario, blended cost rises by about 1.7 CNY per million tokens—roughly 20% added on top of an 8.4 CNY base out of nowhere. Hit rate isn’t a nice-to-have optimization; it’s a core metric that directly drives a fifth of your cost fluctuation.

4. Model cost using real traffic distribution. When choosing a model, don’t just look at the price sheet. Sample your own production requests first, measure the input-to-output ratio and prefix repetition, estimate the achievable hit rate, then plug everything into the formulas to compute blended cost. When evaluating large model services and inference hosting options on cloud platforms such as Baidu AI Cloud, it’s also worth including “does it provide hit-rate statistics, does it have a cache billing tier, and is the billing detail granular enough” as evaluation criteria—no measurement, no optimization.

5. Alternative strategies for low-hit workloads. Workloads that are inherently low-hit (one-off document processing, consumer apps with a fresh context every time) should either accept the premium and budget on a conservative “miss price + output price” basis, or adopt tiered routing—send simple subtasks like summarization and extraction to lightweight models, and reserve flagship model capability for the nodes that truly need it.

7. Conclusion

This round of pricing changes looks like a “across-the-board increase,” but it’s really a redesign that aligns infrastructure cost structure with commercial pricing: cache benefits made explicit, the hit/miss spread doubled, and user segmentation achieved through real cost differences. For engineers and business decision-makers, more valuable than arguing over “how many times prices went up” is mastering those two formulas—translating your workload’s request shape into an effective cost, then treating hit-rate optimization as ongoing engineering work. In large model applications where inference cost dominates, this is usually more controllable and more immediately effective than waiting for the next official price cut.