Deconstructing LLM API Price Hikes: Cache Hit Rate Is the Real Cost Anchor

1. Look at the Sticker Price, but Look at the Ratios: The Real Signal Is in the Three-Tier Structure

Whenever a new model launches, the fastest-spreading message in developer communities is always “output price went up by X times.” But for LLM API billing, any single quoted price is a misleading fragment, because mainstream services generally adopt a three-tier structure:

• Input (cache miss): text entering the model for the first time whose computation results cannot be reused;
• Input (cache hit): the portion of a prefix identical to a recent request, which can directly reuse already-computed KV cache;
• Output: tokens generated autoregressively one by one by the model.

Why is output inevitably several times—or even tens of times—more expensive than input? Inference has two phases: the prefill phase processes input in parallel across the whole sequence, achieving high GPU utilization and low marginal cost per token; the decode phase must generate tokens serially, waiting for each previous step’s result, with memory bandwidth dominating latency and a naturally much higher marginal cost per token. The price sheet’s “output costs far more than input” is essentially a faithful mapping of the cost curve.

To illustrate the methodology, construct a set of typical old-generation and new-generation prices (figures are purely illustrative):

Billing tier	Previous generation (CNY per million tokens)	New generation (CNY per million tokens)
Output	27	100
Input (miss)	6.5	20
Input (hit)	1.1	2

On the surface, everything went up: output roughly 3.7x, miss input roughly 3.1x, hit input roughly 1.8x—the hit tier rose most modestly, the output tier most sharply. But the real signal lies in the ratio structure: the price ratio between miss and hit widened from about 5.9x in the previous generation to 10x; the ratio of output to hit widened from about 25x to 50x. Two key ratios both doubled, meaning the vendor is deliberately widening the tier spread: heavily rewarding traffic that hits cache, heavily taxing traffic that cannot. The essence of this price adjustment is not a broad increase but a repricing based on cost structure—and it implies a technical judgment: the high cache hit rates achievable by the new generation should rise significantly.

2. The Technical Foundation: What Enables Vendors to Widen the Ratios

To understand this pricing logic, start with the basic unit: context caching.

During model inference, every input token produces corresponding Key/Value vectors, which together form the KV cache. If a new request’s prefix is exactly identical to a recent request, that prefix’s KV cache can be directly reused, skipping expensive prefill computation. The compute overhead of a hit is roughly just “retrieving from storage plus minor scheduling,” which is why vendors dare to push hit prices down to one-tenth or even one-fiftieth of miss prices while still maintaining margin.

The historical bottleneck for hit rates was architecture, not algorithms: in traditional inference architectures, KV caches are scattered across each instance’s GPU memory and expire when requests end, making cross-instance prefix reuse very difficult. In recent years, disaggregated inference architectures have gradually become mainstream practice: decoupling prefill and decode computation while sinking KV cache into a unified storage pool, scheduling by prefix matching. With this mechanism, workloads like “long system prompt + fixed tool definitions + multi-turn conversation” can push hit rates to very high levels; in coding agent scenarios, prefix share often exceeds 90%, which is the scenario assumption used in the calculations below.

Why specifically coding agents? Because this workload is naturally cache-friendly:

• The system prompt (task description, coding standards, tool JSON Schema, few-shot examples) runs thousands to tens of thousands of tokens and is highly stable;
• Each conversation turn appends full history and tool returns, but the actual incremental input may be only a few hundred tokens;
• Output is mostly “thinking + tool calls,” with the final code snippet relatively short.

Under this workload, the input-to-output token ratio easily reaches 20:1 or higher, while prefix share can exceed 90%. Widening the ratio structure in the table above is precisely tailored for this traffic: as long as the hit rate is high enough, the weight of expensive tiers is minimized. One could say widening the ratios is the vendor “pricing by cost” against its own inference architecture.

3. Turning the Price Sheet into a Calculator: Deriving the Effective Unit Price Formula

The sticker price answers “how much each tier costs,” but what you actually pay is the “blended effective unit price.” Define:

• P_miss: input miss price; P_hit: input hit price; P_out: output price;
• r: cache hit rate (proportion of input tokens that hit);
• k: input-to-output token ratio (input : output = k : 1).

Effective input unit price: P_in = P_hit × r + P_miss × (1 − r)
Blended unit price (per million tokens): C = (k × P_in + P_out) / (k + 1)

Expressing it in code is more intuitive:

def blended_cost_per_mtok(k, hit_rate, p_miss, p_hit, p_out):
    """
    k        : input-to-output token ratio, e.g. 20 means 20:1
    hit_rate : cache hit rate, 0~1
    p_miss   : input miss price (CNY per million tokens)
    p_hit    : input hit price (CNY per million tokens)
    p_out    : output price (CNY per million tokens)
    """
    p_in = p_hit * hit_rate + p_miss * (1 - hit_rate)
    return (k * p_in + p_out) / (k + 1)

Plug in two typical profiles (using the new-generation prices from the table above):

Profile A: Enterprise coding agent. Stable system prompt, multi-turn sessions, k=20, hit rate 90%:

• P_in = 2 × 0.9 + 20 × 0.1 = 3.8 CNY per million tokens
• C = (20 × 3.8 + 100) / 21 ≈ 8.4 CNY per million tokens

Profile B: Individual developer. Occasional calls, different prompt each time, k=5, hit rate 20%:

• P_in = 2 × 0.2 + 20 × 0.8 = 16.4 CNY per million tokens
• C = (5 × 16.4 + 100) / 6 ≈ 30.3 CNY per million tokens

Same price sheet, same model—effective unit prices differ by 3.6x. The sticker says “output 100 CNY,” yet the enterprise workload’s blended cost is under 10 CNY; the individual developer actually pays over 30 CNY, even exceeding the nominal input price. A cursory glance at the price sheet leads to completely opposite conclusions about “how expensive this model is.”

There is also a structural regularity here: since all three tiers adjust in sync, any workload’s “price increase multiple” must fall within the interval bounded by the hit price increase (~1.8x) and the output price increase (~3.7x). Hit rate r and input-output ratio k together determine where you land in that interval—the higher the hit rate, the closer to the lower bound; the lower the hit rate, the closer to the upper bound.

4. Who Actually Bears the Price Increase: A Layered Cross-Subsidy Perspective

Plugging the previous-generation prices into the same scenarios yields a comparison of blended unit prices across generations:

Scenario	Previous-gen blended cost	New-gen blended cost	Increase multiple
Enterprise agent (k=20, r=90%)	≈2.85	≈8.4	≈2.9x
Individual developer (k=5, r=20%)	≈9.0	≈30.3	≈3.4x

(Previous-gen calculation: enterprise profile P_in = 1.1×0.9 + 6.5×0.1 = 1.64, C = (20×1.64 + 27)/21 ≈ 2.85; individual profile P_in = 1.1×0.2 + 6.5×0.8 = 5.42, C = (5×5.42 + 27)/6 ≈ 9.0.)

Two rigorous conclusions follow:

1. In absolute terms, everyone’s costs went up. This is a real price increase; no traffic type magically “got cheaper.”
2. In relative terms, the increase is layered by caching behavior. High-hit-rate enterprise workloads rose about 2.9x, low-hit-rate individual workloads about 3.4x; the larger the output share (smaller k) and the lower the hit rate, the closer you get to the output tier’s 3.7x upper bound—absorbing the full increase.

So a more accurate description is: this is a layered repricing that shifts the profit center from “reusable traffic” to “non-reusable traffic.” For enterprise customers with stable system prompts and highly reused prefixes, the low-priced hit tier partially offsets the sticker increase, and the perceived increase is noticeably lower than the nominal one; for developers whose prompts change every time and who barely hit cache, the increase lands in full with no buffer. “How to view a multi-fold price hike” has no universal answer—first measure your own k and r, plug them into the formula, and what you get is your own real increase.

5. Developer Action Guide: Four Things You Can Do Right Now

1. Engineer prompts around prefix stability. Put all stable content at the very front of the prompt—task definition, coding standards, tool JSON Schema, few-shot examples; put dynamic content (user input, timestamps, session variables) at the very end. Never place request IDs, random seeds, or current time—fields that change every call—at the start of the prompt; a single differing character invalidates the entire prefix cache. For coding agents, tool lists must be serialized into a fixed structure and avoid key-order drift—this is the precondition for cache hits.

2. Improve cache friendliness at the task-scheduling level. Batch similar tasks within the same time window so requests share the same prefix; keep multi-turn sessions on the same context thread and avoid reordering messages mid-conversation; for RAG workloads, fix the system prompt and retrieval template, changing only the retrieved snippets.

3. Bring hit rate and effective unit price into monitoring. Mainstream APIs’ usage response fields typically distinguish cache-hit from cache-miss token counts, from which you can aggregate “effective unit price = total spend ÷ total tokens,” and build monitoring dashboards grouped by task type and prompt template. Make this metric the basis for every prompt refactor or model upgrade—if a prompt change drags the hit rate from 90% to 60%, it is a negative optimization even if answer quality is unchanged.

4. Model your scenarios before selecting a service. Don’t be scared off by a number on the sticker, and don’t be swayed by “as low as X” marketing. Derive k (input-output ratio) and r (hit rate) from your own historical logs, then plug the three-tier prices of candidate services into the formula above, converting “sticker comparison” into “effective unit price comparison.” In practice, you can pull the three-tier prices from the billing documentation of mainstream platforms such as Baidu AI Cloud, put them into the same calculator for side-by-side estimates, and combine them with the hit rate you can achieve—the conclusion often differs completely from “pick whoever has the lowest sticker price.”

6. Conclusion: Sticker Price Is Posturing; Effective Unit Price Is Truth

An API price sheet is the product of a vendor’s cost structure and customer-structure strategy. The technical narrative of three-tier billing is: the vendor uses context caching and disaggregated inference architectures to carve out a “reusable discount zone,” then uses hit-rate ratios to precisely allocate that discount to the traffic it most wants to retain. For developers, the mature posture is not to argue about “how many times prices went up” around the sticker, but to turn your own traffic profile into parameters—input-output ratio k, hit rate r, three-tier prices—plug them into the formula, and compute your own effective unit price. That number, not any multiple in a headline, is the only quantitative basis for deciding whether to keep using, migrate, or refactor your prompts.