LLM API Pricing Decoded: The Cost Logic Behind Three-Tier Token Prices

1. Background: A Representative Pricing Adjustment

An LLM vendor recently released a new model generation and adjusted its API pricing at the same time. Placing the two generations side by side makes the structural character of this adjustment clear:

Billing tier	Previous generation (CNY per million tokens)	New generation (CNY per million tokens)	Increase
Output	27	100	~3.7x
Input (cache miss)	6.5	20	~3x
Input (cache hit)	1.1	2	~1.8x

If you only look at the surface story—“prices went up across the board”—the conclusion is too coarse. What deserves attention is the uneven distribution of the increases across the three tiers: the cache-hit tier rose the most gently, while the output tier rose the steepest. That gradient is itself a key to understanding the cost structure of LLM inference.

2. Why LLM APIs Use Three-Tier Billing

To read this price table, you first need to understand what cost each tier maps to:

1. Input (cache miss): The first time the model processes a context, it must run forward computation over every input token and generate the KV Cache. This is a one-time cost.
2. Input (cache hit): If the input prefix of a request exactly matches an earlier one (for example, the same system prompt or the same document context), the inference service can reuse the already-computed KV Cache and skip most of the forward computation. This is why the unit price sits far below the miss tier.
3. Output: Autoregressive generation proceeds one token at a time. Every token requires a full decoding pass, the process cannot be parallelized, and it is constrained by memory bandwidth. Output also continuously occupies concurrency resources, raising its time cost—so this tier is always the most expensive of the three.

Output priced above input, and cache hits priced as a deep discount on input, has become the standard pricing paradigm across the industry. Major Chinese LLM service platforms, including Baidu AI Cloud’s model services, also commonly adopt similar tiered billing.

3. Ratio Shifts: More Informative Than Absolute Prices

Focusing on a single tier’s increase loses the picture. Pulling out the ratios between tiers tells you much more.

1. Miss/hit ratio: widened from ~6x to 10x

In the previous generation, uncached input cost about 6x the cached price; in the new generation it is 10x. The value of caching has been amplified: for the same one million tokens of input, the gap between a miss and a hit grew from 6x to 10x. The vendor is using price leverage to push users toward optimizing prompt structure and raising cache hit rates.

2. Output/hit ratio: widened from ~25x to 50x

Output rose the most and cache hits the least, so the ratio between them widened to 50x. This indicates that output-side compute is the biggest pressure point in the vendor’s cost structure. It also suggests the new model is positioned for output-heavy workloads—typically Agent tasks and code generation, which produce many output tokens, run long task chains, and demand the highest model capability. Charging a premium on output is, in essence, pricing the model’s capability premium.

3. The increase gradient: hit < miss < output

Cache hit ~1.8x, input miss ~3x, output ~3.7x. The increases grow along the direction of “closer to raw computation, less reusable”—which matches the real composition of inference cost.

4. The Impact Differs Sharply by Application Type

The same price table feels very different depending on the shape of your application:

• Short input, short output (customer-service Q&A, lightweight chat): input and output are both short, cache benefits are limited, and cost rises roughly linearly, around 3x.
• Long input, short output (RAG, document summarization): input dominates. If the knowledge-base prefix is fixed, cache hit rates can be made very high, so the actual cost increase may fall far below the listed increase.
• Long input, long output (Agents, coding assistants): both ends are expensive, and multi-turn tool calls cause input to accumulate repeatedly. These applications are the most sensitive to output pricing and are hit hardest by this restructured price table. Conversely, if the new model significantly reduces retry rounds and improves first-pass success, total cost may not rise at all.

5. Cost Modeling: Turning List Prices into an Executable Budget

When you receive a new price sheet, the first move is not to panic—it is to model. A minimal cost-estimation function:

def estimate_cost(input_m, cached_m, output_m,
                  p_miss, p_hit, p_out):
    # Units: millions of tokens; prices in CNY per million tokens
    return input_m * p_miss + cached_m * p_hit + output_m * p_out
# Example: daily token consumption of an Agent application
daily = estimate_cost(input_m=30, cached_m=120, output_m=10,
                      p_miss=20, p_hit=2, p_out=100)
print(f"Estimated daily cost: {daily:.0f} CNY")  # 30*20 + 120*2 + 10*100 = 1840

Note that in this example the cached input volume is 4x the uncached volume—only achievable when the prompt structure has been cache-optimized. Every 10-percentage-point gain in cache hit rate effectively moves that share of input from the high-price tier to the low-price tier.

6. Cache Optimization: The Direction the Price Table Makes Explicit

When the miss/hit ratio reaches 10x, cache optimization stops being a nice-to-have and becomes mandatory. Several proven practices:

1. Put stable prefixes first: place fixed content—system prompts, tool definitions, knowledge-base snippets—at the very front of the prompt, and volatile content—user input, timestamps—at the end. Caching works by prefix matching; a single differing byte in the prefix invalidates everything after it.
2. Avoid prefix jitter: never prepend random IDs, current timestamps, or other volatile fields to the start of a prompt; push information that must vary as far back as possible.
3. Reuse within a session: multi-turn conversations naturally share the historical prefix. Keep consecutive reasoning within the same session instead of rebuilding the full context every turn.
4. Share context across batch tasks: when batch-processing the same set of documents, use the shared documents as a common prefix and vary only the task instructions at the end.

Recommended prompt structure:
[System prompt (stable)] + [Tool/knowledge-base definitions (stable)] + [Examples (stable)] + [User input (volatile)]

7. A Framework for Responding to Price Increases

1. Calculate ROI before debating price: the point of a model upgrade is quality. If the new model raises first-pass success from 70% to 90%, the token savings from fewer retries may offset or even exceed the unit-price increase. Evaluate on “cost per unit of effective output,” not on unit price.
2. Tiered routing: send simple tasks (formatting, rewriting, classification) to low-cost models and reserve complex tasks (long-chain reasoning, code generation) for high-performance models, routing by task type at the gateway layer—a common cost-control architecture.
3. Set budget guardrails: meter token usage at the application layer, enforce per-session and per-tenant quotas, and alert on anomalous usage. Combining cloud monitoring and alerting, logging services, and API gateway capabilities helps prevent a runaway prompt (such as a looping call) from blowing up the bill.
4. Watch hidden costs: billing rules for retries, timeouts, and failed requests, and the handling of interrupted streaming output—details that are easy to overlook when estimating.

8. Conclusion

A price sheet is a mirror of a vendor’s cost structure and product strategy: the gentlest increase on cache hits and the steepest on output sends a clear signal—encourage reuse, pay for real compute. For developers, rather than agonizing over how much the unit price rose, put your energy into three things: build an accurate usage model, push your cache hit rate up, and use tiered routing to match tasks to model capability. Pricing will keep changing, but this analytical framework is reusable.