LLM Inference Pricing Tiers: The Engineering Logic Behind Cache, Input, and Output Costs—and What They Mean for Model Selection

1. The Surface: Three Independent Pricing Dimensions

LLM inference services typically break billing into three separate tiers: input tokens that miss the cache, input tokens that hit the cache, and generated output tokens. Because each tier reflects a different resource-consumption pattern, price adjustments rarely move all three in lockstep.

Take a coding-focused LLM as an example. The old version charged about 27 CNY per million output tokens; the new version raises this to 100 CNY—on the surface, nearly a 4x increase in output cost. Cache-miss input pricing climbs from 6.5 CNY to 20 CNY per million tokens, roughly 3x, while cache-hit input pricing rises from 1.1 CNY to 2 CNY per million tokens, an increase of less than 2x.

If you only ask “how much did prices go up overall,” it is easy to reach vague or outright wrong conclusions. The more sound approach is to split the three billing tiers apart, factor in your own application’s request profile, and recompute the cost per task.

2. What a Widening Cache-Hit Ratio Signals

In the old version, the ratio of cache-miss to cache-hit input pricing was about 6x; the new version widens it to 10x. The ratio of output pricing to cache-hit input pricing grows from roughly 25x to 50x.

This change sends an unambiguous signal: the provider is using price structure to steer users toward fuller use of context caching.

Cache-hit input means the request prefix has already been computed and stored on the server, so inference does not need to re-run attention over the prefix. That is why cached input costs far less than uncached input. Widening the ratio from 6x to 10x further raises the relative cost of a cache miss. For requests that repeatedly send the same long prefix, cache hit rate directly determines the range of your actual cost swings.

The steepest increase on the output side also has an engineering explanation. Output generation is token-by-token serial decoding and cannot be parallelized at scale the way input prefill can. As parameter counts grow, inference chains lengthen, and decoding steps multiply, compute and memory pressure on the output side typically grows faster than on the input side. So output unit prices rising faster than the cache-miss input tier is not purely a market tactic—it mirrors a shift in the underlying cost structure.

3. Why Output Prices Rise the Most

The consumption pattern of output tokens makes them the most price-sensitive tier.

First, output is a serial process. The input prefill phase can run highly parallel matrix operations over the entire sequence, but during output generation each token depends on the state of the previous one, so full parallelism is out of reach. Even with optimizations such as speculative decoding and batch scheduling, the ceiling on hardware utilization on the output side remains below the input side.

Second, output length tends to grow as models become more capable. In code generation, a newer model may produce more complete files, longer explanations, and more complex implementations, so output tokens per request naturally climb. The cost growth developers perceive usually comes from “higher unit price” and “longer outputs” compounding each other.

Third, quality gains often come with changes to decoding strategy. To produce more stable reasoning, some models adopt more conservative sampling parameters, more candidate-sequence verification, or longer reasoning chains. These capability improvements also translate into output-side compute overhead.

So when you see output pricing move from 27 CNY to 100 CNY, do not stop at a simple multiple comparison. Re-evaluate instead: does the same task produce far more output tokens on the new model? Do task completion rate and first-pass success improve? If the model can complete equivalent work in fewer interaction rounds, the real cost may be far below the headline increase.

4. Cache Hit Rate: The Decisive Cost Variable

For applications with strong prefix-reuse patterns, cache hit rate is a more critical cost variable than unit price.

Consider a typical enterprise coding-assistant scenario: the developer keeps project context in the session throughout—file trees, existing code snippets, constraint notes, and so on. This prefix content is resent on every conversational turn. If caching works, later turns bill the prefix input at the cache-hit rate; if the cache misses, it is recomputed at the uncached rate.

Assume a session carries a 100,000-token prefix and each turn outputs 5,000 tokens. With a 90% cache hit rate, the blended input cost is roughly:

Input cost = 100000 × 10% × 20 + 100000 × 90% × 2
           = 200000 + 180000
           = 380000  (to be converted separately using the per-million-token pricing unit)

With only a 50% hit rate, the blended input cost becomes:

Input cost = 100000 × 50% × 20 + 100000 × 50% × 2
           = 1000000 + 100000
           = 1100000

Raising the cache hit rate from 50% to 90% cuts input-side cost by roughly two-thirds. Compared with simply comparing unit prices, hit rate has a far more direct impact on the actual bill.

This math also explains why providers widen the spread between cache-hit and cache-miss pricing. The larger the spread, the greater the payoff from hit-rate optimization—and the stronger the incentive for users to design cache-friendly request structures.

5. Cost Impact Varies by Workload Type

Not every application is affected to the same degree. By request shape, workloads fall roughly into three categories.

The first is long-session, heavy-reuse workloads: multi-round changes against the same code repository, or Q&A over the same set of documents. These have high prefix-reuse rates and a large share of cache-hit input. Because cache-hit pricing rises only modestly, the real overall cost increase is usually lower than the headline output-price increase.

The second is short-request, low-reuse workloads: every request is an independent small code generation or a standalone question. These have almost no prefix reuse, so input is billed mostly at the cache-miss rate, and output may account for a large share. These applications take the biggest hit from the price change and need to re-examine both model choice and calling patterns.

The third is output-intensive workloads: long-form generation, bulk test-case generation, and automated documentation. Output tokens far outnumber input tokens, so a higher output unit price directly amplifies total cost. These workloads call for focused output-length optimization or stricter generation limits.

6. Developer Responses Should Center on Structure, Not Emotion

When a pricing structure changes, developers should not stop at the simple judgment of “more expensive or cheaper.” Adjust on three levels: request structure, caching strategy, and model selection.

First, quantify your request profile. Collect a stretch of real traffic data and measure average input length, average output length, prefix repetition rate, and cache hit rate. Only with these numbers can you accurately assess how the new pricing affects your business.

Second, raise your cache hit rate. When designing requests, put stable content at the front of the prefix and variable content at the end. Avoid inserting frequently changing content early in the prefix, which causes repeated cache invalidation. In multi-turn sessions, keep the order of the system prompt, project context, and tool definitions stable.

Third, control output length. Without hurting task quality, tune prompt constraints to request concise answers, and set a sensible maximum output token count to prevent verbose generations. For code generation, prompt the model to emit only the necessary fragments rather than whole files.

Fourth, tier models by task difficulty. Not every task needs the strongest model. Lightweight work such as simple completions, format conversion, and naming suggestions can run on lower-cost models; call the high-capability model for complex architecture design, cross-file refactoring, and hard bug localization. This meaningfully dilutes blended cost.

7. Pricing Structure Is Also a Signal of Technical Direction

When an LLM provider raises output prices and widens the cache-hit ratio, the move itself reflects underlying technical trends.

On one side, the compute overhead of stronger models concentrates on the output side. That tells you output decoding efficiency is now a key bottleneck in inference cost. For some time to come, output-side optimization will remain an engineering priority: more efficient decoding algorithms, finer-grained batch scheduling, and hardware architectures better suited to long outputs.

On the other side, the mild adjustment to cache-hit pricing shows the provider still wants to encourage long-context reuse. Context caching is not just a billing mechanism—it is an important engineering lever for cutting overall inference cost and improving response speed. Expect future inference optimizations to further reinforce “cache-friendly” patterns, such as longer cache lifetimes and finer-grained prefix matching.

For developers, understanding the technical constraints behind the price structure is worth more than memorizing any specific number. Prices will keep moving with infrastructure, model architecture, and market competition, but the basic logic—“cached input matters above all” and “the output side sets the ceiling”—will hold for a long time.

Conclusion

An LLM inference price change should not be boiled down to a single number going up or down. Input, output, and cache-hit pricing are three independent yet interlocking dimensions that together determine real usage cost. Facing higher output prices and a wider cache price spread, the most valuable move for developers is to return to their own workloads: base decisions on request data, optimize cache hit rate and output length, and find a finer-grained match between task complexity and model capability. Only then can you keep active control of both cost and quality as model pricing continues to evolve.