Why LLM Inference Costs Rise Even as Token Prices Fall

Over the past six months, one major LLM vendor has publicly apologized three times for service-quality issues—an unusual pattern in itself. What is more telling is that these apologies often coincided with the launch of new model versions: significant gains in code generation, long-horizon task orchestration, and 3D modeling, with third-party evaluations ranking the new models’ intelligence index among the global leaders and a positive market response. Yet alongside the capability gains came widespread anxiety among subscribers about quota consumption: unit prices clearly dropped, but completing the same task did not produce a cheaper bill.

This is not an isolated problem for one vendor, but a widespread techno-economic tension in the current commercialization of large models. This article unpacks the technical roots of that tension from three angles—the inference pipeline, token economics, and engineering optimization—and offers actionable mitigation strategies.

Unit Prices Fall, Total Costs Stay Flat: A Misread Ledger

Consider a typical set of numbers. Compared with its predecessor, a new model version cut API input pricing from about RMB 20 to RMB 13 per million tokens, and output pricing from RMB 100 to RMB 68—roughly a one-third reduction. On paper, the cost advantage appears to grow. But long-term subscribers found that the new model consumed significantly more tokens for the same task, leaving overall cost nearly unchanged.

The key insight: what users actually care about is not unit price, but per-task cost. Per-task cost = unit price × tokens consumed per task. When unit price falls 33% while token consumption rises more than 50%, total cost can actually go up. This simple multiplication exposes a structural issue in LLM pricing that is easy to overlook.

Why Do Stronger Models Consume More Tokens?

To understand why token consumption balloons as capabilities improve, we need to look at the inference pipeline itself.

1. Chain-of-Thought and the Expansion of Intermediate Reasoning

Newer models generally produce much longer intermediate reasoning on complex tasks. Take code generation: an older model might output the result directly, while a newer model first decomposes requirements, analyzes edge cases, and drafts pseudocode before generating the final code. All of those intermediate steps are tokens.

Older model output structure:
[user request] -> [direct answer]
Newer model output structure:
[user request] -> [intent understanding] -> [task decomposition] -> [solution reasoning]
           -> [code generation] -> [self-check] -> [correction] -> [final answer]

Every reasoning step consumes tokens. The stronger the model, the longer the reasoning chain—and the larger the token overhead.

2. Context Accumulation in Long-Horizon Tasks

In long-horizon scenarios, the model must maintain cross-turn contextual consistency. To keep accuracy high over long tasks, newer models tend to re-read a more complete context at every turn rather than relying on lossy compression. As the number of turns grows, the input tokens per call accumulate.

3. Added Overhead from Tool Calls and Multimodal Inputs

When a model can call external tools or process multimodal inputs, every tool request, tool response, and image-encoded token counts toward consumption. The more comprehensive the capabilities, the more paths there are for additional overhead.

Caching: The Underestimated Cost Variable

In inference cost optimization, cache hit rate is a critical but frequently overlooked metric. Many platforms offer steep discounts on cache-hit input tokens, but whether the cache hits depends on how stable the request prefix is.

Because newer models have longer reasoning chains and more intermediate steps, request prefixes tend to be less similar across calls, driving cache hit rates down. In other words, unit-price discounts can be partially offset by cache misses.

Metric	Older model	Newer model
Input unit price	RMB 20 / million tokens	RMB 13 / million tokens
Cache hit rate	Higher	Lower
Input tokens per task	Baseline	Significantly higher
Actual cost per task	Baseline	Similar or slightly higher

Cost Control from an Engineering Perspective

Faced with the reality of “stronger capability, bigger bills,” developers and enterprise users are not limited to passive acceptance. The following strategies can rein in costs without meaningfully sacrificing quality.

1. Tiered Task Routing

Not every task needs the most powerful model. Tasks can be graded by complexity: route simple ones to lightweight models and reserve high-capability models for genuinely complex work. This routing layer can be implemented on the application side or handled through a general-purpose gateway.

def route_task(task):
    complexity = estimate_complexity(task)
    if complexity < THRESHOLD_LOW:
        return light_model.generate(task)
    elif complexity < THRESHOLD_HIGH:
        return standard_model.generate(task)
    else:
        return advanced_model.generate(task)

2. Context Pruning and Structuring

Before calling a high-capability model, compress and structure the context: strip redundant history and retain key state. This directly reduces input tokens and improves the odds of a cache hit.

3. Constraining Output Length

Use prompts or API parameters to constrain output length and prevent the model from generating unnecessary intermediate explanations. For structured-output scenarios, ask the model to emit the target format directly, cutting natural-language wrapping.

4. Monitoring and Attribution

Set up token-consumption monitoring at the task level to identify high-consumption task types and optimize them specifically. General-purpose monitoring, alerting, and logging services can help achieve this goal.

Conclusion: Rebalancing Capability and Cost

Better models deliver higher task quality, but they also make the inference cost structure more complex. A lower unit price does not mean a lower total cost—a techno-economic fact that deserves to be taken seriously.

For developers, understanding the underlying logic of token consumption and building task-tiered routing, context management, and cost monitoring is the key to balancing capability against budget. For platform providers, improving inference efficiency, raising cache hit rates, and offering finer-grained cost-control tools alongside capability gains will be an important competitive dimension in the next stage.

“The stronger the model, the harder the business”—the real meaning here may not be that capability itself is the problem, but that the cost structure needs to be redesigned and re-understood.