LLM Inference Pricing Logic and Cost Structure Explained

1. The Reality Behind Rising List Prices

A recent major model release triggered widespread discussion after its API pricing was announced. On the surface, the output price rose from roughly 27 yuan per million tokens in the previous generation to 100 yuan. Input prices shifted from about 6.5 yuan for cache misses and 1.1 yuan for cache hits to 20 yuan and 2 yuan respectively. Depending on the tier, prices appeared to rise by 3 to 4 times across the board.

But pricing is never a single-dimensional numbers game. What actually determines a user’s real expenditure is the effective unit price — the blended cost per million tokens after accounting for cache hit rate and the input-output ratio. To understand this properly, we need to examine how several key ratios have changed.

1.1 The Price Gap Between Cache Hits and Misses

In the previous generation, the uncached input price was about 6 times the cached input price. In the new version, that ratio has widened to 10 times. The output-to-cache-hit ratio also expanded from roughly 25x to 50x.

What does this signal? The cached input tier saw the most moderate increase of all tiers, while the output tier rose most aggressively. The pricing strategy is sending a clear message: reuse your cache, and pay a premium for frequent output generation.

1.2 Why Cache Hits Matter So Much

During large language model inference, the KV Cache stores previously computed attention key-value pairs. When the prefix of a new request matches a previous request, the system can reuse these cached values and skip the redundant prefill computation.

From an engineering perspective, the prefill phase is compute-intensive, while the decode phase is memory-bandwidth-intensive. Cache hits primarily save on prefill computation, but decode costs remain. This is why output prices are difficult to bring down, while cached inputs can be discounted substantially.

2. Cost Divergence Between Two Types of Users

Two typical scenarios illustrate how different usage patterns lead to very different costs for the same model.

Scenario A: Enterprise Agent with High Cache Hit Rate

Assume an input-output ratio of about 20:1 (common in long-context coding agents) and a 90% cache hit rate:

• Effective input price ≈ 2 × 0.9 + 20 × 0.1 = 3.8 yuan per million tokens
• Output price = 100 yuan per million tokens
• Blended cost per million tokens ≈ (20 × 3.8 + 1 × 100) / 21 ≈ 8.4 yuan

Scenario B: Independent Developer with Low Cache Hit Rate

Assume an input-output ratio of 5:1 and a 20% cache hit rate:

• Effective input price ≈ 2 × 0.2 + 20 × 0.8 = 16.4 yuan per million tokens
• Output price = 100 yuan per million tokens
• Blended cost per million tokens ≈ (5 × 16.4 + 1 × 100) / 6 ≈ 30.3 yuan

For the same model, Scenario A’s unit cost is roughly 3.6 times lower than Scenario B’s. The apparent across-the-board price increase is effectively a discount for enterprise customers who run stable system prompts with highly reusable prefixes, while independent developers with highly variable prompts face something closer to full price.

3. The Cost Logic Has Changed in the Agent Era

In the chatbot era, a single interaction involved dozens to thousands of tokens. Users were price-insensitive, and total costs were modest. The agent era is different.

In long-context coding agent scenarios, a single task can involve thousands of tool-call steps and hundreds of thousands of tokens. Running such workloads at the previous generation’s prices would lose money on every call — because VRAM usage, KV Cache storage, and decode-phase bandwidth consumption all grow linearly or even superlinearly with context length.

3.1 Architectural Cost Drivers

The new model version uses a larger sparse architecture: roughly 2.8 trillion total parameters, with 16 of 896 experts activated per token. While the activated parameter count is well controlled, several cost factors cannot be ignored:

• Resident weights in VRAM: Even with sparse activation, all expert weights must remain resident in VRAM or be efficiently swapped in and out, placing higher demands on memory capacity and bandwidth.
• Expert routing communication overhead: In MoE architectures, tokens must be routed to different experts. Cross-device communication overhead increases with the number of experts and the degree of distribution.
• Attention mechanism optimization: Kernel optimization under hybrid attention mechanisms requires substantial engineering investment to achieve acceptable throughput in real-world inference.

These costs ultimately show up in pricing. In other words, improved model capability is not free — it requires real compute investment.

3.2 The Role of Disaggregated Inference

One cloud provider’s technical approach uses a disaggregated inference architecture, deploying prefill and decode phases in separate resource pools that scale independently according to their distinct computational characteristics. The core benefits of this approach include:

• Prefill can be batch-processed for higher GPU utilization.
• Decode can be optimized for memory bandwidth, reducing per-token latency.
• Cache hit rates above 90% are achievable in coding scenarios.

High cache hit rates directly reduce the effective input price, bringing the blended cost of enterprise agent workloads below what the headline price suggests.

4. How Developers Can Optimize Inference Costs

Once you understand the pricing logic, you can optimize actual spending in several ways:

4.1 Increase Cache Hit Rate

• Stabilize system prompts: Keep instructions, tool definitions, and context templates that do not change between requests fixed in the prefix.
• Reuse conversation history: In multi-turn interactions, keep prefixes consistent rather than restructuring the prompt each time.
• Use prefix caching interfaces: Mainstream inference services typically offer explicit cache markers or automatic caching. Using them properly can significantly reduce costs.

4.2 Control Output Length

Output is the largest cost component. In agent scenarios, you can control output by:

• Having the model produce structured data instead of long natural-language text.
• Setting reasonable max_tokens limits.
• Using lighter models for intermediate steps and reserving the large model for key decisions.

4.3 Choose the Right Input-Output Ratio

If your use case is output-heavy, consider:

• Breaking generation tasks into retrieval plus short generation.
• Using streaming output with early stopping.
• Using smaller models for non-critical paths.

4.4 Monitoring and Attribution

Set up token-level cost monitoring and attribute costs by business line, agent task, and user to identify high-cost areas. Example code:

# Pseudocode: Track token costs by task
cost_per_million_input_hit = 2
cost_per_million_input_miss = 20
cost_per_million_output = 100
def estimate_cost(input_tokens, output_tokens, cache_hit_rate):
    hit = input_tokens * cache_hit_rate
    miss = input_tokens * (1 - cache_hit_rate)
    input_cost = (hit * cost_per_million_input_hit + miss * cost_per_million_input_miss) / 1_000_000
    output_cost = output_tokens * cost_per_million_output / 1_000_000
    return input_cost + output_cost

5. Summary

You cannot evaluate LLM inference pricing changes by looking at list prices alone. Cache hit rate, input-output ratio, and architectural costs together determine actual spending. Enterprise agent workloads with high cache reuse can achieve effective costs far below the headline price, while independent developers with highly variable prompts bear something closer to full price.

For developers, the takeaway from this pricing logic is clear: optimizing cache strategy, controlling output length, and choosing the right model size matter more to your final bill than simply comparing list prices. In the agent era, cost optimization is itself part of engineering competence.