LLM Tiered Pricing and the Cost Math of the Agent Era

1. The Headline Numbers and What They Really Mean

A recent flagship model release announced new token pricing: the output rate climbed from roughly 27 CNY per million tokens in the previous generation to 100 CNY, while the input rate moved from 6.5 CNY (cache miss) / 1.1 CNY (cache hit) to 20 CNY / 2 CNY. On the surface, output is nearly 4x higher and input about 3x higher—easy to call a “flat-rate increase.” But stopping at headline numbers misses the real story: the relative ratios between tiers have been deliberately redesigned.

Two ratios tell the tale:

• Cache-miss input vs. cache-hit input: roughly 6x in the previous generation, now stretched to 10x.
• Output vs. cache-hit input: roughly 25x before, now 50x.

These ratios reveal a clear intent: reward prefix reuse with the cheapest tier, and charge a premium on long outputs and cold-start inputs. Understanding this is the first step to converting list prices into real business costs.

2. Converting List Prices into Effective Unit Costs

List prices are nominal “per million tokens” rates, but the actual cost depends on your workload profile. Let the cache hit rate be h, the miss-input price be P_miss, and the hit-input price be P_hit. The effective input price is:

P_in_eff = P_hit × h + P_miss × (1 - h)

Plugging in the new-generation numbers (P_hit = 2, P_miss = 20):

• h = 90%: P_in_eff = 2 × 0.9 + 20 × 0.1 = 3.8 CNY / M tokens
• h = 20%: P_in_eff = 2 × 0.2 + 20 × 0.8 = 16.4 CNY / M tokens

Now factor in the input-to-output ratio r (input tokens : output tokens). The blended unit cost becomes:

C = (r × P_in_eff + 1 × P_out) / (r + 1)

Two representative scenarios:

Scenario A — Enterprise agent: r = 20:1, h = 90%
C ≈ (20 × 3.8 + 100) / 21 ≈ 8.4 CNY / M tokens

Scenario B — Infrequent individual developer: r = 5:1, h = 20%
C ≈ (5 × 16.4 + 100) / 6 ≈ 30.3 CNY / M tokens

Same model, but the per-unit cost in Scenario A is roughly one-third to one-quarter of Scenario B. What looks like a universal hike is actually tiered pricing: steep discounts for high-reuse, long-context workloads and full price—or even a premium—for one-shot cold prompts.

3. Why Cache Hit Rates Can Exceed 90%

High hit rates don’t happen by accident; they depend on inference-architecture choices. A decoupled (split) inference architecture separates the prefill and decode phases. KV Cache can be reused across requests and managed centrally, so requests sharing the same system prompt, tool definitions, or conversation history incur near-zero cost during prefill. For workloads like coding agents—fixed system prompts plus large amounts of repeated context—hit rates above 90% are a direct result of architectural design.

This also explains the pricing logic’s internal consistency: if cache reuse substantially reduces server-side compute and memory overhead, the savings should flow back to high-reuse users via lower prices, while higher miss-input prices cover the real cost of cold starts.

4. Why Agent-Era Cost Structures Are Different

In the chatbot era, a single interaction consumed a few dozen to a few thousand tokens; even high per-token rates kept the absolute bill modest, so users were not price-sensitive. The agent era changes that premise: a single task can involve thousands of tool-calling steps, hundreds of thousands of tokens, and hundreds of agents running in parallel for days. Cost now scales linearly—or even super-linearly—with context length, as memory footprint, KV Cache storage, and decode-phase bandwidth all grow.

Under this load, maintaining chatbot-era low prices would make long-running tasks unprofitable on the inference side. Pricing must therefore align with real resource consumption: output is expensive because decode is serial and bandwidth-intensive; miss input is expensive because prefill actually computes; hit input is cheap because reuse is nearly free.

5. Selection Advice for Developers

1. Estimate your cache hit rate before looking at list prices. Agents with fixed system prompts, stable tool schemas, and reusable prefixes may pay as little as one-fifth of the headline input rate.
2. Controlling output length beats squeezing input cost. The output tier saw the largest increase, so concise responses, structured outputs, and reduced redundancy deliver the highest marginal savings.
3. Separate hot and cold paths. Move frequently reused context into stable prefixes and push one-shot content to the tail end—this lifts hit rates significantly.
4. Compare with blended cost, not headline rates. Plug your own r and h into the formula to compute C rather than comparing nominal per-token numbers.

def effective_cost(p_hit, p_miss, p_out, hit_rate, io_ratio):
    p_in = p_hit * hit_rate + p_miss * (1 - hit_rate)
    return (io_ratio * p_in + p_out) / (io_ratio + 1)
# Enterprise agent scenario
print(effective_cost(2, 20, 100, 0.9, 20))   # ≈ 8.4
# Infrequent individual developer scenario
print(effective_cost(2, 20, 100, 0.2, 5))    # ≈ 30.3

6. Conclusion

This round of price adjustments is not a simple hike—it is a shift from “flat-rate chatbot pricing” to “tiered agent-era pricing”: cache-hit discounts bind high-reuse enterprise workloads, while output and miss-input premiums cover actual inference costs. For developers, reading the ratio changes matters far more than memorizing headline numbers—your architecture determines your unit price. When building agent applications on cloud platforms such as Baidu AI Cloud, treating prefix reuse, output brevity, and hot/cold path separation as top-priority cost optimizations—aligning technical architecture with billing structure—is the key to keeping per-unit costs manageable in long-running, high-concurrency agent scenarios.