Behind the LLM API Price Hikes: Reconstructing Cost Models with Cache Pricing

The recent pricing announcement for a leading LLM provider’s next-generation flagship model has sparked intense discussion within the developer community. With listed prices of 100 yuan/million tokens for output and 20 yuan/million tokens for input—compared to the previous generation’s 27 yuan and 6.5 yuan—the hike is visibly significant, once summarized casually as a “5x price increase.” However, a detailed breakdown of the pricing tiers reveals that this adjustment is not a simple across-the-board hike, but a structural reconstruction aimed at the Agent era. This article breaks down the real costs: Is the new pricing actually expensive? For whom is it expensive, and for whom is it cheap? And as developers, how can we reduce costs under this new pricing structure?

1. Analyzing the Pricing Table: Structural Adjustment, Not a Blanket Surge

Using the provider’s publicly available pricing as a reference (unit: yuan/million tokens):

Billing Tier	Previous Gen	New Gen	Multiplier
Output	27	100	~3.7x
Input (Cache Miss)	6.5	20	~3x
Input (Cache Hit)	1.1	2	<2x

The multipliers across the three tiers are inconsistent, which is the most critical piece of information here. Let’s look at the shifts in two internal ratios:

• Miss ÷ Hit: Previous gen ~6x → New gen ~10x
• Output ÷ Hit: Previous gen ~25x → New gen ~50x

The conclusion is intuitive: the price increase for the cache hit tier is the mildest, while output and cache miss input prices have surged the most. The provider is deliberately widening the price gap between “reusing prefixes” and “one-time computation.” This is a classic pricing leverage strategy: using the low unit price of the hit tier to retain high-reuse customers, while re-pricing one-time traffic with premium rates on the miss and output tiers.

2. The Math: Calculating the Effective Cost Model

The listed price is only the starting point; the actual cost depends on two key variables:

1. Input-Output Ratio (I/O Ratio): In long-context coding agents and automated workflow applications, input tokens typically far exceed output tokens. Ratios of 20:1 or higher are common.
2. Cache Hit Rate: This depends on the reuse frequency of prefix content such as system prompts, tool definitions, and historical context.

The estimation formula for the effective input unit price is:

Effective Input Price = Hit Price × Hit Rate + Miss Price × (1 − Hit Rate)

The blended unit cost is then weighted by token volume:

Blended Cost = (Input Tokens × Effective Input Price + Output Tokens × Output Price) ÷ Total Tokens

Translated into Python code, it looks approximately like this:

def blended_cost_per_mtok(
    io_ratio: float,        # Input tokens : Output tokens, e.g., 20 means 20:1
    hit_rate: float,        # Cache hit rate, value between 0 and 1
    p_miss: float = 20.0,   # Cache miss input price (yuan/million tokens)
    p_hit: float = 2.0,     # Cache hit input price (yuan/million tokens)
    p_out: float = 100.0,   # Output price (yuan/million tokens)
) -> float:
    """Estimate blended unit cost (yuan/million tokens, weighted by total input + output)"""
    p_in_eff = p_hit * hit_rate + p_miss * (1 - hit_rate)
    total = io_ratio + 1
    return (io_ratio * p_in_eff + p_out) / total

The significance of this model is that any model selection evaluation should first estimate your specific hit rate and I/O ratio, then calculate the “effective cost,” rather than comparing raw listed prices directly.

3. Two Typical Scenarios: A 3.6x Unit Cost Difference on the Same Model

Scenario A: Enterprise Coding Agent. The I/O ratio is roughly 20:1 (standard for long-context agents). Because system prompts, tool schemas, and project specifications remain stable over time, the cache hit rate is approximately 90%:

• Effective Input Price ≈ 2 × 0.9 + 20 × 0.1 = 3.8 yuan/million tokens
• Blended Cost ≈ (20 × 3.8 + 1 × 100) ÷ 21 ≈ 8.4 yuan/million tokens

Scenario B: Individual Developer making occasional calls with different prompts each time. The I/O ratio is roughly 5:1, and the hit rate is about 20%:

• Effective Input Price ≈ 2 × 0.2 + 20 × 0.8 = 16.4 yuan/million tokens
• Blended Cost ≈ (5 × 16.4 + 1 × 100) ÷ 6 ≈ 30.3 yuan/million tokens

For the exact same model, the unit cost in Scenario A is only about 1/3.6 of Scenario B. So while the surface narrative is “universal price hikes,” the reality is that enterprise customers running consistent system prompts with highly reused prefixes receive a de facto discount. Individual developers changing prompts every time pay the full list price, or even a premium. Furthermore, under the same 20:1 load, simply increasing the hit rate from 20% to 90% drops the blended cost from ~20.4 yuan to ~8.4 yuan—a nearly 60% reduction. The hit rate itself is the most direct cost-reduction lever.

4. The Rationale: Real Inference Costs in the Agent Era

To understand the logic behind this pricing adjustment, we must look at the difference in usage patterns between two eras.

The Conversational Era: A single interaction typically involved only a few dozen to a few thousand tokens. Users were insensitive to unit prices, monthly bills rarely ran high, and a “flat token rate” was sufficient to cover costs.

The Agent Era is completely different. Taking a real-world scenario disclosed by the provider as an example: a single task can easily involve 4,000 tool-calling steps, consume hundreds of thousands of tokens, and even involve 300 agents running autonomously in parallel for 5 days. Under this load profile:

• The system prompt, tool definitions, and task context form a massive, relatively stable prefix.
• The incremental input for each tool-calling step is built upon this prefix.
• The benefit of caching transitions from a “nice-to-have” to “determining the profitability of each call.”

From the provider’s perspective, the real overhead on the inference side scales linearly or even super-linearly with context length:

1. KV Cache Memory Footprint: The longer the context, the larger the Key/Value states that must reside in memory. VRAM costs and fragmentation pressure increase simultaneously.
2. Decoding Phase Bandwidth Consumption: For every token generated, the system must read back the KV cache corresponding to the entire context. Bandwidth often becomes a bottleneck before compute power does.
3. Prefill Computation: The pre-fill compute volume for a one-time long input is proportional to its length. This is a fixed overhead that cannot be skipped in an agent’s multi-step loop.

If the conversational era’s pricing model were still used to support Agent workloads, high-frequency, long-context calls would likely be a loss-making endeavor for the provider. Therefore, the essence of this pricing adjustment is to align prices with the actual cost curve. High-reuse prefixes amortize memory and bandwidth overheads, making them inherently cheaper; one-time long inputs mean tangible prefill compute and memory consumption, justifying a higher price.

5. The Other Half of the Ledger: Hardware Expenses for MoE and Long Contexts

Based on publicly disclosed architectural information, the new generation model features approximately 2.8 trillion parameters and utilizes a sparse Mixture-of-Experts (MoE) design—activating only 16 out of 896 experts per token. The activated parameter count is well-controlled, which is MoE’s core advantage for managing single-token inference overhead. However, you can’t only account for the activated parameters:

• Resident Memory Weights are Full-Scale: Regardless of how many experts are activated, the weights of all experts must be loaded into VRAM. Memory requirements are tied to the total parameter count, not the activated count.
• Expert Routing Incurs Communication Overhead: Every token must undergo routing decisions and be dispatched to the corresponding experts. In multi-GPU environments, inter-card communication becomes a highly sensitive point for both performance and cost.
• New mechanisms like Mixed Attention can effectively compress the KV cache and alleviate memory pressure for long contexts. However, the accompanying operator development, kernel optimization, and inference framework adaptation represent continuous engineering investments.

In other words, while the theoretical compute volume for a single token may not have multiplied several times, the infrastructure required to support long-context, highly concurrent Agent workloads—VRAM, bandwidth, inter-card interconnects, and engineering optimization—demands hard cash. The price adjustment isn’t arbitrary; it’s a move to make these previously hidden costs explicit.

6. Developer Action List: Treat Hit Rate as a First-Class Metric

Rather than agonizing over list prices, treat this pricing structure as your own optimization framework:

1. Prefix Stabilization: Place fixed content like system prompts, tool definitions, and project specifications at the very beginning of the context, maintaining byte-level consistency. Place dynamic content (user input, timestamps, session variables) as far back as possible. A change in a single character within the prefix can invalidate the cache for all subsequent content.
2. Session Structure Management: Continuously append context within the same task rather than rebuilding the prompt every turn. Reuse the same tool schemas and specification documents across tasks to maximize hit probability.
3. Multi-Model Tiered Orchestration: Use flagship models for planning, complex reasoning, and critical code generation. Offload simple tasks like formatting, validation, and summarization to lightweight models or even local small models. Selection criteria should be based on the “calculated effective cost factoring in hit rates,” not list prices.
4. Establish Cost Observability: Integrate cache hit rate, I/O ratio, and per-task token consumption into your monitoring and alerting systems. Mainstream cloud providers—such as Baidu AI Cloud’s model services and monitoring capabilities—generally provide call logs, usage statistics, and cost-dimension analysis. You can use these tools to build a custom cost dashboard, ensuring that changes in hit rates immediately reflect in billing forecasts.
5. Define ROI for Cache Hits: Not every scenario warrants engineering effort to improve hit rates. Use the model from Section 2 to calculate first: how much will the blended cost drop for every 10 percentage point increase in hit rate? Then decide whether it is worth investing manual effort into prompt restructuring and scheduling optimization.

7. Conclusion: From “Token-Based Pricing” to “Usage-Pattern Tiered Pricing”

This recent round of so-called “price hikes” is more accurately described as a “reconstruction of the pricing structure”: transitioning from flat token-based pricing in the conversational era to usage-pattern tiered pricing in the Agent era. For enterprise Agent users, high hit rates and high prefix reuse mean de facto discounts; the core engineering task is to increase cache hit rates and eliminate invalid context. For low-frequency, highly variable individual developers, the list price is the real price, making the rational choice to delegate simple tasks to lighter model tiers. Understanding this structure and accurately calculating your effective cost is far more valuable than simply obsessing over “how many times the price has gone up.”