Why LLM API Prices Are Diverging: Inference Costs and Pricing Logic Explained

1. Pricing Divergence: One Scale Race, Two Price Trajectories

Looking back at the LLM industry over the past two years, “the scale race” is the most fitting summary: flagship model parameter counts climbed from the hundreds of billions into the trillions, with the latest generation approaching several trillion parameters, while long context, multimodality, and deep reasoning capabilities were stacked onto the same generation of models.

Yet as capability ceilings kept rising, API pricing split into two very different curves:

• Some vendors shipped new-generation models that are larger and more capable—while cutting unit token prices significantly;
• Others raised their new flagships across the board, with output prices several times the previous generation and input and cache-hit prices climbing as well. Take one recently publicized flagship price sheet: roughly RMB 100 per million output tokens, RMB 20 per million input tokens, and RMB 1 per million cache-hit tokens.

The community debate centers on one question: do the price hikes match the underlying technical investment? Answering it requires more than staring at a price table—you have to go back to the basic structure of inference cost.

2. The Cost Math of Scaling Laws: Why Bigger Models Are Naturally More Expensive

The server-side cost of a single API call can be roughly broken into four parts:

1. Weight memory footprint. Model parameters must stay resident in GPU memory. At FP8 precision, every trillion parameters corresponds to roughly 1 TB of storage. Even when a Mixture-of-Experts (MoE) architecture compresses the parameters activated per token to a few percent of the total, all experts still need to be addressable and routable across the cluster, so memory and interconnect costs keep growing with total parameter count.

2. KV cache overhead. During autoregressive decoding, every request’s conversation history occupies memory as a KV cache, growing linearly with context length. The ultra-long context windows flagship models now support directly inflate per-request memory cost.

3. Bandwidth and compute consumption. The decoding phase is a classic memory-bound process—every generated token requires re-reading the activated weights and the KV cache. The prefill phase, by contrast, is compute-bound: the longer the input, the greater the compute cost. This is also why output tokens are usually priced several times higher than input tokens.

4. Concurrency and utilization. GPU utilization, batch size, and request-scheduling efficiency determine how many requests share the amortization of fixed costs.

For vendors on the “large-parameter MoE + ultra-long context” path, the first three costs all rise with scale. If inference-side engineering optimization cannot keep pace with parameter expansion, rising unit costs will eventually surface in pricing—hikes are not arbitrary; they follow from the cost structure.

3. Why Some Vendors Get “Bigger but Cheaper”

Facing the same scale growth, other vendors have achieved “the bigger the model, the lower the unit price” by pulling a few key engineering levers:

1. Activation-ratio control. Under MoE, total parameters set the capability ceiling, while activated parameters determine per-token compute and bandwidth cost. Efficient routing algorithms can expand total scale while holding—or even reducing—the activation ratio.

2. Low-precision inference. FP8, INT8, and similar schemes compress memory and bandwidth consumption by multiples, and their accuracy stability has been widely validated in large-scale serving deployments.

3. Memory management and scheduling optimization. Techniques such as paged KV cache management, continuous batching, and disaggregated prefill/decode deployment can significantly raise cluster throughput and memory utilization.

4. Context caching. Repeated prefixes (system prompts, codebases, long documents) are cached and reused by subsequent requests, skipping prefill computation entirely. This is the technical basis of the “cache-hit price”: cached tokens require no recomputation, and serving cost can drop to a few percent of fresh input—which is why vendors can afford to sell those tokens at near cost.

5. Scale effects. The larger the user base and the denser the request traffic, the closer GPUs run to full utilization, and the more thoroughly fixed costs are amortized.

In other words, “bigger means pricier” versus “bigger means cheaper” is fundamentally a race between the speed of parameter expansion and the speed of inference engineering optimization. When the former wins, prices rise; when the latter wins, prices fall.

4. API List Prices Are “MSRP”: Subscriptions Change the Real Cost

One frequently overlooked fact: an API price list is, in a sense, just a “manufacturer’s suggested retail price.” Your actual cost depends on how you consume the tokens.

Today’s two mainstream consumption models have completely different cost structures:

• Subscription plans (fixed monthly fee plus a usage quota): costs are capped at the top. Once your usage approaches the quota ceiling, the effective per-token cost can fall far below the API list price; with light usage, the unit cost far exceeds it.
• Pay-as-you-go APIs: cost is strictly linear with usage, well suited to programmatic, bursty, large-scale workloads.

When evaluating a subscription’s real value, three easily missed details matter:

1. Quota reset policy. Some vendors reset usage quotas on hourly or weekly cycles; certain coding tools even reset quotas at high frequency during specific periods, effectively loosening the usage ceiling.
2. Quota pool separation. Chat products and coding tools may draw on two independently metered quota pools—identical nominal quotas can mean very different effective capacity and flexibility.
3. Rate limits and concurrency. Peak-hour rate limits determine whether heavy workloads can actually burn through the quota.

Stack these factors together and you get a counterintuitive conclusion: “Model X’s API list price is lower” may be completely untrue in a subscriber’s lived experience. This is especially true for heavy coding workloads, where a single task routinely carries hundreds of thousands of tokens of repository context. Differences in cache-hit rates, quota generosity, and reset frequency can quickly erase—or even reverse—a headline price gap. Vendors facing tighter compute supply tend to be less generous with subscription quotas, widening the gap between the “surface competitiveness” of list prices and the real costs paid by heavy users.

5. A Cost Accounting Method for Developers

Rather than arguing over list prices, build your own cost model. The actual cost of a single call breaks down as:

Total cost = uncached input tokens × input price
           + cache-hit input tokens × cache price
           + output tokens × output price
           (ultra-long-context requests may incur tiered surcharges)

Here is a Python snippet estimating a typical coding task:

def estimate_task_cost(input_tokens, output_tokens,
                       cached_tokens=0,
                       price_in=20.0,     # RMB per million tokens
                       price_out=100.0,
                       price_cache=1.0):
    """Estimate the cost of a single task (in RMB)"""
    fresh_in = max(input_tokens - cached_tokens, 0)
    return (fresh_in / 1e6) * price_in \
         + (cached_tokens / 1e6) * price_cache \
         + (output_tokens / 1e6) * price_out
# Typical coding task: 100K input tokens (partially cached), 5K output tokens
low_cache  = estimate_task_cost(100_000, 5_000, cached_tokens=10_000)
high_cache = estimate_task_cost(100_000, 5_000, cached_tokens=90_000)
print(f"Low cache hit rate: {low_cache:.2f} RMB")   # about 2.31 RMB
print(f"High cache hit rate: {high_cache:.2f} RMB")  # about 0.79 RMB

Simply raising the cache-hit rate from 10% to 90% cuts the per-task cost by roughly two-thirds. For a coding agent running hundreds of tasks per day, the monthly gap can reach thousands of RMB—often a better target for engineering effort than “which model to pick.”

Now compare against the subscription break-even point:

monthly_api = 50 * 30 * high_cache   # 50 tasks per day × 30 days
print(f"Pay-as-you-go monthly cost: {monthly_api:.0f} RMB")  # about 1185 RMB
# If a subscription plan costs 200 RMB/month and its quota covers the usage above,
# the effective unit cost is roughly 1/6 of pay-as-you-go

A matching checklist of cache-optimization practices:

• Stable prefixes. Pin system prompts, tool definitions, and codebase summaries to the front of requests to maximize prefix reuse.
• Control context bloat. Prune irrelevant files and stale history promptly; do not let useless tokens accumulate.
• Multi-turn within a session. Complete related tasks in the same session instead of rebuilding the full context each time.
• Monitor hit rates. Log cached-token counts in request logs and treat cache-hit rate as a core engineering metric.

These estimates also make one thing clear: a subscription is only a good deal when your usage approaches the quota ceiling. The core variable in subscription value is the quota policy (reset frequency, pool separation, rate limits)—not the monthly fee itself.

6. The Enterprise View: From Unit Price to TCO

For teams and enterprise buyers, total cost of ownership (TCO) matters more than “whose tokens are cheaper”:

1. Stability and throughput. SLA guarantees, peak concurrency capacity, and tail latency directly affect business availability. A cheap but unstable endpoint is a net liability in engineering terms.
2. Context and cache billing rules. For document batch processing or codebase-level agents, evaluate cache support and billing details first.
3. Data compliance and deployment model. When sensitive data is involved, consider calling, hosting, or privately deploying models through a cloud provider’s model service—such as Baidu AI Cloud—to keep the data boundary within a controllable perimeter.
4. Multi-model routing. Routing simple tasks to small models and complex tasks to flagship models is a common architectural lever for compressing overall cost—price sensitivity should be built on task tiering.
5. Avoid single-vendor dependence. Pricing across the industry is still adjusting rapidly: a new-generation release may raise prices, or be undercut on value by a challenger. Once competing vendors ship official versions trained with reinforcement learning post-training, the capability-per-price ranking could easily be reshuffled—it is too early to call winners.

7. Final Thoughts

The model scale race shows no sign of a finish line, but the pricing watershed has already appeared: some vendors are converting scale dividends into room for price cuts through engineering efficiency, while others need price hikes to cover expansion costs. For developers, agonizing over list prices has limited value—what really determines your cost is the shape of your workload (input/output ratio, context length, cache-hit rate) and whether you pay as you go or subscribe.

Run the numbers on those variables, and you will read any price sheet more accurately than most.