The Cost Logic Behind LLM Pricing Jumps: From Parameter Scaling to Inference Economics

A leading AI model provider recently adjusted the API pricing of its new flagship model to approximately ¥100 per million output tokens, ¥20 per million input tokens, and ¥1 per million tokens on cache hits—an overall increase of about five times compared to the previous generation. This move has sparked widespread discussion in the developer community: why would a provider raise prices while the industry generally trends toward “larger models, lower prices”? This article offers a冷静 breakdown of the underlying technical evolution and cost structure.

1. Parameter Scaling: From Trillion-Level to Multi-Trillion-Level

Understanding the pricing shift requires examining the trajectory of model size. Over the past two years, both open-source and closed-source camps have engaged in an arms race in parameter counts:

• The first generation was dominated by large-scale sparse Mixture-of-Experts (MoE) architectures with total parameters in the hundreds of billions to trillions, while active parameters were kept in the tens of billions, balancing capability and cost through sparse activation.
• The second generation introduced models with roughly 1.6 trillion total parameters, using finer-grained expert partitioning and improved routing strategies to significantly boost capability while keeping inference costs roughly flat.
• The new model at the center of this discussion reportedly reaches approximately 2.8 trillion total parameters, with both active parameters and expert count scaled up.

Growing from 1 trillion to 2.8 trillion parameters is not a simple linear increase. In a sparse MoE architecture, expanding total parameters implies:

1. Rigid growth in memory footprint: Even with FP8 or aggressive quantization, a 2.8-trillion-parameter model must be distributed across dozens or even hundreds of accelerators during inference. Cross-card communication overhead per forward pass rises significantly as the number of experts grows.
2. Increased routing complexity: More experts mean more complex token-to-expert routing decisions. Load balancing becomes harder, often leading to long-tail issues where popular experts are overloaded while less-used experts sit idle.
3. Exponentially higher training costs: Tripling the parameter count typically requires a proportional increase in training tokens to maintain convergence. Overall training compute may reach four to six times that of the previous generation.

In short, the leap in parameter scale directly raises the marginal cost per token—this is the most immediate physical basis for the price increase.

2. What Inference Costs Really Include: More Than Compute

Many developers habitually compare models by “price per million tokens,” but this metric obscures a complex cost structure. A single inference call involves at least the following layers:

2.1 Prefill Phase Compute

Input tokens go through full attention computation and feed-forward networks. Compute grows super-linearly with input length (attention is quadratic in long-context scenarios). The ¥20 per million input tokens reflects the compute cost of the prefill phase.

2.2 Decode Phase Memory Bandwidth Bottleneck

Autoregressive generation reads the full set of activated expert parameters for each generated token. This is a classic memory-bound scenario. The output-side price of ¥100 per million tokens is five times the input-side price—a ratio consistent with established engineering experience that decode is more expensive than prefill. For MoE models, each token routes to different experts, further amplifying cost through cross-card communication latency.

2.3 Cache Hit Economics

The cache-hit price of ¥1 per million tokens is roughly 1/20 of the input price. This corresponds to prefix caching: when multiple requests share the same prefix (system prompts, long document context), previously computed KV Cache can be reused, avoiding repeated prefill. For agent-based applications, system prompts often span thousands of tokens and are invoked repeatedly, making caching a significant lever for reducing real bills. This pricing tier signals that the provider encourages developers to optimize call patterns by placing reusable context at the prefix.

2.4 Long-Tail Costs: Scheduling, Fault Tolerance, and Operations

Large-scale MoE inference clusters incur scheduling overhead, fault recovery, load balancing, and elastic scaling costs, all of which are ultimately amortized into the per-token price. The larger the parameter count, the larger the cluster, and the higher the share of these hidden costs.

A simplified cost model can illustrate the structure:

# Simplified inference cost estimation model (illustrative, not actual parameters)
def estimate_cost(input_tokens, output_tokens, cached_tokens,
                  prefill_price, decode_price, cache_price):
    billable_input = input_tokens - cached_tokens
    cost = (billable_input / 1_000_000) * prefill_price
    cost += (cached_tokens / 1_000_000) * cache_price
    cost += (output_tokens / 1_000_000) * decode_price
    return cost
# Example: one call with 8,000 input tokens (6,000 cache hits) and 2,000 output tokens
cost = estimate_cost(8000, 2000, 6000,
                     prefill_price=20,
                     decode_price=100,
                     cache_price=1)
print(f"Estimated cost for single call: {cost:.4f} RMB")

This model demonstrates that with identical input/output scale, cache hit rate can lead to several-fold differences in actual cost. Developers should treat cache hit rate as a core optimization metric.

3. Why “Larger Model, Cheaper Price” Can Happen

The discussion also highlighted a counter-example: a competitor lowered prices while expanding from a smaller model to 1.6 trillion parameters. This seems contradictory but is actually explainable:

• Architectural efficiency gains: Finer-grained expert partitioning, better routing algorithms, and more aggressive quantization (e.g., FP8 mixed precision) can control or even reduce per-token compute despite parameter growth.
• Economies of scale: When inference clusters are large enough and utilization is high, marginal compute cost declines.
• Pricing strategy differences: Different providers make different choices based on market stage, profitability targets, and ecosystem strategies. Price does not always equal cost.

Therefore, “more parameters necessarily means higher price” is not an iron law. The real determining factor is effective compute consumption per token—the combined result of active parameter count, routing efficiency, quantization precision, and batching efficiency. A 2.8-trillion-parameter model with well-controlled activation and efficient routing may not have a higher per-token cost than a 1.6-trillion-parameter model. Conversely, if sparsity and aggressive quantization are sacrificed for quality, costs rise sharply.

4. Developer-Facing Strategies

Given the reality of higher flagship model pricing, developers and enterprise users can adopt the following engineering optimizations:

4.1 Tiered Routing: Match Tasks to the Right Model

Not every request needs the flagship model. Use lightweight models for intent recognition, simple Q&A, and format conversion. Route only complex reasoning and long-document analysis to the flagship model. This “model cascade” strategy reduces cost significantly while preserving quality.

def route_request(query, complexity_score):
    if complexity_score < 0.3:
        return call_lightweight_model(query)
    elif complexity_score < 0.7:
        return call_mid_model(query)
    else:
        return call_flagship_model(query)

4.2 Maximize Cache Hit Rate

Place stable system prompts, domain knowledge bases, and few-shot examples at the request prefix to ensure they are covered by caching. For multi-turn conversations, keeping the prefix stable and avoiding frequent changes to system messages is key to improving hit rates.

4.3 Control Output Length

Output tokens cost several times more than input tokens. Constraining output length (setting explicit limits, requiring concise structured responses, avoiding redundant explanations) yields more direct savings than optimizing input.

4.4 Leverage Batch Processing and Asynchronous Calls

For offline workloads, batch API endpoints often come with discounts. For tasks with low real-time requirements, asynchronous queues improve cluster utilization and lower cost.

4.5 Build Cost Observability

Instrument your application to log input, output, and cached token counts per call, and attribute costs by business dimension. Only with visibility can you identify optimization opportunities.

5. Conclusion: Pricing Is a Mirror of Technical Choices

Is a five-fold price increase reasonable? From a purely technical standpoint, a three-fold increase in parameters, along with expanded activation and communication overhead, makes rising costs physically inevitable. But whether the price is “expensive” depends on the capability increment per unit cost. If the new model’s improvements in complex reasoning, long-context understanding, and code generation outweigh the price increase, then for high-value scenarios, the cost per “effective task” may actually drop.

For developers, a more meaningful approach is to stop fixating on unit prices and instead build a task-based cost accounting framework: how many calls, tokens, and retries are needed to complete one business task—that is the true cost metric. Model pricing is a mirror of the provider’s technical roadmap; a developer’s competitive edge lies in using engineering discipline to push that mirrored cost as low as possible.