The LLM Price War is Over; the Inference Cost War Has Just Begun

I. Three Signals Marking the End of the Price War

Over the past two years, the dominant theme in the domestic LLM API market has been price reduction. The input price for standard-tier models was once driven down to single-digit RMB per million tokens, and in some scenarios, free quotas became the norm. While effective for user acquisition, this strategy has a natural commercial ceiling. Once API pricing falls below the unit service cost for an extended period, larger call volumes only mean larger losses; funding can delay but not eliminate this problem.

The turning point emerged mid-year. When a leading vendor released its new flagship model, it raised the output price to the order of hundreds of RMB per million tokens, setting a price high for domestic models at the time. Based on the inertia of the previous price war, competitors should have immediately countered with more aggressive price cuts. The reality, however, was that no vendors followed suit, and the market calmly accepted a tiered structure of “expensive flagships, cheap standard models.” According to industry media, the vendor’s Annual Recurring Revenue (ARR) did not fall in the following quarter; instead, it achieved multiple-fold growth.

Three signals can be drawn from this market shift:

1. The primary competitive variable has shifted. Vendors no longer treat “the lowest price on the internet” as their top selling point. Instead, they emphasize whether the unit economics work out—whether they are actually making or losing money on every million tokens sold.
2. Users’ willingness to pay is tiered. For complex tasks that flagship models can reliably accomplish (deep reasoning, long-context processing, complex code generation, multi-step Agent tasks), price sensitivity is much lower than expected. Users are paying for “whether the model can get the job done,” not “how much a few cents per million tokens costs.”
3. Revenue structure is tilting toward subscriptions. ARR has become a more closely watched metric than API call volume, indicating that high-frequency, heavy users are migrating from pay-as-you-go to monthly membership/subscription models. Revenue predictability has become a new competitive dimension.

The end of a price war is never marked by prices hitting rock bottom, but rather by price no longer being the primary competitive tool. The next phase of competition has moved beneath the surface: given the same accelerator card, whoever can push the service cost per token lower will protect their gross margins and retain the space to lower prices in the future. This is the cost war. It is invisible to the user but determines who survives to the end.

II. The Cost Formula Behind Token Pricing

To predict where the cost war is heading, we must first unpack the cost structure behind a single token’s price. The service cost of an inference call can be roughly abstracted as:

Cost per token ≈ (Hardware depreciation + Energy & cooling + Network & datacenter + O&M labor)
               / (Effective throughput per card in tokens/s × Card time utilization)

On the numerator side, hardware depreciation usually accounts for the largest share. Taking a server equipped with eight high-end accelerator cards as an example, the total hardware purchase cost is amortized over three to five years. Amortized down to a single card, this represents a rigid expenditure on the order of dozens of RMB per hour. Energy and cooling are the second largest expense, directly impacted by the datacenter’s PUE (Power Usage Effectiveness). Together, these two items often account for over 70% of the total cost. Lowering prices does not change the numerator—no matter how you adjust the price list, the purchase price of the cards and the electricity bills remain the same.

The denominator consists of two engineering metrics: effective throughput (the actual number of tokens produced per second by a single card) and time utilization (the proportion of time during a day’s 86,400 seconds that the card is serving real requests). Idle time during early morning troughs burns the same depreciation.

This formula simultaneously explains two things: why the price war was unsustainable (the numerator is rigid, so cutting prices means bleeding money), and where the cost war should be fought (the denominator is the only offensive surface). Optimizing the denominator is a systematic engineering effort spanning four layers.

III. Four Battlefields of the Cost War

3.1 Model Layer: Activating Fewer Parameters per Token

Model architecture determines the cost ceiling. Currently, there are three main cost-reduction routes:

• MoE (Mixture of Experts) architecture: Splits the model into multiple “expert” sub-networks, routing each token to only a small fraction of them. This achieves “large total parameter count, small activated parameter count.” At the same quality level, reducing activated parameters directly lowers the computation and memory access requirements per token. This is the most important architecture-level means of improving throughput and is a route widely adopted by flagship models.
• Quantization: Compresses weights from FP16 to INT8 or even INT4, simultaneously reducing memory footprint and memory bandwidth requirements. Quantizing the KV cache can significantly increase the number of supportable concurrent sessions in long-context scenarios. Engineering-wise, this requires evaluating precision loss with calibration datasets to find an acceptable balance between “how hard to compress” and “how much accuracy is lost” for the specific task.
• Distillation and Small Model Tiers: Distilling flagship capabilities into smaller models allows simple tasks like classification, extraction, rewriting, and format conversion to be offloaded to small models, reducing costs by an order of magnitude. The existence of a model tier is a prerequisite for application-level “on-demand routing.”

3.2 Inference Layer: The Decoding Bottleneck is Memory Bandwidth, Not Compute

There is a counter-intuitive fact about autoregressive decoding: the computational load during the generation phase is not high. The card is often “able to compute but starved of data.” To produce a single token, relevant weights and the KV cache must be moved from memory into the compute units; memory bandwidth is the bottleneck. To address this, mature inference engines rely on several core techniques:

• Continuous batching: Dynamically batches different requests at the token level, eliminating the “one request finishes, the whole batch waits” bubble, directly boosting time utilization.
• Paged KV cache management: Manages the KV cache in a manner similar to an operating system’s virtual memory paging, pushing memory fragmentation down to single digits. The same card can host more concurrent sessions.
• Speculative decoding: A small model first “drafts” multiple tokens, and the large model verifies them in parallel in a single pass, accepting the correct parts. This accelerates decoding while keeping the output distribution unchanged.
• Prefill/Decode separation: The prefill phase is compute-intensive, while the decoding phase is memory-access intensive. Deploying them to different instances for independent optimization prevents them from dragging each other down.
• The cost trap of long context: Attention computation grows with sequence length, and KV cache memory footprint grows linearly with length. A small number of 128k long-context requests exert far more memory pressure than an equivalent number of short requests. This is exactly why long-context pricing must be significantly higher.

3.3 Resource Layer: Minimizing Card Idle Time

The cost governance of inference clusters shares similarities with traditional online services, but the load characteristics are more extreme: long-tail decoding phases, rigid memory occupation, and extremely high waste costs during off-peak hours. Available techniques include:

• Elastic scaling: Automatically scales inference instances up or down based on metrics like request queue length and queuing latency, reclaiming resources during troughs.
• Off-peak co-scheduling: Scheduling offline batch tasks (batch data synthesis, model evaluation, log summarization, and other latency-insensitive workloads) to off-peak hours to flatten the utilization curve.
• Memory partitioning: Allowing small-parameter models to share a single large card as multiple instances, avoiding the waste of “a small model exclusively occupying a large card.”
• Full-link observability: GPU utilization, memory footprint, Time To First Token (TTFT), Time Per Output Token (TPOT), and concurrent session count—every metric maps directly to the denominator of the cost formula. Cloud-native infrastructure like container platforms, logging services, and monitoring alerts play a role here no different in essence from traditional microservice governance: make utilization measurable first, then talk about optimization.

3.4 Application Layer: Saving Tokens That Shouldn’t Be Spent

While vendors are compressing costs, there is a parallel cost war on the user side, especially in Agent applications. Loop structures can multiply token consumption exponentially; it is common for a multi-step task to push hundreds of thousands of tokens back and forth. The optimization space at the application layer is often larger than expected:

• Prefix caching (context caching): Fixes content that “doesn’t change every round”—such as system prompts, tool descriptions, knowledge base snippets, and few-shot examples—in the prompt prefix. The inference engine skips repeated prefilling for cached prefixes, and billing usually offers a substantial discount for hit portions. A rational prompt structure design looks like this:

┌────────────────────────────────────────┐
│ System prompt + Tool specs (~2k tokens) │ ← Prefix: Cacheable, billed at discount on hit
│ KB retrieved snippets (~3k tokens)      │ ← Prefix: Cacheable
│ Conversation history summary (~500)     │ ← Updated every round
│ User input for this turn (~200 tokens)  │ ← Changes every time
└────────────────────────────────────────┘

• Model routing: Uses rules or lightweight classifiers to assess task difficulty first. Simple requests go to small models, while complex requests go to flagship models. In most business scenarios, a significant proportion of requests do not require a flagship model.
• Semantic caching: For repeated or highly similar questions, vector similarity matching directly returns existing answers, skipping inference altogether. Hit rates can be impressive in scenarios like customer service, FAQs, and internal knowledge bases.
• Context engineering: Trimming and summarizing historical messages in Agent loops rather than appending them indefinitely. On the output side, setting max_tokens limits, using structured output to reduce redundancy, and detecting task completion during streaming to stop early.

IV. Billing Models are Evolving in Tandem

The cost war is happening not just in the server room, but also in the structure of the price list. Current vendor pricing exhibits several common trends:

1. Tiered pricing replaces flat pricing: Flagship, standard, and lightweight tiers coexist. “An expensive flagship model” is no longer a disadvantage but a signal of capability.
2. Cache discounts: Cached context portions are billed at a steep discount or not billed at all. This passes the engineering benefits of prefix reuse to the user, encouraging users to design prompts in cacheable formats.
3. Batch channels: Providing low-cost batch processing entry points for latency-insensitive tasks (offline evaluation, batch labeling, data synthesis). In essence, this guides user workloads to help vendors fill utilization troughs.
4. Subscriptions and memberships: For high-frequency, heavy users, a monthly subscription is more cost-effective than pay-as-you-go, and vendors gain predictable ARR. The phenomenon mentioned at the beginning—where “revenue rose after a price increase”—is largely due to the stability brought by subscription revenue structures. Value pricing combined with subscription stickiness means vendors can retain core users without relying on being “cheaper” for the first time.

For buyers, this means cost evaluation cannot rely solely on the unit price of the price list. They need to look at what proportion of their workload can leverage caching, go through batch processing, or be offloaded to small models. Under the same price list, the actual cost difference between well-engineered and poorly-engineered implementations can vary by multiples.

V. A Developer’s Cost Governance Checklist

Translating this into daily engineering, an actionable governance process for LLM usage can be built in four steps:

Step 1: Measurement first. Tag every API call with business labels (business line, feature point, user tier). Write token usage, cache hit rates, and model tiers into logging and billing systems. First answer the question: “Where is the money going?”

Step 2: Build a cost dashboard. Observe the “average cost per session” and “cost distribution per task” by feature dimension to identify anomalous amplification points. These are usually caused by an uncapped loop or a piece of knowledge base being repeatedly stuffed into the context.

Step 3: Plot the quality-cost curve. Conduct A/B evaluations for the same feature using different model tiers. Find the lowest-cost tier combination that meets quality standards, rather than blindly using the flagship model for everything:

# Pseudocode: Estimate cost per call by tier (Unit: RMB / million tokens, numbers are illustrative)
def estimate_cost(input_tokens, cached_tokens, output_tokens, tier):
    price = {
        "flagship": {"input": 8,   "cached": 2,    "output": 32},
        "standard": {"input": 1,   "cached": 0.2,  "output": 4},
        "lite":     {"input": 0.3, "cached": 0.06, "output": 0.6},
    }[tier]
    cost  = (input_tokens - cached_tokens) / 1e6 * price["input"]
    cost += cached_tokens / 1e6 * price["cached"]
    cost += output_tokens / 1e6 * price["output"]
    return cost

Step 4: Implement routing and caching. Start routing strategies with rules (does it contain images, context length, task type) and gradually evolve them into lightweight classifiers. For semantic caching, set reasonable similarity thresholds and maintain invalidation and refresh mechanisms for cached answers to avoid “saving money but causing incidents.”

Budgets and alerts are equally indispensable. Set daily/monthly token budgets and threshold alerts for business lines to prevent “cost avalanches” caused by runaway Agent loops. A single uncapped failed retry loop can easily burn through a month’s budget.

VI. Conclusion

The end of the price war essentially reflects the market’s acknowledgment of LLM capability stratification. Complex tasks accomplished by flagship models cannot be replaced by cheaper models, giving legitimacy to “expensive” pricing. Meanwhile, the token consumption explosion brought about by Agentification on the application side has, for the first time, pushed the willingness to “pay for outcomes” past the inertia of “comparing unit prices.”

The cost war is much quieter. It hides in MoE routing strategies, in KV cache page tables, in batch tasks running off-peak at 3 AM, and in the prefix design of every prompt. For vendors, it determines margins and survival; for developers and enterprise users, it determines whether applications can run and keep running under real-world cost constraints.

Those who understand this war will ask one more question about “how caching is billed” during model selection, think one layer deeper about “what content can be fixed as a prefix” when writing prompts, and add one more “loop limit” when building Agents. Every million tokens saved is the most honest score in this game.