The LLM Price War Is Over — the Cost-Efficiency War Has Just Begun

Why Pricing Has Suddenly Stood Still

Over a span of just ten days in mid-to-late October, several leading LLM providers in China released new-generation models in quick succession — a pace dense enough to be called a “launch marathon.” What stood out even more than the release cadence was the pricing: nearly every provider kept prices for their new models identical to the previous generation. Only one vendor made modest adjustments to a subset of models.

Placed in the context of the past two years, this signal carries real weight. Until recently, nearly every model release came with headline-grabbing price cuts. Per-token prices for input and output were repeatedly driven down, often by another order of magnitude at a time. “Calling an LLM is cheaper than hiring someone to write” became a common marketing line. This time, the major players collectively decided not to lower prices.

That is no accident. Pricing is the most honest dashboard of any industry. When all players stop cutting prices at the same time, it means prices have hit — or are dangerously close to — the cost floor. The price war, as a competitive strategy, has run out of room.

Why the Price War Became Unsustainable: Three Hard Constraints

1. Inference is a real cost, and subsidies cannot last forever

Training is a low-frequency, budgetable, one-time expense. Inference, by contrast, grows linearly — or even super-linearly — with call volume. Every API request consumes real GPU time, VRAM, electricity, and bandwidth. Winning market share by pricing below cost is essentially buying scale with cash. When external funding tightens and commercial rationality returns, that path inevitably reaches a dead end.

2. The “model + API” offering is highly homogeneous, and price cuts do not buy loyalty

When product formats converge and capability differences are marginal across most general-purpose tasks, lowering prices builds no moat. It only drains the profit pool of the entire industry. Enterprise customers care about stability, throughput ceilings, long-context support, data compliance, and SLA guarantees — none of which can be replaced by “a bit cheaper.”

3. The industry has entered a margin-focused phase

Shifting from land-grabbing to revenue and margin generation is a familiar trajectory for all infrastructure industries. Holding prices flat is itself a statement: vendors would rather invest in model capability and engineering efficiency than sacrifice profit for market share.

From Open Battle to Quiet War: The Three Layers of the Cost War

The price war was an open battle: per-token prices were published on official websites for everyone to see, and the question was who dared to lose the most money. The cost war is a quiet war: for tokens sold at the same price, whoever has the lowest unit inference cost enjoys the thickest margins — and the strategic option to cut prices again at any time.

The cost war can be broken down into three layers:

• Model layer: Using better architectures and training methods to reduce the effective compute required to achieve the same quality.
• System layer: Applying inference engineering to squeeze every bit of throughput out of the hardware.
• Business layer: Using scheduling, caching, and mixed deployment to eliminate the waste of “using a sledgehammer to swat a fly.”

Five Technical Approaches to Driving Down Per-Token Cost

1. Model Architecture: Sparsity Makes “Large but Affordable” Possible

Mixture-of-Experts (MoE) is the canonical example. The total parameter count can be very large, but each token only activates a small subset of experts during the forward pass. In essence, nominal parameters determine the capability ceiling, while activated parameters determine inference cost. Combined with knowledge distillation — compressing a large model’s capabilities into a smaller student model — high-frequency, simple tasks no longer need to invoke the flagship model at all.

2. Inference Engines: Throughput Is Cost

• Quantization: Compressing weights and activations from high-precision floating point to FP8/INT8 or even INT4 reduces both VRAM footprint and memory bandwidth demand, significantly improving per-GPU throughput with controllable accuracy loss. During the decode phase especially, the bottleneck is often memory bandwidth rather than compute, making quantization’s impact on cost particularly direct.
• KV cache management: The dominant VRAM consumer in autoregressive decoding is the KV cache. Paged memory management can dramatically reduce fragmentation, allowing a single GPU to hold more concurrent sequences and directly raising effective throughput per unit of hardware.
• Continuous batching: Dynamically admitting new requests into the batch while letting completed requests exit promptly avoids the idle compute that occurs in traditional static batching when the whole batch waits for the slowest request.
• Speculative decoding: Using a lightweight model to rapidly draft multiple candidate tokens, then having the main model verify them in parallel in a single pass — increasing decode speed with almost no change to the output distribution.

3. Cost Governance for Long Contexts

KV cache grows linearly with context length, so the real cost curve of long-context scenarios is far steeper than what a per-token price table suggests. Engineering solutions include prefix caching (context caching), which retains and reuses the computed results of common prefixes such as system prompts and knowledge base documents, and prompt compression, which expresses the same information density in fewer tokens. For multi-turn agent applications, prefix cache hit rate often directly determines the margin of each session.

4. Resources and Scheduling: Utilization Is the Underappreciated Core Metric

How many GPUs you purchase only sets the ceiling on cost; utilization determines the actual per-token cost. Off-peak scheduling, co-locating offline batch jobs with online services, and elastic scaling based on load can all improve the ratio of effective throughput to total cost without adding hardware.

5. Business Layer: Let the Right Model Do the Right Job

A large share of online traffic consists of simple tasks — rewriting, classification, extraction, format conversion — that do not require a flagship model at all. Routing by task complexity, caching high-frequency requests with stable answers, and moving non-real-time tasks to low-load batch windows are all cost reductions achieved without touching a single line of model code.

Illustrative pseudocode (a common pattern for model routing plus multi-level caching):

def answer(query, context):
    # 1. Exact cache: high-frequency, stable requests hit directly, cost approaches zero
    if cached := cache.get(hash(normalize(query, context))):
        return cached
    # 2. Task tiering: route simple tasks to a small model, complex reasoning to the flagship
    if is_simple_task(query):        # classification, extraction, rewriting, etc.
        return small_model(query)
    # 3. Reuse common prefix cache to avoid redundant computation
    return flagship_model(query, prefix_cache=context)

Unit Economics: The Ledger That Decides the Cost War

The per-token cost of an inference service can be roughly expressed as:

Per-token cost ≈ total hardware cost (per second) ÷ effective throughput (tokens per second)

Total hardware cost includes depreciation or rental, electricity, data center, and operations amortization. Effective throughput is determined jointly by model architecture, engine optimization, and request load. As a purely illustrative calculation: suppose a cluster has a per-GPU total cost of 20 yuan per hour and effective throughput of 4,000 tokens per second. The cost per million tokens would be approximately 20 ÷ (4,000 × 3,600) × 10⁶ ≈ 1.39 yuan. If quantization and scheduling double effective throughput, per-token cost is immediately halved — the kind of work inference engineering teams do every day.

For buyers, this ledger matters just as much. The official price is only the “explicit cost.” True total cost of ownership (TCO) must also account for failed retries, rate-limit waits, long-context pricing premiums, hidden savings from cache hits, and switching costs.

A Practical Checklist for Developers and Enterprises

1. Select models by TCO, not by list price: Include throughput ceilings, stability, how long contexts are billed, and whether prefix caching is supported in your evaluation — not just the price per million tokens.

2. Build cost reduction into application architecture: Beyond model routing, multi-level caching, and batching, govern token consumption itself. If a task can be expressed in 200 tokens, do not write a 2,000-token verbose prompt.

3. Make costs observable: You cannot optimize what you cannot measure. Record input/output token counts for every request, tag them by business line, and set up cost dashboards and budget alerts. A skeleton for a billing middleware:

async def with_billing(req, handler):
    resp = await handler(req)
    usage = resp.usage  # prompt_tokens / completion_tokens
    metrics.emit(team=req.meta.team, model=req.model,
                 prompt_tokens=usage.prompt,
                 completion_tokens=usage.completion)
    if over_budget(req.meta.team):
        alert(f"{req.meta.team} token consumption is approaching the budget this cycle")
    return resp

4. Use hybrid deployment: For high-frequency, stable, predictable workloads, consider open-weight models running on your own infrastructure. For burst traffic and new feature exploration, prefer APIs first. Reserve heavy capital investment for scenarios that have already been validated.

5. Leverage platform capabilities: Teams without a dedicated inference engineering group can use managed inference services from cloud providers such as Baidu AI Cloud, offloading specialized work like VRAM tuning and elastic scaling to the platform. Pay-as-you-go pricing exchanges variable cost for predictable unit economics, which is often more economical than building and operating your own infrastructure.

Conclusion: The Price War Decides Near-Term Winners; the Cost War Decides Long-Term Survival

The end of the price war marks the LLM industry’s shift from “burning cash for scale” to “efficiency deciding survival.” For providers, every fraction of per-token cost reduction adds margin buffer and pricing flexibility. For enterprises and developers, understanding cost structure is what enables better decisions in model selection and architecture design. The key metric to watch in the coming years will no longer be the prices listed on official pricing pages, but the effective throughput curves buried in inference engine logs. The price war was only the opening act — the cost war is the deep water that will determine who is still standing at the end.