The Price War for Large Language Models is Over; the Cost War Has Just Begun

Over the past year, the pricing for Large Language Model (LLM) services has experienced a rapid decline. From early cautious per-thousand-token billing to single-digit quotes per million tokens, and even completely free conversational products, price competition has been pushed to its limits in an extremely short time. According to industry monitoring data, by the second half of 2025, the normalized average price for LLM token services had already fallen to just a few yuan per million tokens. For developers, this seems like great news: invocation costs are drastically reduced, and the barrier to trial and error is significantly lowered.

However, the end of the price war does not mean the cost problem has disappeared. On the contrary, when the unit price is compressed close to the marginal cost, the real competition has just begun—the Cost War. It is no longer about who offers the lowest quote, but about who can reliably complete the same tasks using fewer resources. For developers and enterprises, this means shifting from “looking at the unit price” to “looking at the Total Cost of Ownership (TCO),” and from “choosing the cheapest API” to “building a measurable, optimizable, and scalable inference cost system.”

1. The Fundamental Difference Between Price War and Cost War

The price war is a market behavior; the cost war is an engineering capability. The core of a price war is pricing strategy: capturing market share by lowering unit prices, even enduring phased losses in exchange for scale. The core of a cost war, however, is unit economics: how much compute, bandwidth, storage, and human resources are consumed to complete a single effective inference, and whether a positive gross margin can be maintained at a given unit price.

The goals of these two stages are different, and the focus areas for developers differ accordingly:

• Price War Stage: Focus on API quotes, free tiers, and promotional campaigns for rapid integration and validation.
• Cost War Stage: Focus on inference latency, throughput, VRAM usage, cache hit rates, and auto-scaling efficiency, as well as how these metrics impact the actual bill.

In other words, the unit price is just one factor in the cost formula. The true cost equals:

Total Cost = Invocation Volume × Unit Price + Infrastructure Cost + Engineering & Ops Cost + Failed Retry Cost + Opportunity Cost

When the unit price approaches zero, the latter four items become the decisive factors.

2. The Composition of Inference Cost: Five Dimensions Developers Must Break Down

To fight the cost war, you first need to understand where costs originate. LLM inference costs can be roughly broken down into the following five dimensions:

1. Compute Cost

This is the most intuitive part. Inference requires GPUs or specialized accelerator chips, billed by the second or hour. Compute cost depends on model size, precision, batch size, and concurrency. A common misconception is comparing only the per-million-token price while ignoring the difference in token consumption across models for the same task—a “cheaper” model might actually drive up the total cost due to verbose outputs or the need for multiple retries.

2. VRAM and Memory Bandwidth

LLM inference is a memory-intensive task. Model weights, the KV Cache, and intermediate activations all consume VRAM. Insufficient VRAM limits the batch size, which in turn reduces throughput and drives up the unit cost. Memory bandwidth affects the token generation speed; low bandwidth utilization means idle compute power.

3. Network and Storage

Transmitting inputs and outputs, logging, and storing caches all generate network and storage costs. For high-concurrency scenarios, network latency and bandwidth fees can become a hidden major expense. If you use object storage to save inference logs or cached results, you also need to consider request counts and data retrieval fees.

4. Engineering and Operations

This includes service deployment, monitoring and alerting, auto-scaling, version upgrades, and troubleshooting. These costs are often underestimated but account for a significant portion over long-term operations. An inference service lacking observability might continuously waste resources due to slow queries, memory leaks, or configuration errors.

5. Failures and Retries

LLM outputs are inherently uncertain. If an API times out, gets rate-limited, or returns substandard quality, a retry is required. Retries not only increase the invocation volume but may also trigger additional context processing. Under high concurrency, the amplification effect of failed retries is highly significant.

3. Core Paths for Cost Reduction: Optimization from Model to Architecture

The cost war is not simply about cutting budgets, but about systematically improving efficiency. The following paths are ranked by return on investment (ROI), and developers can choose based on their current stage.

1. Model Selection and Tiered Routing

Not all tasks require the largest and most powerful model. Grading requests by complexity—routing simple tasks to a small model and complex tasks to a large model—can significantly reduce average costs. For example:

def route_request(prompt, complexity_score):
    if complexity_score < 0.3:
        return call_model("small", prompt)
    elif complexity_score < 0.7:
        return call_model("medium", prompt)
    else:
        return call_model("large", prompt)

The key to tiered routing is accurately assessing complexity. This can be achieved using rules, lightweight classifiers, or historical statistics. The goal is to apply compute power only where necessary.

2. Prompt and Context Optimization

Prompt length directly impacts the input token count. Redundant system prompts, repeated examples, and untrimmed conversation history all increase costs. Optimization methods include:

• Compressing system prompts and removing redundant descriptions;
• Using summaries or vector retrieval instead of full-text context;
• Limiting output length to prevent the model from generating irrelevant content;
• Applying rolling windows or summaries to conversation history.

These optimizations do not require changing the model but can yield substantial cost reductions.

3. Caching and Reuse

Many requests are repetitive or highly similar. For identical or near-identical inputs, you can cache the inference results and return them directly. Caching can be built at multiple levels:

• Exact Cache: Direct hit for identical inputs;
• Semantic Cache: Matching similar inputs via vector similarity;
• Prefix Cache: Reusing the KV Cache for requests with the same prefix to reduce redundant computation.

Prefix caching is particularly effective in multi-turn conversations, as system prompts and early dialogue are often repeated.

4. Batching and Concurrency Control

Batching improves GPU utilization. Combining multiple requests into a single batch allows compute costs to be amortized. However, batching increases latency, requiring batch size adjustments based on business tolerance. Concurrency control prevents overload and avoids retries caused by queue timeouts.

# Pseudo-code: Dynamic batching
batch = []
while len(batch) < max_batch_size:
    request = queue.get(timeout=0.01)
    batch.append(request)
results = model.generate(batch)

5. Quantization and Inference Acceleration

Quantization reduces model precision, lowering VRAM usage and computational load. Common solutions include 8-bit integer quantization and 4-bit quantization. Quantization introduces some accuracy loss, requiring a trade-off between cost and performance. Inference acceleration technologies like operator fusion, attention optimization, and continuous batching can also boost throughput.

6. Auto-scaling and Resource Scheduling

Inference workloads typically experience peaks and valleys. Auto-scaling dynamically adjusts the number of instances based on traffic, preventing idle waste. Combined with container platforms and monitoring alerts, on-demand allocation can be achieved. For latency-insensitive tasks, you can further reduce costs by using spot instances or idle-time compute.

4. Observability: The Eyes of the Cost War

You cannot optimize what you do not measure. The cost war requires establishing a comprehensive cost observability system. Key metrics include:

• Cost per Request: Total cost divided by valid request count;
• Cost per Token: Tracked separately for input and output;
• Cache Hit Rate: Tracked separately for exact and semantic caches;
• Retry Rate and Failure Rate: To identify abnormal amplifications;
• GPU Utilization and VRAM Usage: To spot resource waste;
• P50/P95/P99 Latency: To balance cost and user experience.

These metrics should be integrated into your monitoring and alerting system with defined budget thresholds. When the cost per request exceeds expectations, it should immediately trigger an investigation.

# Example: Cost monitoring metrics configuration
metrics:
  - name: cost_per_request
    threshold: 0.005
    window: 5m
  - name: cache_hit_rate
    threshold: 0.6
    window: 15m
  - name: gpu_utilization
    threshold: 0.7
    window: 10m

5. Cloud Practices: Building a Cost System Using General Capabilities

In a cloud environment, developers can combine general-purpose services to build a cost system. For example:

• Use a container platform to deploy inference services for elastic scaling;
• Use object storage to save model weights and logs, accessing them on demand;
• Use message queues to buffer requests and smooth out traffic peaks;
• Use logging services and monitoring alerts to collect cost metrics;
• Use caching services to accelerate semantic cache queries.

These capabilities are universal and not tied to a specific vendor. The key is how to orchestrate them to form a closed loop. General cloud product categories provided by Baidu AI Cloud can support the architecture described above, but specific selections should be based on your team’s tech stack and cost objectives.

6. Long-Term Strategies for the Cost War

The cost war is not a one-time optimization; it is continuous engineering. The following long-term strategies are recommended:

1. Establish a Cost Baseline: Record your current cost per request and cost per token as a starting point for optimization.
2. Set Cost Targets: For example, aim to reduce costs by 20% per quarter, and break this down into specific modules.
3. Conduct Regular Load Testing and Tuning: Simulate real traffic to identify bottlenecks.
4. Track Model and Hardware Evolution: New models and new accelerator cards may bring orders-of-magnitude improvements.
5. Foster Cost Awareness: Ensure that development, operations, and product teams jointly focus on cost metrics.

The price war made LLMs accessible, but the cost war will determine who stays at the table for the long haul. For developers, this is both a challenge and an opportunity: those who first establish fine-grained cost engineering capabilities will be able to do more with the same budget and spend less for the same results.

The unit price of LLMs may continue to fluctuate, but the room for cost optimization will always exist. Starting today, treat cost as a first-class citizen, and your inference services will become more robust and sustainable.