The LLM Price War Is Over: How to Measure and Cut API Costs by Task
After months of steep cuts, the LLM API price war has wound down—but the pressure on cost hasn't. Competition has shifted from list price to the actual cost of completing a business task, and "list price" is no longer "your bill." This article traces how the price war ended, explains the six factors that silently inflate token bills, and lays out a task-level cost measurement method: defining tasks, building a tiered price table, instrumenting usage, and setting a cost regression gate. It then offers seven practical cost-cutting levers—model routing and cascading, context compression, output constraints, tiered caching, batching and off-peak scheduling, failure governance, and budget guardrails—plus guidance on treating model capability as a schedulable resource and a four-step rollout plan. Developers will learn to measure, reduce, and protect LLM costs without sacrificing quality.
1. How the Price War Ended
1.1 A Price Curve That Kept Falling
In May 2024, a model vendor in China pushed its flagship model’s input price down to RMB 1 per million tokens, and leading cloud providers followed within a week. The decline didn’t stop there: in 2025, some vendors made their consumer-facing chat products free (Baidu among them), and a new generation of flagship models priced their APIs at roughly one-hundredth of comparable overseas flagships. Monitoring by domestic research institutions shows that by the second half of 2025, the normalized average price of token services in the Chinese market was about RMB 6.8 per million tokens.
One important caveat: this average is normalized from official list prices at each model’s release. It is not the average of what users actually pay. Real bills depend on the mix of input and output tokens, cache hit rate, number of retries, free quotas, and commercial discounts.
1.2 Three Structural Reasons the Cuts Lost Steam
First, diminishing marginal utility. When input prices fall to RMB 1 per million tokens, the model cost of a typical call is usually just a few cents. Pushing lower has little effect on purchasing decisions. At this point users care more about whether the task completes reliably and whether responses arrive on time; meanwhile data governance, systems integration, and labor often make up the bulk of total cost.
Second, revenue constraints. Low prices quickly grow call volume but struggle to cover ongoing R&D and compute investment. Some vendors have begun raising flagship prices to return pricing to a normal commercial value, clearly separating the accessible tier from the flagship tier.
Third, a shift in what’s being compared. Once prices are low enough, vendors no longer compete on “how much per million tokens” but on “how much it costs to run the same task on your platform.” The latter depends simultaneously on model capability (how many calls are needed), context length (how much input per call), output length (including reasoning tokens), failure and retry rates, and caching and batching capabilities. It’s the result of a vendor’s overall strength and can’t be achieved by simply lowering the list price.
1.3 Price Tiering and the Cluster of Releases in H2 2025
The end of the price war doesn’t mean prices rose across the board; rather, the market has been cleanly split into two tiers:
- Accessible tier: for high-frequency, standardized, cost-sensitive tasks such as classification, extraction, rewriting, and customer-service Q&A. This tier keeps falling, and some vendors even offer free quotas. Differences between vendors are small, and competition has shifted to cost per task.
- Flagship tier: for high-value tasks such as complex reasoning, code generation and repair, long-document analysis, and multi-step agents. These tasks carry high unit prices, but a single successful call can save substantial labor, and customers are willing to pay for results.
The logic of tiered pricing resembles instance tiers in cloud computing: general-purpose, compute-optimized, and high-performance instances each serve different workloads. The premium on the flagship tier in turn supports the affordability of the accessible tier, and forces enterprises to first answer a business question: what is each successful run of this task worth?
In October 2025, multiple vendors released a new generation of models within about ten days. Based on public pricing, most new products held steady with their predecessors; the focus of the launches shifted to “how much it costs to complete the same task,” and the unit of measurement moved from model list price to cost per task. The market won’t turn on a few launches, but customers’ bills will change noticeably over the coming period.
1.4 From the Enterprise Procurement View
Price tiering turns procurement from “pick the cheapest vendor” into “allocate compute by task”:
- Inventory tasks: break the business into measurable tasks such as ticket classification, contract extraction, code completion, and report interpretation.
- Assess value density: estimate the business value of each successful task—labor minutes saved, conversion lift, rework avoided.
- Choose a tier: assign each task an “effect-sensitive” or “cost-sensitive” attribute, then pick a flagship model, a lightweight model, or a cascade that tries lightweight first and escalates to flagship.
This mirrors read/write splitting and hot/cold tiering in system architecture. The only difference is that “hot data” becomes high-value tasks and “cold data” becomes high-frequency simple tasks.
2. List Price Is Not Your Bill: How Costs Get Amplified
2.1 The Basic Structure of a Token Bill
Mainstream LLM services generally bill “by token,” distinguishing at least input (prompt) tokens from output (completion) tokens, with output priced higher than input. On top of that, three common mechanisms exist:
- Cache pricing: discounted rates for repeated prompt prefixes; some inference engines reuse key-value caches to cut redundant computation.
- Time-of-day pricing: different prices by call window to steer load toward idle periods.
- Reasoning tokens: some reasoning models bill their internal reasoning process as output.
Code 1: A token-level cost accumulator
# Illustrative code: assume input, cached input, and output have different unit pricesdef token_cost(usage, price):uncached = usage.input_tokens - usage.cached_input_tokensreturn (uncached * price.input+ usage.cached_input_tokens * price.cached+ usage.output_tokens * price.output)task_cost = sum(token_cost(resp.usage, price) for resp in task.responses)
2.2 Six Factors That Drive Up Real Costs
Factor 1: Context bloat. Retrieval-augmented generation (RAG) stuffs retrieved snippets into the prompt, multi-turn conversations keep full history, and tool-call responses keep accumulating. Poorly managed context can multiply input tokens.
Factor 2: Output and reasoning tokens. Generating full explanations, large structured objects, or enabling deep-thinking mode all lengthen output. Output is usually priced higher than input, and reasoning tokens weigh even more.
Factor 3: Tasks split into many calls. One agent task may cycle through “plan → call tool → read result → reflect → call again.” Even if each call is cheap, the total adds up.
Factor 4: Failures and retries. Malformed output, failed tool calls, and failed validation all trigger retries, and each retry is billed on the full input again. This is the most easily overlooked cost sink.
Factor 5: Cache utilization. If system prompts, tool descriptions, and fixed knowledge are resent every request and can’t hit the cache, they’re billed at full price. Whether you hit the cache depends on how calls are made and how prompts are organized.
Factor 6: Time-of-day and billing mode. With tiered time-of-day pricing, concentrating calls during the day pushes up the average. Teams without committed usage or off-peak scheduling effectively pay a “peak premium.”
2.3 An Illustrative Example
The table below uses three tasks to show that even on the same model tier, costs can differ by tens of times. All figures are illustrative:
| Item | Task A: Ticket classification | Task B: Long-document Q&A | Task C: Multi-step data-processing agent |
|---|---|---|---|
| Task flow | Single call, outputs a label | Single call, summarizes key points | Retrieve, compute, validate, generate report—4–6 calls |
| First-call input tokens | 800 | 20,000 | 12,000 |
| First-call output tokens | 150 | 1,500 | 800 |
| Average retries | 0.05 | 0.3 | 1.2 |
| Per-call total cost index | 1.0 | 17.5 | 16.0 |
| Per-task total cost index | 1.05 | 22.75 | 115.2 |
| Output cost share (incl. retries) | ~63% | ~46% | ~24% |
Assumptions: input unit price is 1, output unit price is 4; retries are estimated at the same token volume as the first call.
Three tasks on the same model tier, at the same prices, cost anywhere from about 1 to about 115 per task—a hundredfold difference. The conclusion is direct: what determines the bill isn’t the unit price, but how many times the task is called, how much input and output each call has, and how many times it fails and retries.
Looking further into cost structure reveals where to optimize:
- Task A: output is about 60% of cost. Reducing output length and constraining output format gives the most direct payoff.
- Task B: input is more than half. Compressing retrieved snippets, summarizing history, and prefix caching offer the most value.
- Task C: the most calls, with retries amplifying cost. Routing tiers, streamlining the flow, and failure governance are the key.
Different tasks need different optimization tactics—which is exactly why “accounting by task” is more valuable than “comparing by model.”
3. Task-Level Cost Measurement: From Measurable to Controllable
The first step to cutting cost isn’t switching models—it’s measuring cost accurately. Proceed in these steps.
3.1 Define “Task” Consistently
A task is the smallest unit of value from a business perspective, usually corresponding to one complete handling of user intent: a user asks a question and gets an answer—that’s a task; a user issues an instruction and an agent runs multiple steps to produce a result—that’s also a task. A task can contain any number of model calls, but they should share the same task ID, and only ultimately successful tasks are counted.
3.2 Build the Cost Formula and a Tiered Price Table
Per-task cost = (input token cost + output token cost + reasoning token cost − cache discount) / number of successful tasksPer-call cost = Σ(each token type × its unit price)
The formula itself isn’t complicated; the difficulty lies in two points:
- Whether unit-price data is accurate: each model’s input, cached, output, and time-of-day prices must be synced promptly and normalized to a consistent billing basis.
- Whether the statistics are complete: if usage isn’t recorded when calls fail, time out, or are abandoned mid-way, the numerator comes out too small, per-task cost looks like it dropped, but the cost has actually been shifted into a worse user experience.
Maintain a tiered price table injected centrally by the platform so business code doesn’t hardcode prices:
# Illustrative code: configure prices and budgets by "tier × capability type" in one placePRICE_TABLE = {"light": {"input": 1.0, "cached": 0.1, "output": 4.0},"flagship": {"input": 12.0, "cached": 1.2, "output": 48.0},}def estimate_cost(model_tier, usage):p = PRICE_TABLE[model_tier]uncached = usage.input_tokens - usage.cached_input_tokensreturn (uncached * p["input"]+ usage.cached_input_tokens * p["cached"]+ usage.output_tokens * p["output"])
3.3 Full Instrumentation, Unified Attribution
Instrument the single entry/exit point for model calls, and record each call’s usage, price, and task ID. Suggested fields (illustrative):
{"task_id": "task-20250921-0001","trace_id": "trace-7f3a","scene": "ticket_classify","tenant_id": "org-42","model": "light-v3","route_reason": "rule_hit","call_seq": 1,"input_tokens": 812,"cached_input_tokens": 0,"output_tokens": 153,"reasoning_tokens": 0,"retry_index": 0,"status": "success","quality_check": "pass","latency_ms": 860,"cost": 0.00081}
# Illustrative code: a lightweight "task context" handles metering and budget controlclass TaskContext:def __init__(self, task_id, scene, tenant_id, budget):self.task_id, self.scene, self.tenant_id = task_id, scene, tenant_idself.budget, self.cost = budget, 0.0def call(self, model_tier, messages, **options):if self.cost >= self.budget:raise BudgetExceeded(self.task_id) # Budget guardrail: prevent runaway loopsresp = client.chat(model_tier, messages, **options)self.cost += estimate_cost(model_tier, resp.usage)emit_usage(self, model_tier, resp) # Report asynchronously, don't block the main flowreturn resp
Business code only needs to create a context at the task entry; all model calls go through it, and cost is automatically aggregated to the task level.
3.4 Build a Core Metrics Dashboard
| Metric | Meaning | Use |
|---|---|---|
| Cost per task (RMB/task) | Total cost of all calls to complete the task divided by successful tasks | Core comparable metric externally and a KPI internally |
| Average input/output tokens | Tokens consumed per successful task | Judge whether cost is input- or output-dominated |
| Reasoning token share | Reasoning tokens as a share of output tokens | Judge whether the reasoning mode is overused |
| Cache hit rate | Share of input tokens that hit the cache | Measure prefix-cache and semantic-cache effectiveness |
| Retry rate and first-pass success rate | Share of calls that triggered retries; share of tasks that passed on the first call | Measure stability and failure-governance effectiveness |
| Cost P50/P90/P99 | Percentiles of per-task cost | Identify long-tail high-cost tasks |
| Output per unit cost | Successful tasks per RMB of model cost | Tie cost to technical investment and business value |
3.5 Bring Cost into the Regression Gate
Cost measurement must eventually enter the development workflow as one of the “quality gates” for model upgrades and prompt changes. Illustratively:
# Illustrative code: validate both quality and cost in the evaluation pipelinefor case in regression_set:result = run_task(case)assert result.quality_score >= BASELINE[case.id] * 0.98assert result.cost <= BASELINE[case.id] * 1.05
Without this gate, a seemingly harmless prompt tweak can quietly push per-task cost up. The “normalized average price” that industry research institutions monitor is a statistic of release list prices, whereas the real average bill depends on your own call structure. The per-task cost you measure yourself is the reliable basis for budgeting and optimization.
4. Seven Cost-Cutting Levers, Matched to Task Structure
Lever 1: Model Routing and Cascading
Not every task deserves the flagship model. Process tasks in tiers “rule → lightweight model → flagship model”: if a rule can decide, decide directly; if a lightweight model can complete the task and pass validation, finish on the spot; only escalate hard cases to the flagship model.
Code 3: Routing and cascading
def solve_with_routing(task, ctx):# Tier 1: tasks a rule can decide cost zero model callsif hit := rule_engine.match(task):return hit# Tier 2: lightweight model + structured validationdraft = ctx.call("light", build_messages(task),response_format=task.schema)if validate(draft, task) and confidence(draft) >= 0.85:return draft# Tier 3: escalate hard cases to the flagship modelreturn ctx.call("flagship", build_messages(task, draft),response_format=task.schema)
The key to routing is that the signal for “should we escalate” must be cheap and reliable—validation results of structured output, confidence, rule-conflict detection. If the escalation decision itself requires an extra model call, the savings may be canceled out.
Lever 2: Context Compression and Stable Prefixes
Context is the main source of input cost and can be governed in three directions:
- Rerank and compress retrieved snippets: sort by relevance first, keep only the top few, and drop redundant phrasing.
- Summarize conversation history: multi-turn conversations don’t need to keep all history; use rolling summaries that keep only key conclusions and constraints.
- Stable prefixes: put system prompts, tool descriptions, and fixed knowledge at the front and keep them unchanged, to help hit prefix caches.
Code 4: Stable context assembly
def build_messages(task, history, docs):stable = system_prompt() + tool_specs() + output_schema() # Keep unchanged as much as possibledynamic = (summarize(history, max_tokens=500)+ rerank_and_compress(docs, max_tokens=2000)+ f"Current task: {task.input}")return stable + dynamic
A stable prefix has an extra benefit: the same prompt prefix can be reused across multiple requests. Some inference engines reuse already-computed key-value caches for repeated prompt prefixes, cutting the cost of redundant computation.
Lever 3: Constrain Output Length and Reasoning Depth
Output is priced higher, so controlling output is often more cost-effective than controlling input:
- Constrain format via structured output, avoiding lengthy natural-language explanations.
- Change “detailed analysis” to “conclusion plus key points,” expanding on demand.
- Enable reasoning mode by task difficulty; simple tasks don’t enter deep thinking.
Beware: over-compressing the reasoning process can degrade quality and trigger retries. Saving one call at the cost of two retries is a losing trade.
Lever 4: Tiered Caching
Caching is one of the highest-ROI tactics, but it must be designed in tiers:
- Prefix cache: use the model service’s prefix-caching capability, putting system prompts and fixed knowledge at the front.
- Exact cache: use a hash of “prompt content + key parameters” as the key, caching results for identical requests.
- Semantic cache: reuse answers for similar questions; set a similarity threshold, a sensible expiry time, and record the cache source for easier debugging of wrong answers.
Code 5: Semantic cache with a confidence constraint
def cached_answer(query, cache, threshold=0.95):hit, score = cache.search(query)if hit and score >= threshold:return hit.answer, {"source": "semantic_cache", "score": score}return None, None
In high-repetition scenarios such as classification, customer-service Q&A, and knowledge Q&A, semantic caching often significantly lowers per-task cost. But it essentially trades “possible errors” for cost, so the threshold and expiry policy must be designed together with the business’s quality requirements.
Lever 5: Batching, Async, and Off-Peak
- Batching: for offline tasks such as bulk extraction and mass labeling, prefer services that support batch calls; these are usually processed asynchronously and offer price discounts.
- Async: decouple non-real-time tasks via message queues, with consumers sending smoothly under the provider’s rate limits.
- Off-peak: if the model service uses time-of-day pricing, schedule deferrable tasks into low-price windows.
# Illustrative code: route offline jobs into separate lanes by prioritydef dispatch(jobs):for job in jobs:queue = REALTIME_Q if job.latency_sensitive else BATCH_Qqueue.put(job)
The cost of batching is slower responses, so it suits only latency-insensitive tasks.
Lever 6: Failure Governance and Retry Control
Failed retries are often where the bill rises fastest. Recommended:
- Validate structured output immediately and regenerate if non-compliant.
- Set a retry cap and a total budget cap per task to avoid infinite loops.
- Narrow the scope on retry—retry only the failed step, not the whole flow.
- Record the distribution of failure reasons, distinguishing model, prompt, and tool problems, to reduce retries at the source.
Lever 7: Usage Governance and Budget Guardrails
- Tenant-level quotas and rate limiting: allocate budgets by team and scenario to prevent a few abnormal calls from dragging down the whole.
- Abuse detection: identify and block high-frequency, repeated, or suspected malicious requests.
- Budget alerts: alert promptly when per-task or per-tenant cost exceeds a threshold.
- Cost capacity planning: forecast call volume alongside business growth, and evaluate whether to keep pay-as-you-go or adopt committed-usage options to optimize long-term cost.
5. Supply-Side Restructuring: Treat AI Capability as a Schedulable Resource
The cost war isn’t just prompt engineering—it’s a resource-scheduling problem. A mature architecture should manage model capability the way it manages compute and storage:
- Unified model gateway: abstract away differences between models and handle auth, routing, rate limiting, metering, and fallback in one place.
- Elastic queues: use message queues to smooth peaks and troughs, spreading burst traffic into low-price windows.
- Observability: logging and monitoring/alerting record call chains and usage; object storage keeps evaluation samples and cost reports.
- Hybrid deployment: route high-frequency simple tasks to self-hosted lightweight models and complex tasks to cloud APIs, splitting dynamically by unit cost.
For self-hosted inference, three technical factors strongly affect cost:
- Sparse expert (Mixture-of-Experts, MoE) structure: activates only part of the parameters per call, lowering compute per token, but all parameters must stay resident in memory, demanding more GPU memory.
- Key-value cache management: long contexts linearly increase KV-cache memory use; paged management helps reduce memory fragmentation and improve concurrency.
- Speculative decoding: a small model guesses ahead and a large model verifies in parallel, lowering decode latency and also helping throughput and unit cost.
These techniques share one goal: serve more requests on the same hardware, directly diluting cost per token. Typical cloud capabilities include object storage, message queues, container platforms, logging, and monitoring/alerting. Baidu AI Cloud, for example, offers such capabilities to support the cost-governance system above: inference services run on a container platform, offline tasks are consumed asynchronously via message queues, call data flows into logging services, and monitoring/alerting handles budget and anomaly notifications.
6. Implementation Roadmap: Four Steps to Turn the Cost War into a Long Game
| Stage | Duration | Key actions | Deliverables |
|---|---|---|---|
| Step 1: Unify the basis | 1–2 weeks | Define task boundaries, unify the price table, map existing call chains | Cost-basis doc, tiered price table |
| Step 2: Observability | 2–3 weeks | Instrument the single entry/exit, build a task-level cost dashboard and alerts | Per-task cost dashboard, budget guardrails |
| Step 3: Attribution analysis | 2 weeks | Break down cost by scene, tenant, and task type to find long-tail high-cost tasks | Cost-attribution report, list of optimization opportunities |
| Step 4: Optimization loop | Ongoing | Combine routing, compression, caching, batching, and failure governance; hold quality with the regression gate | Cost-reduction curve, quality-regression report |
Avoid three common mistakes:
- Blanket downgrading: switching all tasks to the cheapest model; quality drops, retries surge, and total cost rises rather than falls.
- Looking only at averages: averages hide the long tail; watch P90 and P99 too, and identify the few high-cost tasks.
- One-off optimization: model capability, prices, and traffic all change; cost optimization should be a standing mechanism folded into the iteration cadence.
7. Conclusion
The end of the price war doesn’t mean cost pressure is gone—competition has moved from “list price” to “the bill.” Enterprises no longer ask “who’s cheaper” but “which task, done how, at what cost, for what result.”
For developers, three things are urgent:
- Measure cost down to the task.
- Engineer the optimization tactics.
- Make the cost gate routine.
When unit price is no longer the scarce resource, real competitiveness lies in cost per task, quality stability, and the ability to keep improving. Whoever puts every cent of model spend on the tasks that matter will go further in the cost war.
