LLM Output Token Pricing Jumped 5x: A Technical Guide to the New Inference Cost Structure
When a model vendor's new flagship model raised API prices by roughly 5x, developers asked what the increase actually means. This article breaks down the shift from input-side to output-side costs in the reasoning era, explains why prompt-cache hit rate is the single biggest cost lever, and shows how per-task pricing replaces per-token pricing in agent workflows. It also compares open-weight self-hosting with managed services, covers TCO and cost-control tactics, and closes with a practical cost-governance checklist for teams that need replaceable, observable, and controllable inference spending.
Recently, a model vendor’s new flagship model (referred to here as “the new-generation model”) drew heavy discussion across the developer community over its API pricing: output tokens cost about RMB 100 (¥100) per million tokens, input about ¥20, and cache-hit input about ¥1. Compared with the previous generation, overall pricing rose by roughly 5x. The core of the debate is not simply “is it expensive,” but a shift developers need to understand: the cost structure of the inference era has changed fundamentally.
1. The Cost Center Has Shifted: Why Output Pricing Is the Main Battleground
1.1 The Fundamental Difference Between Prefill and Decode
Transformer inference has two phases:
- Prefill: encodes the entire input prompt in one parallel pass. It is compute-intensive and can leverage GPU parallelism to process efficiently.
- Decode: generates tokens autoregressively, one token per step. Each step must repeatedly read model weights and a growing KV cache, making it memory-bound.
This difference directly drives pricing strategy. The previous generation was conversation-centric: short outputs, long inputs, with the cost center on the input side. With reasoning models now widespread, a model performs long chain-of-thought before producing an answer, so output token counts rise sharply and the cost center slides toward the output side.
1.2 Pricing Multipliers and the Cost-Center Shift (Illustrative)
| Billing item | Previous gen | New-gen model | Multiplier |
|---|---|---|---|
| Input | ¥4 | ¥20 | 5× |
| Cache-hit input | ¥0.4 | ¥1 | 2.5× |
| Output | ¥20 | ¥100 | 5× |
Suppose a scenario has an input-to-output token ratio of 3:1 on the previous generation. Input then accounts for about 37.5% of the bill and output about 62.5%, so the weighted bill increase is:
0.375 × 5 + 0.625 × 5 = 5.0×
For reasoning models the output ratio can reach 1:1 or higher (long chain-of-thought scenarios often produce more output than input), and the weighted increase works out to:
0.5 × 5 + 0.5 × 5 = 5.0×
If the new-generation model exposes more reasoning tiers, thinking tokens get consumed in large amounts, and the actual bill increase may exceed the listed-price increase.
1.3 From Unit-Price Comparison to Scenario Fit
| Metric | Previous gen weight | New-gen weight |
|---|---|---|
| Output price | 0.5 | 0.5 |
| Cache hit rate | 0.3 | 0.3 |
| Average output length | 0.2 | 0.2 |
- Previous-gen weighted price: 4×0.5 + 1×0.3 + 20×0.2 = ¥6.3
- New-gen weighted price: 20×0.5 + 1×0.3 + 100×0.2 = ¥30.3
- Gap of about 4.8×, close to the listed-price multiplier.
When output makes up a larger share (typical for reasoning tasks), the gap widens further.
2. Cache Hits: An Underrated Cost Lever
2.1 How Billing Works
A cache hit means the prefix tokens of a request that are identical to those of an earlier request don’t need to be recomputed and are billed at a discounted rate. The benefit depends on two things: the stability of the prefix and the reuse rate across requests.
2.2 Principles for Building a Stable Prefix
The following practices improve the cache hit rate:
# Stable prefix: system prompt + tool definitions + knowledge base (byte-for-byte identical across requests)def build_stable_prefix(system_prompt, tools_schema, knowledge_base):parts = [system_prompt.strip(),json.dumps(tools_schema, sort_keys=True, ensure_ascii=False),knowledge_base.strip()]return "\n\n".join(parts).encode("utf-8")# Put volatile content (timestamps, request IDs, random values) at the end of the promptdef build_request_prefix(stable_prefix, user_input, timestamp, request_id):suffix = f"\n[ts={timestamp}]\n[req={request_id}]\n\nUser input: {user_input}"return stable_prefix.decode("utf-8") + suffix
2.3 Multi-Turn Conversations
In multi-turn conversations, the message history naturally forms a prefix. If the input is append-only, the hit rate is high; if the history is “summarized/rewritten” or dynamically trimmed, the prefix is invalidated.
# Append-only: history only grows and is never modified, preserving prefix hitsdef build_append_only_messages(history, new_message):messages = list(history) # do not modify existing messagesmessages.append({"role": "user", "content": new_message})return messages
2.4 Common Pitfalls That Hurt the Hit Rate
- Embedding volatile fields such as timestamps, random IDs, or user IDs in the system prompt;
- Non-deterministic tool-definition ordering (e.g., unstable alphabetical or lexicographic sorting);
- Randomizing or re-ranking knowledge-base entries on every request (e.g., by similarity);
- Compressing, summarizing, or truncating history in multi-turn conversations;
- Carrying dynamic content (such as current time or real-time data) as part of the prefix.
2.5 Quantifying the Benefit
# Illustrative price for savings from cache hitsdef cache_savings(input_tokens, hit_tokens, input_price, hit_price):saved = hit_tokens * (input_price - hit_price) / 1e6 # in RMBreturn saved
Hit tokens are billed at the discounted rate, and repeated computation—and the latency it brings—is avoided.
3. From Token Pricing to Task Pricing
3.1 In Agent Scenarios, the Billing Unit Changes
In agent scenarios, users don’t compute “how much per million tokens”; they compute “how much per task.” Task cost includes multiple tool-calling rounds, retries, and the tokens consumed by failed tasks.
# Task-level cost accounting@dataclassclass TaskCost:task_id: strtotal_input_tokens: intcached_tokens: intoutput_tokens: intsuccess: booldef per_task_cost(records, input_price, hit_price, output_price):total_cost = 0.0for r in records:uncached = r.total_input_tokens - r.cached_tokenscost = (uncached * input_price+ r.cached_tokens * hit_price+ r.output_tokens * output_price) / 1e6total_cost += costsuccess_count = sum(1 for r in records if r.success)return total_cost / success_count if success_count else float("inf")
3.2 What Makes Up Task Cost
Task cost breaks down into:
- Per-round token consumption: itemized input / output / cache-hit;
- Average number of rounds: determined by task complexity;
- Retry rate: failed retries amplify cost linearly;
- Success rate: the denominator is the number of successful tasks, and failed tasks are billed too.
3.3 Cost Control Measures
- Budget circuit breakers: set a per-task token budget, max steps, and timeout;
- Fallback strategy: switch to a smaller model or return partial results when the budget is exceeded;
- Retry strategy: exponential backoff plus a maximum retry count;
- Prompt control over output: request structured output (JSON schema), explicitly cap output length.
# Budget-control exampledef run_with_budget(task, max_tokens=50000, max_steps=10):tokens_used = 0for step in range(max_steps):response = call_model(task)tokens_used += response.input_tokens + response.output_tokensif tokens_used > max_tokens:return {"status": "budget_exceeded", "partial": response.text}if response.done:return {"status": "ok", "result": response.text}return {"status": "max_steps_exceeded"}
4. Open-Weight Self-Hosting vs Managed Services
4.1 The Price Ecosystem After Weights Are Opened
Once model weights are open, third-party inference providers quickly list low-price versions, and the vendor’s official list price gradually loses its grip once the weights are commoditized. Developers need to build a replaceable inference layer:
# Inference-layer abstraction: a unified interface so switching backends doesn't affect business codeclass InferenceBackend:def chat(self, messages, **kwargs) -> str:raise NotImplementedErrorclass OpenWeightBackend(InferenceBackend):def __init__(self, endpoint, api_key):self.endpoint = endpointself.api_key = api_keydef chat(self, messages, **kwargs):# Call a compatible interfaceresp = requests.post(f"{self.endpoint}/v1/chat/completions",headers={"Authorization": f"Bearer {self.api_key}"},json={"messages": messages, **kwargs},)return resp.json()["choices"][0]["message"]["content"]class HostedBackend(InferenceBackend):def __init__(self, provider_endpoint, api_key):self.endpoint = provider_endpointself.api_key = api_keydef chat(self, messages, **kwargs):# Call the hosted-service interfaceresp = requests.post(f"{self.endpoint}/chat",headers={"Authorization": f"Bearer {self.api_key}"},json={"messages": messages, **kwargs},)return resp.json()["text"]
4.2 Selection Dimensions
| Dimension | Open-weight self-hosting | Managed service |
|---|---|---|
| Per-token cost | Low (lower still at scale) | Billed at list price |
| Ops complexity | High (needs GPU capacity and inference-engine tuning) | Low |
| Model update pace | Follows weight-release cadence | Updated by the vendor |
| Data compliance | Can be deployed privately | Depends on the provider’s compliance |
| Best fit | High concurrency, cost-sensitive, data must stay in-house | Fast validation, low ops, high compliance needs |
4.3 Self-Hosting TCO Considerations
The total cost of ownership (TCO) of self-hosting should account for:
- GPU utilization: batching, continuous batching;
- KV-cache management: paged memory management (e.g., PagedAttention-style approaches) to improve memory utilization;
- Quantization: low-precision inference such as INT8/FP8 to reduce memory footprint and boost throughput;
- Speculative decoding: a small draft model plus large-model verification to reduce average decoding steps;
- Throughput optimization: higher throughput directly amortizes per-token cost.
5. Cost-Governance Implementation Checklist
- Usage instrumentation: log input/output/cache-hit token counts per request, attributed to business line and task ID;
- Cost accounting dashboard: track per-successful-task cost, cache hit rate, and rework rate by business line / task type;
- Model routing: route simple tasks to small/cheap models and complex tasks to the flagship model;
- Semantic caching: add semantic-level caching (vector-similarity hits) for high-frequency similar requests;
- Result caching: add result-level caching for deterministic outputs (e.g., classification, extraction);
- Budgets and quotas: project-level budget alerts (e.g., alert at 80% of the daily budget) and circuit-breaker policies;
- Capacity planning: estimate GPU needs from peak QPS and average output length;
- Evaluation regression: after each model/parameter change, run a dual “cost + quality” regression to avoid cutting cost at the expense of quality.
6. Conclusion
The new model’s output pricing reaching about ¥100 per million tokens is not simply “an exit from the price war”; it is a signal that the cost structure of the inference era has changed. For developers, three takeaways:
- The cost center has shifted to output: optimization moves from compressing input to controlling output and thinking tokens;
- Cache hits are the biggest lever: a stable prefix plus a high hit rate can significantly lower the actual bill;
- Task pricing replaces token pricing: in agent scenarios, per-successful-task cost is the real decision basis.
With an ecosystem of both open weights and managed services, developers have ample room to choose: the key is to build a replaceable, observable, and controllable inference cost-governance system based on task complexity, budget constraints, and compliance requirements.
