LLM API Pricing Logic and Inference Cost Optimization: A Practical Guide
This article explains how LLM API pricing is built on three billing tiers—cache-miss input, cache-hit input, and output—and why each tier maps to a different computation stage and hardware cost. It shows why reasoning models shift the bill's center of gravity toward the output side, and how to reason about cost per task instead of per million tokens. It also covers the economics of open-weight model pricing and five practical levers for cost governance: tiered routing, cache reuse, output control, context engineering, and budget circuit breakers.
1. Introduction: Look at the Cost Structure, Not Just the Unit Price
Price changes to LLM APIs often spark more debate than capability improvements do. When a new flagship model raises the output price to several times that of the previous generation and bumps up input and cache-hit prices along with it, the most immediate question is: is this price driven by cost, or by value?
Looking at the unit-price number alone easily leads to a one-sided conclusion. A mature LLM API usually has at least three price tiers: cache-miss input, cache-hit input, and output. Each tier corresponds to a different computation path and hardware cost. If the workload is a reasoning model with long chains of thought or an autonomous agent, the output token volume often far exceeds intuitive estimates, and a unit-price increase gets significantly amplified.
This article tries to answer four questions:
- What computation cost does each of the three billing tiers correspond to?
- Why do reasoning models shift the bill’s center of gravity toward the output side?
- How do you switch from “how much per million tokens” to “how much per task”?
- Under the engineering conditions of open weights and tiered routing, how do you keep costs within a predictable range?
2. The Computation Behind Three-Tier Billing
From prefill, to autoregressive decoding, to KV cache reuse, the computation patterns of LLM inference differ greatly. The three-tier billing is a direct mapping of these differences.
1. Input tokens: the prefill stage, compute-intensive
Once the prompt enters the model, the entire content is processed in parallel in a single forward pass, generating a KV cache for each token. This stage can fully exploit the GPU’s matrix-parallel capability, so the amortized cost per token is relatively low. The input price is usually noticeably lower than the output price.
2. Output tokens: token-by-token decoding, memory-bandwidth-intensive
For each new token, the full model weights and the existing KV cache must be read once, while the matrix operation is only a “1×d” thin vector multiplication. Arithmetic intensity is very low, hardware compute sits largely idle, and the real bottleneck is memory bandwidth. On top of that, token-by-token serial generation is hard to parallelize, so the amortized cost per output token is naturally higher than for input. The output price is usually several times the input price—this is a technical rule, not arbitrary vendor pricing.
3. Cache-hit tokens: reuse of an already-computed KV cache
When the prefix of a new request exactly matches a previous one (for example, the same system prompt or the same long document), the KV cache for that content already exists and does not need to be re-prefilled. The provider’s main cost then becomes storing and retrieving the cache, so the cache-hit price is usually only about one-tenth of the input price. Refer to each platform’s billing documentation for specifics.
The relationship among the three can be summarized in the table below:
| Billing item | Corresponding stage | Main cost driver | Price characteristic |
|---|---|---|---|
| Cache-miss input | Prefill | GPU compute, parallel throughput | Relatively low |
| Cache-hit input | KV cache reuse | Cache storage and retrieval | Lowest, about one-tenth of the input price |
| Output | Token-by-token decode | Memory bandwidth, serial scheduling | Highest, usually several times the input price |
One point worth emphasizing: there is no fixed conversion formula among the three prices. With the same three-tier structure, model A might have input 20, output 100, cache-hit 1, while model B might have input 4, output 16, cache-hit 0.4—each tier keeping roughly the same ratio to the others. Comparing unit prices is of limited meaning here; what really determines cost is which tier the workload pushes the tokens into.
3. Why Reasoning Models Shift the Cost Center to Output
Traditional conversational models follow a “question–answer” pattern, and the output length is roughly predictable. Reasoning models are different: before giving the final answer, they generate a longer chain of thought. If the provider counts this content as billed output, token usage grows from hundreds to thousands or even tens of thousands.
Let’s use a simplified model to see the structure clearly. For a single request:
- Let the input token count be $T{in}$, the output token count be $T{out}$, and the cache-hit token count be $T_{hit}$;
- Let the corresponding unit prices (yuan per million tokens) be $P{in}$, $P{out}$, $P_{hit}$;
- Also let the ratio of output price to input price be $k = P{out} / P{in}$.
Without considering cache:
$C = T{in} \times P{in} \times (1 + k \cdot \tfrac{T{out}}{T{in}}) \div 10^6$
Considering cache:
$C = \left[ (T{in} - T{hit}) \cdot P{in} + T{hit} \cdot P{hit} + T{out} \cdot P_{out} \right] \div 10^6$
From the formula, when $k \cdot T{out} / T{in} \gg 1$, almost only the output term remains in the bracket. For example, with $T{in}=1000$, $T{out}=10000$, $k=5$, the output contributes about 50 times the cost of the input. At this point, no matter how much you compress the input, the impact on the total bill is negligible.
When checking the bill, developers should especially confirm two things:
- Whether the chain-of-thought content is billed separately: some platforms fold thinking content into output token billing, while others don’t bill it or list it separately.
- Whether there is a maximum thinking tier: some models ship initially with only one thinking-intensity tier, and the model expands its reasoning as much as possible, with the number of thinking tokens entirely determined by the model.
This means a reasoning model’s bill is not just “unit price × usage”—it also stacks a multiplier that the model decides autonomously and that is hard to estimate in advance.
Let’s use illustrative code to verify that “output length is the main variable.” The prices below are illustrative values:
def estimate_cost(t_in, t_out, t_hit, price_in=20, price_out=100, price_hit=1):"""Estimate the cost of a single request (in yuan). Prices are illustrative;refer to the platform's billing documentation for actual values."""cost = (max(t_in - t_hit, 0) * price_in+ t_hit * price_hit+ t_out * price_out) / 1_000_000return cost# 1000 input tokens, no cache hit; increase output length to observe cost changesfor n_out in (1000, 2000, 5000, 10000):print(f"Output {n_out:>5} tokens -> {estimate_cost(1000, n_out, 0):.2f} yuan")# 1000 output tokens, 2000-token system prompt hits the cache (3000 input tokens total)print("Cache miss:", estimate_cost(3000, 1000, 0))print("Cache hit :", estimate_cost(3000, 1000, 2000))
Output:
Output 1000 tokens -> 0.12 yuanOutput 2000 tokens -> 0.22 yuanOutput 5000 tokens -> 0.52 yuanOutput 10000 tokens -> 1.02 yuanCache miss: 0.16 yuanCache hit : 0.12 yuan
As output grows from 1,000 to 10,000, the cost rises from 0.12 yuan to 1.02 yuan, while the input part stays at only 0.02 yuan. After a cache hit, 2,000 input tokens cost only 0.002 yuan—40 times less than on a cache miss.
Also note the mismatch between “list-price increase” and “bill increase.” A 5× increase in the output unit price translates into how much of a bill increase depending on how large a share output takes in the bill:
- If the previous generation was a conversational model, output is a small share, and the bill increase gets diluted by the input price increase;
- If the previous generation was already a reasoning model, output accounts for more than 90% of the bill, and the bill increase will approach or even exceed the unit-price increase.
4. From “Per Million Tokens” to “Per Task”
The unit price is a quote; the task cost is the budget. The calling pattern of agent-type applications differs significantly from simple Q&A:
- Multiple calls per task: planning, retrieval, coding, and verification may be broken into a dozen to dozens of model calls.
- Context grows turn by turn: each step must resend all prior messages, tool returns, and retrieved snippets into the model, so total input tokens grow roughly quadratically.
- Failure retries: tool-call errors and non-compliant output formats can trigger retries, so actual token consumption is far higher than the theoretical value.
So when evaluating model cost, change the question from “how much per million tokens” to “how much to complete a typical task.” This requires first fixing the task set, fixing the output-length constraint, and fixing the retry policy, then comparing the average task cost of each model under the same conditions.
| Illustrative tier | Input price | Output price | Cost per task (20K input / 10K output tokens) |
|---|---|---|---|
| Flagship | 20 yuan | 100 yuan | 1.40 yuan |
| Open-weight, same tier | 4 yuan | 16 yuan | 0.24 yuan |
| Lightweight | 1 yuan | 4 yuan | 0.06 yuan |
The prices in the table are illustrative. For the same task, the flagship model’s cost is nearly 6 times that of the open-weight same-tier model and about 23 times that of the lightweight model. Note that these tiers usually correspond to different capability levels in practice; a lightweight model can’t necessarily complete complex tasks on its own. The true cost should be understood as “the average spend needed to complete a qualified task,” not the quote for a single call.
Per-task pricing has two easily overlooked details:
- The stronger the model, a higher unit price may actually save more: a more capable model often completes a task in fewer call rounds, so the per-task cost is not necessarily higher.
- Pricing currency: different platforms may quote in different currencies, subject to exchange-rate fluctuations. When procuring and budgeting across borders, unify the conversion basis.
On this basis, the task-level total cost can be further broken down:
$C{task} = N{calls} \times (c{avg} \times C{single} + C_{tool}) \times (1 + r)$
where:
- $N_{calls}$: the average number of calls per task;
- $c_{avg}$: the context growth factor between calls, i.e., the ratio of the average input per call to the first input;
- $C_{single}$: the cost of a single call;
- $C_{tool}$: the cost of supporting tools such as retrieval and code execution;
- $r$: the retry rate.
The engineering focus points also become clear: $N{calls}$ is improved through task-orchestration optimization, $c{avg}$ is controlled through context compression, and $r$ is reduced through structured output and validation.
5. The Economics of Open Weights and “Narrative Pricing”
If a model chooses open weights, its official list price and actual market price are very likely two separate curves.
1. The list price has limited binding force. Once the weights are public, any cloud provider with inference-deployment capability can serve on the same weights, and price competition pushes the market price toward “compute cost + reasonable profit.” This level is usually determined by hardware prices, inference-framework efficiency, batch-processing capability, and economies of scale—unrelated to the publisher’s brand premium.
2. The official list price is a value anchor. It is more a statement of the publisher’s valuation of the model’s capability than a faithful reflection of cost. Before the weights are fully open, the official list price serves as the market reference: even when a clearly lower quote appears on a third-party platform, users still unconsciously treat the official price as the “original price.”
3. Don’t judge open-weight model prices by discounts. Open-weight model prices follow a different economic logic:
- Closed-source models’ cost structures are opaque to the outside, so a “discount” is essentially a business move by the publisher;
- Open-weight models’ actual cost can be inferred from compute cost and competition;
- Therefore, to analyze the true cost of an open-weight model, look at the hardware bill and inference efficiency, not the list price itself.
4. What this means for enterprise procurement. Enterprises can handle it in two scenarios:
- Long-term, stable calls: horizontally compare hosting-service quotes on the same weights, test throughput and context-length constraints, and estimate the total cost of ownership of self-hosted inference.
- Rapid validation at launch: the official channel usually provides more stable service guarantees, which suits prototype validation; after scaling up, switch to a more cost-effective channel.
6. Five Levers for Cost Governance
Once you understand the pricing logic, cost governance has a clear direction: if you can’t change the unit price, change the usage structure.
1. Tiered routing: leave the expensive model for hard tasks
Use a lightweight model for deterministic tasks like classification, extraction, and formatting; use a prompt-based classifier or a small model to predict task difficulty and escalate complex tasks to the flagship model; and set a separate budget for each tier.
def route_and_call(text, budgets):# step 1: a lightweight model classifies intent; the classification call# itself counts against the light budgetlevel = light_model.classify(text, max_tokens=8)if budgets["light"].remaining <= 0:level = "reject" # light-tier budget exhausted, reject or downgrade directly# step 2: choose the tier based on the classification resultmodel = {"simple": light_model,"normal": standard_model,"hard": flagship_model}.get(level, standard_model)# step 3: downgrade tier by tier when budget runs low, to keep the core path availablewhile model is not None and budgets[model.tier].remaining <= 0:model = model.fallbackif model is None:raise BudgetExhausted("All tier budgets are exhausted")return model.invoke(text, max_tokens=model.output_cap)
2. Cache reuse: a stable prefix is the prerequisite for cache hits
Stable content such as the system prompt, tool definitions, and few-shot examples should sit at the very front of the request and stay word-for-word identical, with new content appended after each turn. Don’t put dynamic content such as timestamps or random IDs in the prefix, or it will break the cache.
def build_prompt(system_prompt, history, new_input):# stable prefix: keep it word-for-word identical so the cache can be hitmessages = [{"role": "system", "content": system_prompt}]messages.extend(history) # append incremental content after itmessages.append({"role": "user", "content": new_input})return messages
In addition, clear expired history promptly, so that slowly accumulated dynamic content in long conversations doesn’t erode prefix stability.
3. Output control: constrain maximum output length and thinking budget
For reasoning models, output length is the dominant cost variable. You can explicitly set an output cap in the request; if the platform offers thinking-intensity or thinking-budget parameters, adjust them dynamically based on task difficulty. Simple tasks don’t need the highest intensity, while complex tasks can be relaxed appropriately. You can also constrain the output format with prompts, e.g., “conclusion under 200 words, at most 3 reasons,” to cut redundant tokens at the source.
4. Context engineering: control $c_{avg}$
Summarize and compress long history, truncate or structurally distill tool-returned results, and control the count and length of retrieved results. In theory, after maintaining a sliding window plus rolling summaries, the task’s total cost drops from quadratic growth in the number of turns to linear growth.
| Turn | Input per turn (sliding window, last 2 turns kept) | Input per turn (full retention) |
|---|---|---|
| 1 | 1,000 | 1,000 |
| 5 | 1,000 | 5,000 |
| 10 | 1,000 | 10,000 |
| 20 | 1,000 | 20,000 |
| Cumulative input over 20 turns | 20,000 | 110,000 |
At the flagship illustrative unit price of 20 yuan per million tokens, a 20-turn task costs 2.20 yuan in input under full retention, while a sliding window needs only 0.40 yuan—about 18% of the former. If the task needs cross-turn memory, you can have the model generate a rolling summary; the summary length is far smaller than the original history, preserving key conclusions while controlling cost.
5. Budget circuit breakers and usage observability
Set daily/monthly budgets for each model tier and each business line, and check the remaining quota before each request. When a threshold is exceeded, automatically downgrade to a lower tier or reject the request to avoid abnormal loops driving up the bill. Usage monitoring should cover at least three dimensions:
- Input, output, and cache-hit token counts;
- Cache hit rate;
- Average cost per task.
On platforms such as Baidu AI Cloud, you can use log services and monitoring/alerting capabilities to aggregate token consumption by model and by business line and set threshold alerts, so cost anomalies are detected the same day they occur.
7. Conclusion
From prefill and decoding to KV cache reuse, the technical differences behind three-tier pricing are clearly traceable; reasoning models push the bill’s center of gravity toward the output side, making “the more it thinks, the more you pay”; and open weights create a clear separation between the official list price and the market transaction price.
For enterprise users, more worth asking than “is it expensive” is: can this model do the work on my task with fewer calls, shorter context, and a higher cache hit rate? By making tiered routing, cache reuse, output control, context engineering, and budget circuit breakers into engineering defaults, the impact of price fluctuations will be greatly smoothed, and the space for model selection will be far more comfortable.
