LLM Inference Cost Breakdown: From Parameters to Pricing
This article explains why large language models with similar capabilities can have very different API prices. It covers how parameter count and active parameters affect inference, why the decode phase dominates cost, how price differs from underlying cost, and how to estimate monthly spend. It also provides a practical tiered-routing example, caching strategies, observability fields, and common pricing mistakes.
LLM Inference Cost Breakdown: From Parameters to Pricing
When people first look at a model pricing table, they often see something confusing: models with comparable capabilities can differ by several times in output price, sometimes by more than ten times. One model may be called “great value,” while another may be called “overpriced.” Both views can be reasonable because they are comparing different things. The first usually compares against overseas models in the same capability tier; the second often compares against low-price alternatives in the local market.
If the question is changed from “Is this model expensive?” to “What determines LLM pricing, and how should I choose?”, the answer becomes much clearer. This article starts with two technical facts: the relationship between parameter size and inference cost, and the difference between price and cost. It then provides a practical cost-estimation method, selection strategies, and cost-reduction practices.
1. Two technical facts behind pricing differences
1.1 Total parameters determine whether the model fits in memory; active parameters determine how fast it runs
Most current mainstream large models use a Mixture-of-Experts (MoE) architecture. Each layer contains multiple “expert” subnetworks, and a lightweight router, or gating network, selects a small number of experts for each token. The difference from a traditional dense model can be summarized as follows:
- Dense model: Every token passes through all parameters. Compute is roughly proportional to total parameter count.
- MoE model: Each token only passes through the activated experts. Compute depends on the number of active parameters, but the weights of all experts still need to be stored in GPU memory.
For example, suppose a trillion-parameter MoE model activates only one-tenth of its parameters per token. Its compute cost for one forward pass is comparable to that of a hundred-billion-parameter dense model. However, loading the full model into GPU memory still requires memory capacity for a trillion parameters.
This leads to an important conclusion:
Total parameters determine whether the model fits into memory. Active parameters determine how fast it runs.
MoE can achieve strong model capability with less compute. But whether this compute advantage translates into a service-cost advantage depends on whether hardware utilization is high enough, as discussed in Section 3.2. When comparing model costs, looking only at total parameters can overestimate inference compute. Looking only at active parameters can overlook the memory constraints imposed by the full model.
1.2 Inference cost is mainly determined by the generation phase
A model call usually consists of two internal phases:
- Prefill: The model processes all input tokens at once and generates the first output token. All tokens can be processed in a single batch with high parallelism. This phase has high arithmetic intensity and is mainly limited by compute, so GPU utilization is usually high.
- Decode: After the first token, each additional token requires the current full context to be passed through the network for another forward pass. This phase is mainly limited by GPU memory bandwidth.
The decode phase is key to understanding cost. GPU compute capacity often grows faster than memory bandwidth. During decode, generating one token requires relatively little computation but requires reading a large amount of model weights. Much of the compute capacity is idle, and if the hardware cannot be fully utilized, the effective cost per token rises.
A rough approximation of the decode bottleneck is:
Memory bandwidth required per generated token≈ active parameters per token × bytes per parameter # read weights+ context length × 2 × KV heads × head dimension × layers × bytes per element # read KV cacheTheoretical single-batch token generation rate≈ memory bandwidth ÷ memory bandwidth required per generated tokenCost per token≈ GPU-hour price ÷ (single-batch token generation rate × batch size × GPU count × utilization factor)
The utilization factor accounts for losses from communication, scheduling, memory-access alignment, and other overheads. It should be calibrated through real measurements. Illustrative values are usually between 0.4 and 0.7.
Combining this with Section 1.1, MoE decoding only needs to read a subset of expert weights for each token, so the memory bandwidth requirement is much lower than that of a dense model of the same total size. This is a structural advantage for MoE inference. However, the KV cache still grows with context length and concurrency, making it another bottleneck in long-context scenarios.
This formula also explains several common industry patterns: output prices are often higher than input prices; very long context may be priced separately; and providers continuously optimize KV cache management.
2. Price is not the same as cost: four cost categories in a bill
Inference compute is only part of the bill. From the provider’s perspective, the price of a single call includes at least four types of cost:
| Cost category | Main components | How it changes with call volume |
|---|---|---|
| R&D amortization | Pre-training and post-training investment, evaluation systems, inference engine optimization | Diluted as call volume grows; strong scale effects |
| Compute cost | GPU depreciation, datacenter and power, network and storage, utilization loss | Roughly linear with call volume; highly sensitive to utilization |
| Capability premium | Long context, chain-of-thought reasoning, reliable tool calling, SLA and compliance guarantees | Higher-capability models usually cost more to serve |
| Strategy and ecosystem | Market share, customer acquisition, ecosystem subsidies, commercial positioning | Adjustable in the short term; tends toward cost levels over the long term |
Understanding these four cost categories explains two seemingly contradictory phenomena.
First, low prices are not necessarily unsustainable. Inference cost is mainly composed of marginal compute, while training and other R&D investments account for a large share of total spending. The larger the call volume, the more thinly these fixed costs are spread, leaving more room for low prices.
Second, high prices do not necessarily mean excessive profit. More capable models with longer context consume more compute and memory during inference. In long reasoning modes, thousands of output tokens may be generated, and a large portion of the bill may come from the model’s internal “thinking process.” These costs must be covered. In addition, as shown by the utilization factor in Section 1.2, the same hardware can have very different costs across providers.
So instead of asking “Is this model overpriced?”, it is more useful to focus on three practical questions:
- Is its price reasonable relative to models with similar capabilities?
- How much of the price difference comes from inference cost, and how much comes from R&D amortization and business strategy?
- For my use case, which model has the lowest effective cost per unit of useful output?
3. Converting “price” into “cost”: an estimation model you can use directly
3.1 Understand the structure of the price sheet first
Before comparing prices, separate the billing dimensions:
- Input price and output price: Output is usually more expensive, for the reasons explained in Section 1.2.
- Cache hit price: For prefix content that is highly repetitive with previous requests, cached tokens are often billed at a significantly lower price.
- Context-length tiers: Context that exceeds the base tier may be billed at a higher tier.
- Batch pricing: Asynchronous, non-real-time batch jobs often receive dedicated discounts.
- Platform add-ons: Retrieval augmentation, code execution, model fine-tuning, and private deployment may incur additional fees.
3.2 Cost estimation formula
Based on these dimensions, a decision-oriented per-request cost estimate can be written as:
Cost per request ≈ [ input_tokens × input_price+ cached_tokens × cache_price+ uncached_tokens × input_price+ output_tokens × output_price ] × (1 + retry_rate)Effective cost per unit of output = cost per request ÷ task success rate
For monthly planning, multiply the per-request cost by the monthly request volume. To compare quality and cost together, divide the per-request cost by the task success rate to obtain the effective cost per successfully completed task.
It is also worth noting the amplification effect between unit price and utilization:
Service-side cost per token ∝ 1 / (decode batch size × hardware utilization)
The reason is that once the GPU has loaded the model weights for a request, processing one more token adds relatively little time. With higher concurrency and larger batches, reading the same weights is spread across more tokens. However, the KV cache also occupies GPU memory. Longer context leaves less memory for concurrent requests, creating a tradeoff between capacity and efficiency.
This leads to a practical rule:
For high-concurrency, short-input, structured-output tasks, lower-price models usually offer better cost efficiency. For low-frequency, high-value tasks, the higher unit price of a more capable model is often acceptable because the cost is diluted by business value.
3.3 A worked example: the same pricing assumptions, very different bills
Assume a customer-service system receives 500,000 calls per day. The average input is about 1,200 tokens, including system prompts, conversation history, and retrieved knowledge snippets. The average output is 200 tokens. The table below compares the monthly cost of four usage strategies. Prices and hit rates are illustrative assumptions.
| Scenario | Input price (CNY per 1M tokens) | Output price (CNY per 1M tokens) | Cache hit rate | Other assumptions | Monthly cost (10k CNY) |
|---|---|---|---|---|---|
| General model only | 0.5 | 2 | 0 | None | ≈ 2.1 |
| High-capability model only | 3 | 15 | 0 | None | ≈ 12.15 |
| High-capability model with cache reuse | 3 | 15 | 70% | Cache price is 1/4 of input price | ≈ 8.53 |
| Tiered routing with cache reuse | — | — | 70% | 70% of requests handled by general model, 30% escalated | ≈ 5.13 |
Reference calculations, in CNY:
- General model per request:
(1200 × 0.5 + 200 × 2) ÷ 10^6 = 0.001 - High-capability model per request:
(1200 × 3 + 200 × 15) ÷ 10^6 = 0.0066
Looking only at unit prices, the high-capability model is about 6.6 times more expensive than the general model. After considering the full scenario, the final bill is about 5.8 times higher.
These illustrative numbers show three things:
- Savings from caching and tiered routing are often larger than savings from comparing unit prices among similar models. In this example, cache reuse alone reduces the bill for the high-capability model by about 30%. Adding tiered routing reduces the bill by about 58% compared with using the high-capability model everywhere, and it is only about 1.4 times higher than using the general model everywhere, while reserving harder cases for the stronger model.
- The output-to-input ratio determines where to focus cost reduction. In scenarios where input is much larger than output, caching provides clear benefits. In scenarios with long outputs, such as long-form generation or deep reasoning, optimizing output structure may be more effective.
- Task success rate must be included in the cost formula. Each call to a high-capability model is more expensive, but if it reduces failed requests that require human fallback from 10% to 2%, the effective cost per successful task becomes much closer.
4. Model selection and cost-reduction practices
4.1 Segment by task, not by parameter count
Task segmentation is the first step in cost reduction:
- Low-risk, pattern-based tasks: Intent recognition, information extraction, content moderation, fixed-template generation. Prefer lightweight models, supplemented by caching and batch calls.
- Medium-complexity tasks: Summarization, rewriting, standard question answering. Start with a lightweight model and add quality checks.
- High-value, high-difficulty tasks: Complex reasoning, code generation, long-document analysis, multi-step agent tasks. Use a high-capability model directly, while strictly controlling context length and output length.
4.2 Tiered routing: “fast and cheap” first, then “accurate and reliable”
The idea of tiered routing is to handle requests with a lightweight model first and only escalate to a high-capability model when the result fails validation or confidence is insufficient. Example code:
def solve(query, task, client_small, client_large, threshold=0.8):# 1. Put rules first: for low-risk patterns, use the small model directly.if task.is_low_risk(query):return client_small.generate(query, max_tokens=512)# 2. Generate a draft with the small model.draft = client_small.generate(query, max_tokens=1024)confidence = task.score(draft, query) # rule check or lightweight discriminator# 3. Return the draft if quality passes; otherwise escalate to the high-capability model.if confidence >= threshold:task.log("resolved_by_small", confidence)return draftanswer = client_large.generate(query,max_tokens=2048,extra={"previous_draft": draft,"error_feedback": task.feedback(draft, query),},)task.log("escalated", confidence)return answer
Two points matter in practice:
- The scoring function (
score) is critical to routing quality. It can be implemented with a lightweight classifier, rule-based checks, or executable validation logic, but it must be validated on business data. A threshold that is too high increases escalation rate and weakens cost savings. A threshold that is too low lets low-quality answers flow into production. - When escalating, pass the draft and feedback to the high-capability model. This reduces repeated reasoning and makes the escalated call more effective.
4.3 Use context engineering to unlock cache benefits
Cache reuse depends on the “common prefix” of a request. If the system prompt, few-shot examples, or knowledge-base documents change every time, cache hits will be difficult. Practical approaches include:
- Put stable content, such as system prompts, tool descriptions, and few-shot examples, at the beginning of the request. Put variable content, such as user input and retrieval results, later.
- In scenarios such as batch document analysis or multi-turn conversations, warm the cache with the document portion first, then send multiple questions in sequence.
- Control context length. In retrieval pipelines, add reranking and deduplication, and keep only the fragments that are truly relevant.
4.4 Simplify prompts and output structures
- Remove redundant instructions and duplicate examples.
- When precise extraction is required, use structured outputs such as JSON and parse them strictly on the server side to reduce retries caused by format errors.
- Do not require long chain-of-thought output on every call. Use templates that say, “Expand steps only when necessary.”
- Break generation tasks into smaller steps such as “retrieve → extract → generate,” and give the model only the information needed for each step.
4.5 Build an observable cost loop
Cost reduction requires measurable data. At the API layer, record the following fields uniformly:
{"trace_id": "req-8f3a...", // full-chain trace ID"biz_line": "customer_service", // business line"model": "small", // model actually called"input_tokens": 1200, // input token count"cached_tokens": 840, // tokens served from cache"output_tokens": 200, // output token count"latency_ms": 1850, // end-to-end latency"success": true, // whether the call succeeded"task_result": "resolved" // whether the task was effectively completed}
With this data, teams can:
- Break down cost by business line, scenario, and model, and identify calls that are “high-cost, low-value.”
- Track cache hit rate, escalation rate, and retry rate to evaluate whether optimization measures are actually working.
- Integrate logging, monitoring, and alerting to warn about abnormal call volume and sudden cost increases.
- Replay real requests periodically to evaluate candidate models, avoiding situations where “the model changed, and the output quality changed too.”
4.6 Pricing is not static
LLM prices are changing rapidly. New model releases, inference-engine improvements, and falling hardware costs can all drive price adjustments. It is advisable to treat model services as a replaceable abstraction layer. Decouple model selection, prompt templates, and parameter configuration from business code, and preserve the ability to call multiple models in parallel. The model should be treated as an implementation detail, allowing the business to switch to a better option at any time.
5. Common mistakes
- Comparing only output unit price. In scenarios where input dominates, input price and caching strategy often matter more.
- Looking only at parameter count. The cost of MoE models depends on active parameters and utilization, not total parameter count alone.
- Ignoring tokenization differences. The same text can produce different billing token counts under different tokenizers. When comparing models, convert costs into business-oriented units such as “cost per 1,000 Chinese characters” or “cost per support ticket.”
- Ignoring long outputs and long contexts. Long reasoning models and ultra-long-context scenarios can significantly amplify output or input costs.
- Looking only at cost, not quality. Improvements in accuracy may reduce manual intervention, and this benefit should be included in total cost accounting.
- Treating pricing as a static fact. Prices are dynamic, so the architecture should leave room for switching models.
6. Conclusion
LLM pricing may look like a question of “expensive or not,” but it is fundamentally an engineering problem. It is determined by three layers:
- Technical layer: active parameters, memory bandwidth, batch size, and utilization.
- Cost layer: R&D amortization, hardware depreciation, and capability premium.
- Business layer: task value, call patterns, caching, and routing strategies.
Returning to the original question: a high price for a high-capability model may reflect higher inference cost and capability premium, but it may also include R&D amortization and market strategy. It is not necessarily “excessive profit,” and a low-price model is not necessarily a “free lunch.” The key is whether your business can actually use the capabilities you are paying for.
Instead of arguing whether a model is expensive, it is more useful to do three things:
- Use your own business data and the formulas in this article to calculate the bill clearly.
- Build an evaluation set to quantify success rates and effective cost per unit of output across different models.
- Use cache reuse, tiered routing, prompt simplification, and cost monitoring to keep control over model selection.
This is more reliable than any “cost-performance ranking.” If you are looking for a unified LLM access point, cloud model platforms such as Baidu AI Cloud Qianfan may provide a single API to access different model sizes, subject to availability in your region. You can use such a platform to run side-by-side evaluations and cost accounting for your own use cases.
