LLM Inference Cost Modeling: How to Evaluate and Compare Blended Per-Token Prices
Choosing a large language model API by comparing a single "input price" or "output price" is misleading, because mainstream providers bill a request in three separate tiers: cache-hit reads, cache-miss input, and generated output, and the gaps between those tiers can span several-fold to tens of times across vendors and model tiers. This article breaks down the three-tier pricing structure, explains why prompt caching drives most of the cost difference, and shows how to collapse the three prices into one blended per-token price using weights sampled from your own traffic. It also covers how to build list-price and effective-price rankings, how to handle peak/off-peak pricing and long-context tiers, and includes a Python script that recalculates the ranking whenever prices or weights change, so model selection moves from guesswork to calculation.
LLM Inference Cost Modeling: Evaluating and Comparing Blended Per-Token Prices
1. Why the “lowest unit price” is usually a trap
When choosing an LLM service, many developers compare a single “input price” or “output price” and pick whichever is lower. Real bills rarely match that intuition. Mainstream LLM APIs are no longer priced with one number for everything: a call is split into three separately billed segments — cached reads, uncached input, and generated output. The gap between these three prices can differ by several times, or even tens of times, across vendors and model tiers.
Unless all three prices, cache hit rates, and discounts are put on a common footing, any so-called “cost-effectiveness comparison” will be distorted. To make a sound selection, you first need to lay out the billing structure, then compress it into a metric that can be compared side by side.
2. Breaking down the three-tier pricing structure
From a billing perspective, the tokens consumed by one inference request usually fall into three categories:
- Input (cache miss): prompt tokens in this request that were not served from cache — content appearing for the “first time.” This is normally the most expensive tier.
- Cached read (cache hit): the request prefix matched content already cached on the server side (system prompts, few-shot examples, multi-turn conversation history). This tier is billed at a “read” rate, typically the cheapest of the three.
- Output: tokens actually generated by the model. Price is generally at the same level as input or higher, and output cannot be cached.
All three prices are usually quoted in the same unit — per million tokens — so they can be added directly. One point deserves emphasis: cheap cached reads only apply when the content can be reused reliably. If every request sends full, new content, the cache will almost never hit, and the “read” price advantage disappears entirely.
3. Prompt caching: the root of the cost difference
Cached reads are cheap because the server reuses the intermediate computation results (KV cache) derived from processing that input earlier, rather than recomputing the same prefix. This is what ties hit rate and cost together so tightly. Three typical scenarios follow:
- High hit-rate workloads: a fixed, long system prompt plus multi-turn long conversations, or stable few-shot templates. The system prefix is hit on nearly every call, cached reads make up a large share, and overall cost drops substantially.
- Low hit-rate workloads: one-off summarization of a long document, or retrieval snippets assembled differently every time. The prefix is almost never reused, input dominates, and the caching advantage effectively vanishes.
- Output-heavy workloads: code generation, long-form writing, and similar tasks where output tokens make up a large share. Since output cannot be cached, the output price becomes the deciding factor in total cost.
Conclusion: talking about price levels apart from workload structure is meaningless. For the same two models, the cost ranking can be completely reversed under different businesses.
4. Blended price: collapsing three prices into one metric
For a side-by-side comparison, introduce a weighted blended price: first assume how large a share each of the three token types takes in your workload, then take a weighted average of the three unit prices using those shares. The result is a “real average cost per million tokens.”
A common default set of weights is “read 0.8 / input 0.1 / output 0.1,” which assumes cached reads make up about 80% of tokens and uncached input and output take 10% each:
blended_price = 0.8 × read_price + 0.1 × input_price + 0.1 × output_priceunit: per million tokens
Example (numbers are only to demonstrate the calculation): a model with a read price of 0.1, an input price of 1.0, and an output price of 2.0:
blended_price = 0.8 × 0.1 + 0.1 × 1.0 + 0.1 × 2.0= 0.08 + 0.10 + 0.20 = 0.38 (per million tokens)
This merges “three prices + workload structure” into a single number that can be sorted directly.
5. Setting the weights: making the metric match your real workload
0.8/0.1/0.1 is only a default assumption, not a universal truth. The right approach is to sample your logs, measure the real share of the three token types in your business, and then plug those shares into the formula. If the weights are wrong, the ranking loses its value as a reference.
A simple way to measure:
- Sample the last N calls and accumulate the actual billed tokens for each segment.
- Divide each by the total token count to get the read share r, input share i, and output share o (r + i + o ≈ 1).
- Substitute (r, i, o) for the weights in the formula.
For the same business, applying one identical set of weights to every candidate model produces a blended-price ranking that is meaningful for decisions.
6. Building a cost ranking in practice
Engineering the approach above reliably produces two kinds of rankings:
- List-price ranking: compute the blended price directly from official reference prices. This reflects the “theoretical upper bound” of cost and is suitable for setting budget guardrails.
- Effective-price ranking: start from list prices, add the discounts you actually receive (tiered volume discounts, cache discounts, peak/off-peak differences), and recompute. This reflects “real spend.”
Two additional dimensions need attention:
- Peak vs. off-peak hours: some models are priced differently during peak hours than at other times, so the same model may have two blended prices. List them separately and shift workloads off-peak where scheduling allows.
- Long-context tiers: some models price by context length (for example ≤256K / >256K), and long requests jump to a more expensive tier. When building a ranking, pick the tier that matches your real request lengths.
For models whose prices are marked “unverified” or “not published,” exclude them from the ranking explicitly and state the reason, so that guessed numbers do not pollute the ordering.
7. Codifying the calculation
Turning the formula and ranking logic into a script lets you recompute in one command after prices or weights change, and avoids errors from manual table lookups:
from dataclasses import dataclass@dataclassclass Model:name: strread: float # cached-read priceinput: float # cache-miss input priceoutput: float # output price# unit: per million tokensdef blended_price(m: Model, r=0.8, i=0.1, o=0.1) -> float:"""Compute the blended price using workload weights; r/i/o should come from real sampling"""assert abs(r + i + o - 1.0) < 1e-6, "weights must sum to 1"return r * m.read + i * m.input + o * m.outputdef rank(models, **weights):"""Return models sorted by blended price, lowest first"""return sorted(((blended_price(m, **weights), m.name) for m in models),key=lambda x: x[0],)# Numbers are for demonstration only; use official list prices in practicecandidates = [Model("Model A", read=0.1, input=1.0, output=2.0),Model("Model B", read=0.3, input=0.6, output=4.0),]for idx, (price, name) in enumerate(rank(candidates), 1):print(f"{idx}. {name}: {price:.4f} per 1M tokens")
If your business is high-hit-rate, long multi-turn conversations, use rank(models, r=0.9, i=0.05, o=0.05). For output-heavy generation tasks, raise o — the ranking will shift accordingly.
8. Common pitfalls and reminders
- Don’t compare on a single price: comparing only output prices ignores the read-price advantage; comparing only read prices ignores output overhead.
- Don’t treat default weights as a conclusion: weights must come from your own sampling, or the ranking is only a rough reference.
- Don’t ignore discounts and peak/off-peak pricing: the effective price and the list price can differ by an order of magnitude. Real settlement is based on the effective price.
- Don’t assume cache hits are free money: hits depend on a stable prefix structure. Changes to your business reset the hit rate, so cost needs to be rechecked continuously.
The same methodology applies to evaluating model services from any cloud provider, such as Baidu AI Cloud. The key is to build a cost model with three steps: tiered pricing, weights from real workloads, and a consistent basis for ranking. Once those three steps are solid, you can quickly arrive at a trustworthy, reusable cost comparison — no matter how the candidate set changes or prices are adjusted — and move model selection from “guess by intuition” to “calculate from data.”
