LLM API Pricing and Caching: How to Calculate True Inference Costs
LLM API prices change with model versions, and vendors often highlight both price cuts and capability gains. For developers and enterprises, the real question is how the same workload will affect the bill and whether extra capability is worth the extra spend. This article covers billing structure, caching, inference tiers, and model routing in vendor-neutral terms, using example prices only.
LLM API list prices shift with each model generation. Vendors typically announce both percentage discounts and capability improvements at launch. Developers and teams need to know what the same set of tasks will actually cost under their own traffic mix, and whether the extra capability justifies the extra spend. This article stays vendor-neutral and focuses on billing structure, prompt caching, inference tiers, and routing. All prices and scores below are illustrative examples drawn from public descriptions; always confirm official rate cards and evaluation docs before choosing a model.
Start with absolute prices, not percentage cuts
Percentage discounts can be misleading. Suppose a high-end model lists at $8 per million input tokens and $40 per million output tokens, while the previous generation was $10 and $50. Both dropped 20 percent, but that only describes history. What matters for selection is where a model sits in the current product matrix, because absolute price sets the budget ceiling.
In the examples used here, a standard flagship model costs $4 per million input tokens and $20 per million output tokens. The high-end model is therefore twice as expensive on both input and output.
| Item | High-end model (example) | Standard flagship (example) |
|---|---|---|
| Input price ($ / M tokens) | 8 | 4 |
| Output price ($ / M tokens) | 40 | 20 |
| Cache-read price ($ / M tokens) | 0.20 | 0.20 |
| Cache-read rate (as % of input price) | 2.5% | 5% |
Cache reads dominate long-session bills
Coding assistants and long-context chats repeatedly send the same prefix. Most of that prefix hits the cache and is billed at the cache-read rate, so cache-read volume is the main cost driver for long sessions.
Consider one session: 1 million input tokens total, 90 percent cache hits, 10 percent billed at the regular input rate, plus 50 k output tokens. Cache-write cost is ignored for now.
- High-end model: 0.9 × 0.20 + 0.1 × 8 + 0.05 × 40 = $2.98
- Standard flagship: 0.9 × 0.20 + 0.1 × 4 + 0.05 × 20 = $1.58
- Ratio: 2.98 / 1.58 ≈ 1.89
The 2× list-price gap shrinks to about 1.89× in this mix. Identical cache-read prices dilute the input-cost difference, yet output still dominates: the high-end model’s $2.00 output charge is roughly two-thirds of its total. The more output tokens you generate, the harder it is for caching to close the gap.
Factor in cache writes
The previous numbers omitted cache writes. Writes usually carry a separate rate when new content first enters the cache. Vendor write pricing varies widely; for illustration we assume write price = 1.25 × input price.
Total cost is:
Total = I × [h × p_cr + m × p_cw + (1 − h − m) × p_in] + O × p_out
where
- I = total input tokens in the session
- h = cache-hit rate (share of input tokens served from cache)
- m = cache-write share (share of input tokens newly written to cache)
- p_in, p_cr, p_cw = regular input, cache-read, and cache-write prices
- O, p_out = output tokens and output price
Baseline: h = 0.9, m = 0.05. High-end total ≈ $3.08, flagship ≈ $1.63, ratio still ≈ 1.89.
When hit rate falls and write share rises (h = 0.5, m = 0.4), high-end ≈ $6.90, flagship ≈ $3.50, ratio ≈ 1.97. Higher write volume inflates the bill quickly and reduces the savings from cache. The 1.89× ratio holds only while write prices stay a fixed multiple of input prices. Different write-price structures can move the ratio substantially, so results obtained under a no-write assumption should not be applied to write-heavy workloads.
Match inference tier to the default you actually use
Vendors often quote benchmark scores obtained at the highest reasoning tier. Higher tiers typically emit more reasoning tokens, increasing both latency and cost.
The source material did not specify baseline versions, evaluation protocols, or tier definitions, so those exact numbers are not used here. A more reliable process is:
- Evaluate on a representative internal task set, not public leaderboards.
- Run both models at the default tier your team actually uses.
- Record pass rate, average output tokens, latency, and cost per task.
- Raise the tier only when the extra pass-rate gain exceeds the extra cost.
Looking only at a single top-tier score is a common benchmark trap.
Capability limits on long-horizon agent tasks
Multi-agent, long-running collaboration on complex tasks is one way to probe a model’s upper bound. Public reports of such cases usually lack verifiable test conditions, so no specific figures are cited.
These workloads are characterized by repeated tool calls, accumulating context, and high output volume; the same pricing and caching formulas above still apply, but the absolute token counts (and therefore the bill) grow much larger.
