LLM API Cost Calculation: Breaking Down Pricing Dimensions and Optimization Practices
This article explains why comparing LLM API prices by unit rate alone almost always leads to the wrong conclusion, then walks through the full set of billing dimensions that determine what you actually pay: input/output price asymmetry, prompt caching (read, write, and storage), time-of-day pricing, batch discounts, and discount exclusivity. It provides a computable cost function, the caching payback threshold and the hit-rate red line for deciding between "online + cache" and "batch," three unit-economics metrics, seven practical cost-reduction techniques, and guidance on observability and budget governance. Readers will learn to turn pricing tables into a cost model and to pick the cheapest option for their specific workload rather than the nominally cheapest one.
1. Why Comparing Unit Prices Alone Usually Reaches the Wrong Conclusion
The most common move in technology selection is to drop several vendors’ input and output unit prices into a spreadsheet, sort them, and declare the “cheapest” one. That conclusion is almost always wrong, for three reasons.
First, pricing is asymmetric. In the same table, the output price is usually significantly higher than the input price, with a ratio commonly in the 2× to 5× range across the industry. This means a model that is “30% cheaper on input but 50% more expensive on output” can end up costing more overall in long-text generation scenarios. A true cost function must model input and output separately.
Second, the interaction between discounts determines the real price. Five dimensions—cache-hit price, cache-write fee, cache-storage fee, time-of-day discount, and batch discount—do not simply add up independently. They carry mutual-exclusion and stacking rules. If you ignore those interactions, you will mistake the “nominal lowest price” for the “lowest amount actually paid.”
Third, the real workload decides which layer carries the most weight. For a service with a fixed system prompt and a highly repetitive RAG prefix, the cache hit rate can exceed 80%; for a service where every request has fresh context and long outputs, caching yields almost nothing, and the batch discount may be the only real lever. On the same price list, the optimal choice is completely different under the two workloads.
So the correct approach is: first abstract the billing dimensions into a cost function, then plug in the characteristics of your real workload and solve. This article follows that order.
2. Pricing Dimensions in Full: Five Questions to Ask Before Signing
2.1 Pricing Units and Input/Output Asymmetry
The mainstream pricing unit is “currency / million tokens.” Pay special attention to three points:
- Tokens are not word counts. Chinese and English differ in tokenization density: in Chinese, one character typically corresponds to roughly 1 to 2 tokens, and mixed code and symbols inflate the token count further. For capacity planning you must measure against real samples, not estimate from character counts.
- Input includes the full history. In multi-turn conversations, each turn must resend the complete prior context, so input tokens grow roughly linearly with the number of turns, while cumulative consumption grows roughly quadratically (n turns cost about n(n+1)/2 times a single turn’s length). This is the primary reason bills balloon over long-running use.
- Output often includes the reasoning process. Some models return intermediate reasoning content and count it as output tokens, so “it only output 200 characters” can actually be billed as thousands of tokens. During selection, confirm whether reasoning content is billed and whether it can be disabled.
2.2 Caching: Hit, Write, and Storage Billed Independently
Context caching (prompt caching) is the most important cost-reduction lever of recent years, but it consists of at least three independently billed items:
| Billing item | Billed on | Typical scale (illustrative multiplier, not actual price) |
|---|---|---|
| Cache read / hit | Input tokens served from cache | 5%–20% of the cache-miss input price |
| Cache write | Prefix tokens that create a new cache | Equal to or about 2× the input price |
| Cache storage | Tokens held in cache × duration | On the order of ~0.3% of the input price per million tokens per hour |
Three engineering points:
- A cache write is typically paid once. After that, each hit is read at the discounted price, so the write fee is a one-time investment.
- Caches have duration tiers (commonly a minute-level and an hour-level tier). The longer tier costs more per unit and suits low-frequency requests with very large prefixes.
- A cache hit requires the prefix to be byte-for-byte identical. A change anywhere—system prompt, tool definitions, few-shot examples, timestamps, or randomly ordered retrieval snippets—invalidates the entire prefix and triggers a new write charge.
2.3 Time-of-Day Pricing
Some platforms apply differentiated pricing for peak and off-peak hours. The peak tier can reach 2× the off-peak tier. A typical definition of peak is fixed morning and afternoon blocks on business days, with weekends and public holidays counting as off-peak all day. Two details that are easy to get wrong:
- Billing time is typically based on request completion, not request start. A long task submitted before 18:00 may still be billed at the tier in effect when it was submitted if it completes after the peak period ends—rules vary by platform, so confirm them before integrating.
- Batch tasks should be optimized jointly with the time-of-day strategy. Scheduling offline tasks into off-peak hours already earns one discount; stacking batch discounts on top is what delivers the full benefit.
2.4 Batch / Asynchronous Discounts
A batch interface for offline, non-real-time tasks is another major cost-reduction path. Industry discounts are commonly 50% or 60% of the original price, and the trade-off is that response time moves from seconds to minutes or even hours, with no SLA guarantee. Suitable workloads include offline data labeling, batch log summarization, content-moderation rescans, batch translation, and evaluation-set scoring.
2.5 Discount Exclusivity: The Easiest Thing to Miss
Most platforms state explicitly in their rules that the batch discount and the context-caching discount cannot both apply, usually computed as “the higher of the two actual amounts” or “only one of them.” This creates a clear dividing line:
When the cache hit rate is high enough, the effective unit price of “online call + cache” can be lower than that of “batch call + no cache”—and giving up real-time delivery to run a batch actually costs more.
3. Writing the Price List as a Computable Cost Function
Build the model with abstract symbols (not tied to any specific product):
Let:P_in = input unit price (cache miss)r = cache-hit price / P_in (typically 0.05 ~ 0.2)P_out = output unit pricem = P_out / P_in (typically 2.5 ~ 5)c = prefix cache hit rate (0~1)d = time-of-day factor (off-peak 1.0, peak typically 2.0)b = batch discount factor (typically 0.5 or 0.6)Unit: cost per 1M input tokens (with P_in normalized to 1)Online, no cache C1 = 1 + m * (output_tokens / input_tokens)Online, with cache C2 = (1 - c) + c * r + m * (output_tokens / input_tokens)Batch (not stacked with cache) C3 = b * [1 + m * (output_tokens / input_tokens)]All three are multiplied by the time-of-day factor d
A minimal implementation (pseudocode that can be ported to any language):
def cost_per_1m_input(p_in=1.0, m=4.0, r=0.1, c=0.0,out_in_ratio=0.25, d=1.0, b=1.0,use_cache=False, use_batch=False):"""Return the cost per 1M input tokens (in units of p_in)."""if use_batch: # batch and cache discounts are mutually exclusiveunit = b * (1.0 + m * out_in_ratio)elif use_cache:unit = (1.0 - c) + c * r + m * out_in_ratioelse:unit = 1.0 + m * out_in_ratioreturn d * unit * p_in# Criterion: is online + cache cheaper than batch?# (1 - c) + c*r < b => c > (1 - b) / (1 - r)def cache_beats_batch(b=0.5, r=0.1):return (1.0 - b) / (1.0 - r) # hit-rate red line
Plugging in common parameters: at b = 0.5, r = 0.1, the red line is about 56%; at b = 0.6, it is about 45%; and with a deeper cache discount (r = 0.05), the red line drops further. This red line is the most actionable conclusion in this article: batch is not inherently cheaper—measure the hit rate first, then decide your scheduling strategy.
4. The Economics of Caching: Payback Threshold and Hit-Rate Red Line
The write payback threshold is very low. Suppose the write price is 1× to 2× the input price and each hit saves about (1 − r) times the input price relative to a miss; then payback requires roughly 1 to 2 hits. In other words, whether caching pays off depends almost not on the write cost, but entirely on whether the prefix can be reused reliably.
Storage cost is usually negligible but accumulates. Cache storage is billed by tokens × duration, at a scale far below the input price, but the combination of a long tier + large prefix + low hit rate produces a “pay rent but collect none” loss. Set the minimum necessary TTL for each cache, and proactively disable any cache whose hit rate falls below a threshold.
Engineering measures to raise the hit rate (ordered by payoff):
- Fix the prefix order: system prompt → tool definitions → a few examples → user content, with stable content hashes for the first three.
- Strip dynamic content: move timestamps, request IDs, and randomly ordered results after the prefix, or sort retrieval results stably by ID.
- Control context drift: use an incremental-prefix design in multi-turn conversations, and avoid rewriting the full history each turn, which invalidates the whole prefix.
- Segment caching: split large documents into stable blocks; the finer the hit granularity, the smaller the loss from a partial invalidation.
- Run change audits during canary releases: any change to a prompt or tool definition can trigger a full cache cold start, so evaluate its cost impact before release.
5. From Unit Price to Unit Economics: Three Higher-Level Metrics
Technical metrics must be tied to business metrics. Establish at least three layers of measurement:
- Cost per 1K calls = average tokens per call × unit price × 1000. Used for capacity planning and load-test comparison.
- Cost per successful task = total cost / number of successful tasks. Retry, timeout, and empty returns must all be counted to reflect the true cost of quality. If a model is 30% cheaper but has a high failure rate that forces 1.6 retries on average, its actual cost is higher.
- Cost per active user / cost per order (revenue). This is the only common language with business stakeholders, and the anchor for setting budget-alert thresholds.
Load-normalized comparison: candidate models should not be compared directly by “per million tokens,” but by “the average total cost to complete the same set of real tasks.” Build an evaluation set that covers high-frequency scenarios (200 to 500 real, desensitized samples is a good target), record each candidate’s average input tokens, average output tokens, average retries, and cache hit rate, then plug them into the cost function. Only that produces comparable data.
6. Seven Directly Actionable Cost-Reduction Techniques
- Tiered routing: use a lightweight classifier (rules, regex, or a tiny model) to judge request difficulty; route simple requests to lower-tier models and complex ones to the flagship model. The expected cost is
E[C] = p·C_small + (1−p)·C_large, minus the rework cost from misrouting. Validate routing accuracy via offline replay before going live. - Context compression and trimming: summarize history turns instead of retaining all of them; rerank and truncate retrieval snippets; deduplicate and streamline the system prompt. Every 30% reduction in context lowers input-side cost roughly proportionally.
- Output-length governance: set an explicit maximum generation length, use structured-output constraints, and implement early-termination logic for streamable scenarios. Output prices are several times input prices, and the output side is the most easily overlooked source of waste.
- Off-peak scheduling + batch mix: send real-time requests through the online channel and offline tasks through the batch channel during off-peak hours, with both sharing the same cached prefix to maximize hits.
- Reduce invalid calls: deduplicate requests, use idempotent caching (return historical results for identical inputs), limit user resends, and implement exponential backoff instead of blind retries on timeouts.
- Model-degradation fallback: set timeouts and fallback paths for downstream dependencies, so upstream congestion does not cause requests to pile up and then retry in a concentrated burst, forming “cost spikes.”
- Contract-based cost budgets: write token budgets into the interface contract—for example, per-request input and output caps—and reject or truncate when exceeded, preventing abnormal calls (such as accidentally passing an entire file) from generating huge bills.
7. Cost Observability and Budget Governance
Without observability there is no optimization. A minimum viable observability setup has three parts:
- Dimensioned logging: record for every call the model identifier, business line, environment, tenant, scenario tag, input/output tokens, cache-hit tokens, tier factor, and failure reason.
- Dashboards and alerts: the core metrics are daily call volume, average token consumption, cache hit rate, and daily cost with week-over-week change, plus daily budget thresholds and abnormal growth-rate alerts (for example, a 50% day-over-day spike).
- Per-request cost accounting: maintain pricing parameters consistent with the platform on the application side, to answer business questions like “how much did this task cost.” Note that the platform bill is the authoritative source; application-side accounting is for attribution analysis, not reconciliation.
It is worth folding cost-control actions into the release process: prompt changes, context-template changes, and model switches should each come with a cost-impact assessment, running a before/after comparison on the same evaluation set.
8. Five Common Misconceptions
| Misconception | Reality |
|---|---|
| Comparing input prices only | Output prices are typically 2–5× input, so conclusions reverse in long-output scenarios |
| Assuming caching always saves money | When the prefix is unstable, the hit rate approaches 0 and you pay only write and storage fees |
| Assuming batch is always cheaper | With a high hit rate, online + cache can win; compare against the hit-rate red line |
| Ignoring time-of-day pricing | The 2× peak premium stacks up, and monthly cost differences can reach tens of percent |
| Estimating tokens from character counts | Chinese, English, and code tokenize at very different densities, so you must measure |
9. Conclusion
Cost control for LLM APIs is essentially about turning the static question of “which vendor to pick” into the dynamic question of “what structure, at what time, and at what reuse rate to call.” The price list only provides the parameters; the cost function determines the conclusion. A recommended practice path is: first abstract the cost model → collect workload characteristics with a real evaluation set → compute the trade-off between the caching red line and the batch discount → roll out tiered routing and context governance → establish dimensioned observability and budget alerts → keep recomputing on the same basis. Once this loop is in place, a vendor price change or a model iteration is no longer a risk you passively absorb, but an opportunity that can be automatically evaluated and absorbed.
