LLM API Cost Evaluation: Why the Unit Price Is Not the Whole Story
This article explains why evaluating LLM API costs requires more than comparing advertised prices per million tokens. It covers token accounting, tokenizer differences, prefix caching, time-of-day pricing, and the impact of multi-turn retries. It also provides a reusable estimation method and practical engineering steps for reducing real inference costs.
1. The Gap Between Pricing Labels and the Real Bill
Whenever a flagship LLM API is launched, the first comparison is often a simple price label: input tokens per million and output tokens per million. This comparison is intuitive and easy to share, but it often hides one fact: unit price is just one factor in the bill, not the bill itself.
The cost of a real inference workload depends on at least several variables:
- token distribution of input and output;
- how differently tokenizers segment text;
- discounts from caching and reused computation;
- time-of-day or tiered pricing;
- extra calls needed to reach a target accuracy level.
If you compare only unit prices and ignore the latter four factors, the conclusion that one service is “cheaper” may be wrong. Understanding this is essential for making better technical and cost decisions.
2. The Basic Model of Token-Based Billing
Most mainstream LLM APIs use a dual-rate structure: one price for input tokens and one for output tokens. Output tokens are usually more expensive than input tokens. This is reasonable from an engineering perspective: generation uses autoregressive decoding token by token and is difficult to parallelize, while the input phase, or prefill, can be highly parallelized. Their computational costs are inherently different.
Suppose a call has N_in input tokens and N_out output tokens, with input price p_in and output price p_out per 1M tokens. The cost of one call can be written as:
cost = (N_in / 1e6) * p_in + (N_out / 1e6) * p_out
An often-overlooked point is that input cost can dominate in long-context scenarios. When models are used for document summarization, codebase understanding, or multi-turn conversations, input tokens can easily reach tens of thousands or even hundreds of thousands. In these cases, small differences in input token prices can become significant absolute cost differences. Cost evaluation should therefore use the actual input/output token ratio of your own workload, not conclusions based solely on advertised unit prices.
3. Time-of-Day and Tiered Pricing: How Cost Changes Over Time
Some services introduce dynamic pricing across business periods, such as peak, off-peak, and low-traffic tiers. This means that the same request may have a different cost depending on when it is sent.
When comparing services, if one uses fixed unit prices and another uses time-based variable prices, comparing advertised prices directly can be misleading. A more rigorous approach is:
- Determine the call-time distribution of your workload: is it evenly spread, or concentrated during peak hours?
- Use a weighted average rate, not peak or trough rates, for comparison.
- For tasks that can be delayed, such as offline batch processing or log analysis, schedule them during low-traffic periods to capture cost benefits.
This creates a typical engineering opportunity: decouple real-time paths from offline paths. Real-time requests from users are latency-sensitive and must respond immediately. Offline tasks such as data cleaning, batch generation, and evaluation runs are less sensitive to latency and can be queued for lower-cost periods. Reasonable scheduling can reduce overall spending while barely affecting user experience.
4. How Caching Reshapes the Real Cost
Modern inference services commonly support prefix caching: if multiple requests share the same input prefix, such as the same system prompt, the same few-shot examples, or the same long document context, the server can reuse already-computed intermediate states, such as the KV cache. This can significantly reduce billing for cached portions, sometimes with discounts that approach zero.
This means that cache hit rate is a heavily weighted factor in the cost model. Given the same advertised input price, if service A has a friendly cache strategy and can achieve a hit rate above 90%, while service B has weaker cache support, their actual input costs may differ by several times.
Engineers should rethink architecture around this:
- Stable prefix first: Put unchanged system prompts, tool definitions, and few-shot examples at the beginning, and put variable parts, such as user input and retrieved results, at the end to maximize prefix reuse.
- Structured batching: Keep requests with the same template prefix-consistent and avoid breaking cache reuse with small dynamic concatenations.
- Context ordering: Extract repeated public context as a stable prefix, and put dynamic content, such as timestamps, random IDs, and retrieved snippets, after it.
Here is code illustrating the prefix-stability principle:
def build_prompt(system_rules, examples, retrieved_docs, user_query):# Stable prefix: rules and examples rarely change and can hit the cache long-termstable_prefix = f"[SYSTEM]\n{system_rules}\n\n[EXAMPLES]\n{examples}\n"# Dynamic suffix: retrieved snippets and user questions differ each time, so place them at the enddynamic_suffix = f"\n[CONTEXT]\n{retrieved_docs}\n[QUESTION]\n{user_query}"return stable_prefix + dynamic_suffix
5. Tokenizer Differences: Same Text, Different Token Counts
A frequently underestimated variable is the tokenizer. Different models use different tokenization algorithms and vocabularies. The same text may produce different token counts under different tokenizers, sometimes differing by 10% to 30% or more. For Chinese, this difference can be especially pronounced: some tokenizers split Chinese more finely, causing token inflation.
This leads to a counterintuitive conclusion: a service with a lower advertised unit price may not actually be cheaper. Suppose service X has an input price that is 90% of service Y, but X’s tokenizer produces 20% more tokens for the same Chinese text than Y. The real cost of X may then be 1.08 times that of Y.
A fair cost comparison must therefore be based on token counts measured with each service’s own tokenizer on the same real business texts. Token counts should be obtained through the tokenizer or count API officially provided by each service, not through rough approximations.
The article also mentions using a “tokenizer probe” to identify model attribution. Its principle relies on the distinguishability of tokenization results from different models. This also supports the view that tokenizers are an important “fingerprint” of models and an unavoidable part of cost accounting.
6. Multi-Turn Retries: The Trade-Off Between Accuracy and Cost
For production systems, the goal is never simply “the cheapest single call,” but “reaching target accuracy at an acceptable cost.” This introduces strategies such as self-consistency, retry, and multi-candidate voting.
The logic is: for uncertain questions, let the model generate multiple candidate answers or sample independently multiple times, then use voting, best-of selection, or consistency checking to obtain a more reliable conclusion. This can significantly improve accuracy, but the cost is multiplied calls. In essence, it is trading cost for quality.
Therefore, cost evaluation must not look only at “how much one call costs.” It should calculate the expected total number of calls required to reach target accuracy:
expected_cost = single_call_cost * expected_calls_to_reach_target
Some services have strong cache optimization and scheduling. In multi-turn scenarios, repeated prefixes are reused sufficiently, so the actual multi-turn cost increase is limited. Others recompute the full context on every turn, making multi-turn cost amplification very significant. This is a common reason why “similar advertised unit prices” can hide real differences.
7. A Reusable Methodology for Comparing API Costs
Based on the analysis above, a rigorous cost estimation process can be summarized:
- Sample real business text: Collect input/output samples that cover typical scenarios rather than toy data.
- Measure token distributions with each tokenizer: Obtain the real
N_inandN_outfor each service on your corpus. - Estimate cache reuse ratios: Separate stable prefixes from dynamic suffixes and estimate expected hit rates.
- Account for time: Calculate weighted rates for time-of-day pricing.
- Account for retry rounds: Estimate expected calls based on target accuracy.
- Calculate total cost with one consistent formula: Compare services using the same method, not isolated unit prices.
The following code turns this method into a reusable estimation function:
def estimate_monthly_cost(calls_per_day,avg_in_tokens, # measured with the actual tokenizeravg_out_tokens,p_in, p_out, # price per 1M tokens, use weighted ratescache_hit_ratio=0.0, # share of input prefix that can reuse cachecache_discount=0.1, # discount factor for cached portionexpected_calls=1.0, # expected calls needed to reach target accuracy):days = 30total_calls = calls_per_day * days * expected_calls# Input cost: apply discount to the cached portioneffective_in = avg_in_tokens * ((1 - cache_hit_ratio) + cache_hit_ratio * cache_discount)in_cost = total_calls * (effective_in / 1e6) * p_inout_cost = total_calls * (avg_out_tokens / 1e6) * p_outreturn in_cost + out_cost
Only when different services are plugged into the same function and evaluated on the same corpus are the results comparable.
8. Engineering Practices for Reducing Inference Cost
With this methodology, engineering teams can systematically reduce inference spending through the following practices:
- Prompt engineering: Remove redundant instructions and duplicate examples to reduce input tokens directly.
- Context trimming and retrieval augmentation: Instead of stuffing an entire long document into context, use retrieval to inject only the most relevant snippets.
- Tiered routing: Send simple tasks to smaller or more economical models and route complex tasks to flagship models, avoiding overkill.
- Batching and asynchronous processing: Aggregate compatible requests to improve cache reuse and smooth call peaks.
- Output length constraints: Use clear format requirements and length limits to avoid long and redundant generations.
- Observability: Use monitoring, alerts, logging, and usage metering to track the real cost of each business category and detect abnormal growth early.
These methods often produce meaningful cost improvements without changing business logic, and their effects can be measured and regression-tested.
9. Conclusion
“How much per million tokens?” is a metric that is easy to communicate, but it is far from the whole picture of cost decisions. Cache hit rate, tokenizer differences, time-of-day pricing, and multi-turn calls are four hidden variables that together determine your final bill. The professional approach is to return to your own real business corpus, use one consistent formula, plug in these variables one by one, and then combine the results with reasonable architecture design and scheduling optimization.
Upgrading cost evaluation from “comparing unit prices” to “comparing real total costs” makes technical selection more rational and helps ensure that every inference call delivers better cost-to-business value.
