LLM API Cost Comparison: How Token Billing Really Works and How to Cut the Bill
When you build an application on top of large language models, invocation cost is often the most underestimated factor and the one that most directly determines whether a project stays sustainable. A lot of people compare models by looking at nothing more than "price per 1,000 tokens" — then discover that the same price list produces wildly different bills once the workload changes. This article stays vendor-neutral. Starting from the general billing structure, it explains how LLM API cost is actually composed, how to compare models correctly, and which levers reduce spend.
In building LLM applications, cost is easy to underestimate and hard to reverse. Here is how token-based billing is structured, why unit price comparisons mislead, and how to normalize pricing against your own workload to cut per-call cost without changing output quality.
1. Billing Is a Matrix, Not a Number
Almost every commercial LLM API is billed per token, but what actually determines the bill is a set of interlocking dimensions:
- Input token price — the cost of the context you send to the model: prompts, conversation history, retrieved passages, and so on.
- Output token price — the cost of the content the model generates. Across virtually every vendor’s price list, the output rate is substantially higher than the input rate, commonly by a factor of 3–5x, and higher still on some flagship models. This means workloads that generate long text have far more cost elasticity than workloads that merely read long text.
- Cache hit price — when a request’s prefix matches a previous request exactly, the system can reuse the already-computed intermediate state. Tokens billed this way are heavily discounted; a typical industry level is roughly 1/10 of the input price.
- Cache write price and cache lifetime (TTL) — populating the cache is not free. It is usually billed at the input price or a multiple of it, and it is differentiated by retention window (“5 minutes”, “1 hour”, and so on): the longer the retention, the higher the write cost.
- Model tier and speed tier — a single model family often ships a standard version alongside a “fast” or “low-latency” version. The latter typically costs about twice as much in exchange for lower response latency.
The key to reading this matrix is this: caching turns one-off token billing into dynamic billing with a time dimension. That is precisely the variable most often overlooked in head-to-head comparisons.
2. Why Comparing Unit Prices Is Dangerous
Consider two models, A and B. A has the higher input price, but A also offers much deeper cache-hit discounts and a longer cache lifetime. In a customer-support bot scenario built on a long system prompt with high-frequency repeated calls, A’s real cost may end up far below B’s, even though B looks cheaper on paper. Reverse the scenario — one-shot generation where every request is completely different — and the cache almost never hits, so A’s high input price is exposed in full.
The sensible approach, then, is not to compare unit prices but to normalize each price list against your actual workload.
3. A Reusable Normalization Method
The core idea is to build a weighted blended cost model that weights each of the four token prices by your business’s actual call mix:
def estimate_cost_per_request(input_tokens: int,output_tokens: int,cache_hit_tokens: int = 0,cache_write_tokens: int = 0,price_input: float = 0.0, # per million tokensprice_output: float = 0.0,price_cache_hit: float = 0.0,price_cache_write: float = 0.0,) -> float:"""Priced per million tokens; returns the cost of a single request in USD."""million = 1_000_000cost = (input_tokens * price_input +output_tokens * price_output +cache_hit_tokens * price_cache_hit +cache_write_tokens * price_cache_write) / millionreturn cost# Example: long-prompt workload with a 90% prefix cache hitcost = estimate_cost_per_request(input_tokens=1000, output_tokens=500,cache_hit_tokens=9000, cache_write_tokens=0,price_input=20.0, price_output=100.0,price_cache_hit=2.0, price_cache_write=20.0,)print(f"Cost per request: {cost:.6f} USD")
From there, multiply by your daily request volume and a monthly scaling factor to get a cost curve that closely tracks reality. The critical inputs — cache hit rate and the input-to-output token ratio — have to be measured from real telemetry logs, not guessed at.
4. The Engineering Levers That Actually Move the Bill
- Raise the cache hit rate. Put stable content (system prompt, tool definitions, knowledge-base prefix) at the very front of the request and volatile content at the tail. Most caching mechanisms rely on prefix matching: change the prefix and the cache is invalidated.
- Control output length. Since the output rate is far higher than the input rate, constrain generation length deliberately — cap word counts, require concise structured output. The payoff is often immediate.
- Set cache lifetime sensibly. If calls arrive in tight succession, choosing a longer TTL raises per-call write cost but pays off once amortized across frequent hits. Sparse call patterns invert the tradeoff.
- Pick the speed tier per workload. For latency-insensitive background batch jobs, the standard tier instead of the fast tier can save close to half the inference cost.
- Track token usage in detail. Instrument input, output, and cache-hit counts per request through ordinary logging plus a monitoring and alerting stack. Without that, optimization becomes guesswork reverse-engineered from the bill after the fact.
5. Choosing a Model: Build Your Own Comparison Table
A practical workflow: first, sample real production traffic and measure the distribution of input, output, and cache-hit tokens; then plug each candidate model’s price list into the normalization function above to compute the blended cost per 1,000 requests; finally, weigh non-price factors such as latency, context window, and reliability on top of that number. Conclusions drawn this way are far more trustworthy than an isolated price list.
6. Bottom Line
Comparing LLM API costs is fundamentally a weighted estimation problem, not a unit-price problem. Understand the four layers — input/output price gap, cache-hit discounts, cache lifetime, and speed tiers — normalize them against your real workload, and “this feels expensive” becomes “this is measurable.” Put the three engineering levers to work — cache hit rate, output length, and speed tier — and you can usually cut per-call cost significantly without changing business outcomes.
