LLM API Pricing: Why Unchanged Prices Don't Mean Unchanged Bills — Calculate Cost per Task, Not per Token
When a new-generation LLM ships its API at the same input, output, and cache-hit prices as its predecessor, the price table stays flat, but the bill may not. This article looks past unit pricing to explain the three price dimensions that matter, why output tokens dominate cost, why cache hits depend on prefix stability, and why composite benchmark scores should only build a candidate pool. It shows how per-task token use, retries, and routing determine actual spend, then offers six cost-reduction techniques and a practical checklist for teams, from evaluation-set testing to observability.
A recent release of an open-weight large language model introduced a new generation with an API. What developers have discussed most is not the jump in capability itself but the pricing: the model moved up a tier in ability, yet the per-token prices for input, output, and cache hits stayed the same as in the previous generation.
A move like this — “more capability, same price” — usually means the provider has made real gains in inference efficiency, architecture, or hardware utilization. For developers, though, the price table isn’t what matters. The bill is. Unit price is only one variable. What actually determines spending also includes token consumption per task, cache hit rate, the number of failed retries, and model routing strategy.
This article isn’t about the merits of any one product. It treats the situation as a general engineering problem: when several models sit in the same intelligence tier, how do you complete production tasks reliably at the lowest cost?
1. Understand the Three Price Dimensions
Priced per million tokens, the new generation comes in at roughly 1.40 for input, 4.40 for output, and 0.26 for cache hits. Viewed together, the three numbers lead to two conclusions.
First, output costs more than three times what input costs. This is the general pattern in current mainstream LLM pricing. During autoregressive decoding, every output token has to pass through the full model and the prefix KV cache, so compute utilization is inherently lower than during prefill. This dictates where to optimize: compressing context pays off far less than compressing output. Cutting a few thousand tokens of conversation history usually does less good than trimming half of a verbose reasoning trace or a repeated code block.
Second, a cache hit costs about 18.6% of the input price, an effective discount of over 80%. Cache pricing isn’t a marketing concept; it prices the underlying mechanism. When a request’s prefix exactly matches content already in the cache, the system can reuse the cached KV state for that prefix and skip the redundant computation. So cache hits are essentially about prefix stability, not price. That also explains why cache hit rates stay low in so many projects. When a system prompt embeds timestamps, random IDs, or per-user variables, the prefix differs every time and the cache never applies.
2. Composite Intelligence Scores: Useful, but Not Enough to Choose By
A composite score that weights benchmarks such as math, coding, instruction following, and long context into a single number is valuable because it allows side-by-side comparison. A product moving from 53 to 60 points shows that the improvement is observable and puts it level with other models in the same tier.
Using that score as the sole basis for a purchase decision, however, is risky, for three reasons.
First, a single score discards a lot of information. Two models that both score 60 might differ by 5 points on coding and 8 on long-context retrieval, with the gaps canceling out in other dimensions. If your business is code generation, look at the code-specific score. If it’s document review, look at long-context and structured output performance.
Second, benchmarks don’t match the distribution of real workloads. Public benchmark questions have been refined and cleaned over a long time, while production requests are full of noise: vague requirement descriptions, huge repositories, private frameworks, internal jargon. A 7-point lead on a leaderboard might translate into almost no difference in your scenario. It might also mean task success rates rising from 70% to 85%.
Third, scores don’t reflect latency or stability. In real SLAs, time to first token, throughput, P99 latency on long requests, rate limits, and retry behavior often weigh more than a two- or three-point capability gap.
The takeaway is clear: use leaderboards as a filter to build a candidate pool, and use your own evaluation set as the final judge.
3. Cost per Task: The Only Directly Comparable Metric
Vendors’ unit prices can’t be compared across models, but cost per task can. The math isn’t complicated:
task_cost = (in_tokens * p_in+ out_tokens * p_out+ cached_tokens * p_in * r_cache) / 1_000_000
Here, r_cache is the discount factor for a cache hit relative to the input price, about 0.186 in this example. Take a typical scenario: 20,000 input tokens, 80% of them cache hits, and 5,000 output tokens.
- Cached portion: 16,000 × 1.40 × 0.186 ≈ 4166
- Uncached input: 4,000 × 1.40 = 5600
- Output: 5,000 × 4.40 = 22000
- Total: ≈ 3.18 (in per-million-token pricing units)
Output alone makes up nearly 70% of the total. That’s why cost differences between models in the same tier often come not from the price table but from how many tokens a model has to produce to finish the job.
In public evaluation data, three models in the same intelligence tier cost roughly 0.68, 0.84, and 1.23 per task. The most expensive is nearly double the cheapest. This isn’t a pricing difference but a capability-efficiency difference: a model that gets things right on the first try needs no retries, and one that reasons more concisely doesn’t need to keep correcting itself. Selection should therefore measure the average cost of completing the same batch of real tasks, not read off the price table.
4. The Overlooked Variable: Output Token Inflation
One more detail from the same evaluation is easy to skim past: the new generation uses about 20% more output tokens than the previous one to complete a task. Plug that into the example above, with output rising from 5,000 to 6,000:
new_cost = 4166 + 5600 + 6000 * 4.40 ≈ 36166
Total cost climbs from about 3.18 to 3.62, roughly a 13.9% increase. If the cache hit rate also falls because of prompt changes, the increase will be larger.
In other words, “the unit price didn’t go up” and “the bill didn’t go up” are two different claims. To tell whether an upgrade saves or costs money, you need to look at three things at once:
- Task success rate: failed retries are the biggest source of hidden cost.
- Output tokens per task
- Cache hit rate
Watching only the third price dimension makes it easy to notice rising spending only when the quarterly bill arrives.
One point is easy to misjudge, though: longer output isn’t necessarily waste. More thinking tokens often go with a higher rate of correct answers, which avoids the full round of retries that one failure triggers. The right measure is cost per successful task — total cost divided by the number of successful tasks — not token count alone.
5. Engineering Implications of the Capability Upgrades
This generation publicly emphasizes three directions: coding and engineering ability, defensive cybersecurity, and long-horizon autonomous task execution. Each has different engineering implications.
Coding and engineering ability means repository-level understanding beyond single-file completion. It covers cross-file dependencies, existing code style, and test framework conventions. The practical value for developers is that the model can produce a patch that merges cleanly without changing existing interfaces, rather than snippets that “look right but don’t compile.” Making this work depends on supplying the right repository context. Use retrieval to bring relevant symbols, interface definitions, and test cases into the context, instead of stuffing an entire directory into the window.
Defensive cybersecurity refers to capabilities aimed at fixing and protecting: locating vulnerabilities, auditing code, identifying risks in dependency components, and generating patches. A clear boundary applies here. No model should be used to produce ready-to-use attack payloads or techniques for evading detection. Enterprise technology choices and prompt design should state defensive use explicitly in policy, backed by content safety controls and human review. The real value of these capabilities is shifting security left: catching injection, deserialization, authorization bypass, and hardcoded secrets before code is committed.
Long-horizon autonomous task execution is the dimension where models differ most today and the hardest to evaluate. The longer the task chain, the more history piles up in the context and the larger the output. Cost and latency therefore grow faster than linearly. Engineering needs checkpoints and resume support, intermediate artifacts written to disk, and a “bring only the context each step needs” policy. Otherwise, token consumption spirals out of control as the step count rises.
6. Six Ways to Cut Token Costs
Tiered routing. Use rules, small models, or lightweight classifiers for intent recognition, formatting extraction, and simple classification, and send only complex reasoning to the large model. In typical projects, more than 60% of requests fall into categories that can be downgraded.
Stable prefixes for cache hits. Keep the system prompt, tool definitions, and few-shot examples at the very front and byte-for-byte unchanged. Put timestamps, session IDs, and user variables after the prefix. This usually pays off more than any prompt trick.
Output constraints. Specify the output format and length explicitly, for example, “return only the patch code, no explanation.” Structured output instead of free text cuts tokens and reduces parsing failures.
Context trimming and compression. Replace long conversations with summaries. Summarize tool call results in a structured form rather than pasting them back in full. Rerank retrieved snippets and keep the top K. The principle: include only what the current step needs.
Caching and idempotency. Cache results for semantically similar requests. For failed tasks that can be replayed, use backoff retries rather than retrying immediately. Retry amplification is the most common cause of runaway bills.
Observability. Log per-request input, output, and cache-hit token counts, along with whether the task succeeded, and send them to your logging and monitoring services to build a cost dashboard. Without cost attribution at the task level, no optimization can be verified.
7. Action Checklist for Teams
- Run candidate models on your own evaluation set (50 to 200 real business samples). Record success rate, average output tokens, and average latency for each.
- Use cost per successful task (total cost divided by the number of successful tasks) as the selection metric, not the price table.
- Check the cache hit rate before launch, and make it a standard monitoring metric rather than an optional one.
- Manage multi-model routing, rate limiting, credentials, and usage tracking through a unified model access layer, such as a platform service like Baidu AI Cloud’s Qianfan, instead of scattering cost governance across individual business codebases.
- Enforce mandatory human review and automated testing gates for code generation and security-related scenarios.
To return to the bigger picture: as leading models cluster in the same intelligence tier, competition shifts from “who is smarter” to “who can get the same job done right at a lower cost.” For developers, that calls for a new selection methodology. Don’t compare unit prices; compare cost per task. Don’t compare leaderboard scores; compare your own evaluation set. Don’t compare parameter counts; compare cache hit rates and failure retry rates. Teams that know how to run these numbers will get a lower bill at the same level of intelligence.
