LLM API Prices Are Rising: How to Calculate Unit Task Cost
Rising list prices for large-model APIs and headline subscription quotas do not reveal the cost a team actually pays. This article explains why the meaningful metric is cost per completed task, breaks down prefill and decode pricing, cache hits, input/output ratios, and package economics, and provides a practical cost model, code examples, evaluation criteria, engineering optimization methods, and observability metrics for controlling AI spend.
Over the past two years, large-model API prices have risen noticeably. In one recent model generation, output-side pricing reached roughly 100 per million tokens, input-side pricing around 20, and cached-hit pricing around 1, often several times the previous generation. Debates tend to focus on two questions: whether the absolute unit price is expensive, and whether a subscription package’s quota is worth it.
Both questions miss the point. Absolute unit price reflects the provider’s compute cost and pricing strategy, while package quotas only convert payment into a different form. Neither equals the amount your team actually pays. The number that matters for budgeting is the cost per task: the average amount spent to complete one real business request. This article breaks down that metric and gives a practical way to estimate and optimize it.
1. Why there are three pricing tiers
Large-model inference can be divided into two stages with different characteristics:
- Prefill: Ingest all input tokens at once. It can be highly parallelized and makes heavy use of GPU compute; it is compute-intensive.
- Decode: Generate output tokens one by one in an autoregressive loop. Each step rereads the key-value cache for all previous tokens; it is memory-bandwidth-intensive, and parallelism is constrained by sequence length.
The output-side price is usually three to five times the input-side price for this reason. With the same compute budget, decode naturally produces fewer tokens than prefill. This is not an arbitrary pricing choice; it reflects the different hardware efficiency of the two workloads.
The cache-hit tier can be much lower because it skips prefill computation. If the prefix of a request—such as the system prompt, few-shot examples, long documents, or tool definitions—is identical to a previous request, the inference service can reuse the cached key-value state and run incremental prefill only for the new part. Therefore, cache hits are not a promotional discount; they are a cost structure that callers can actively pursue through engineering.
This distinction matters: the 20x and 100x gaps between tiers are fundamentally an engineering problem, not a financial one.
2. The gap between listed price and real cost: usage structure
Providers publish a price per million tokens, but your bill is determined by three variables:
- Input/output ratio: For example, 2,000 input tokens with 200 output tokens versus 200 input tokens with 2,000 output tokens can differ by several times.
- Cache hit ratio: Increasing the hit ratio from 10% to 60% can cut input-side cost by more than half.
- Effective unit price of a plan: The plan price divided by the quota you can actually consume is what you really pay per unit.
The third is often misread. A plan tier may claim to support a large amount of pay-as-you-go usage, but if your real usage is lower, the effective cost may be much higher. Conversely, a lower-priced tier can have a lower effective unit price if it covers enough real usage. The only sound calculation is:
effective unit price = plan price / expected real usage
Do not judge a plan by its tier level or by the absolute amount of usage it claims to support.
3. A practical cost-per-task model
To perform sensitivity analysis, write cost as an explicit formula:
C_task = (N_in_hit × P_hit + N_in_miss × P_in + N_out × P_out) / 1_000_000
where N_in_hit is cached input tokens, N_in_miss is uncached input tokens, and P_* are the corresponding unit prices. Given N_in_miss + N_in_hit = N_in, we can derive:
C_task = (N_in × P_in - N_in × h × (P_in - P_hit) + N_out × P_out) / 1e6= (N_in × P_in × (1 - h × 0.95) + N_out × P_out) / 1e6
where h is the cache-hit ratio, and the second line holds when P_hit ≈ 0.05 × P_in, as in the example where cached input is about 5% of normal input price.
This yields an important conclusion: input-side cost is roughly linearly sensitive to the hit ratio, while output-side cost is unaffected. The priority of optimization effort should therefore depend on your business’s N_in / N_out ratio.
A calculation snippet that can be embedded in load-testing scripts:
from dataclasses import dataclass@dataclass(frozen=True)class PriceBook:"""Unit price: cost units per 1M tokens"""input: float = 20.0output: float = 100.0cache_hit: float = 1.0def cost(self, n_in: int, n_out: int, hit_ratio: float) -> float:assert 0.0 <= hit_ratio <= 1.0, "Hit ratio must be between 0 and 1"hit = n_in * hit_ratiomiss = n_in - hitcost = (hit * self.cache_hit + miss * self.input + n_out * self.output) / 1e6return costdef compare(n_in: int, n_out: int, pb: PriceBook, ratios=(0.0, 0.3, 0.6, 0.9)):"""Compare task cost under different cache-hit ratios for model selection and budgeting."""base = pb.cost(n_in, n_out, ratios[0])for r in ratios:c = pb.cost(n_in, n_out, r)print(f"Hit ratio {r:>4.0%} task cost {c:.5f} relative to baseline {c / base:>5.0%}")if __name__ == "__main__":compare(n_in=8_000, n_out=800, pb=PriceBook())
For a typical long-context task with 8,000 input tokens and 800 output tokens, raising the hit ratio from 0 to 0.9 reduces cost per task by about 60%. If your scenario has high input volume—such as document question answering or codebase analysis—cache optimization is usually the highest-ROI work. If output volume is high—such as long-form generation or bulk translation—focus on output-length control and model tiering.
4. Evaluation: benchmarks cannot be converted directly into business cost
Forum-style discussions often make one of two mistakes: assuming that a model ranked third overall is therefore worth the higher price, or assuming that a higher-priced model is cheaper because it is more expensive. Neither inference holds. Benchmarks measure capability ceilings, while cost measures consumption per task. The missing conversion factor is how much capability your task actually needs.
The right approach is to build a small, representative evaluation set for your business—tens to hundreds of real requests are enough—run each candidate model against them, and record three metrics for each request: task completion rate, average token consumption, and cost per task. Then select on a Pareto frontier of completion rate versus cost.
In practice, most requests can be handled by mid- or lower-tier models. Only about 5% to 15% of difficult requests need the most capable and expensive model. If you pay for 100% of requests in order to serve that 10%, cost will spin out of control.
5. Six engineering practices to bring unit cost back under control
5.1 Stabilize the prefix: make cache hits the norm
Caches apply only to byte-for-byte identical prefixes. In practice, cache misses are often caused by trivial issues:
- the system prompt includes timestamps, request IDs, or randomly ordered examples;
- tool definitions, such as function schemas, are re-serialized on every call in map iteration order, so field order is unstable;
- user history messages are inserted between the system prompt and long documents, causing the long documents to shift position each time.
The constraint is: keep the prefix strictly incremental and immutable. Structure requests as static system prompt → static tool definitions → static long documents or knowledge snippets → current dynamic input, with all dynamic content placed at the end.
import jsondef build_prompt(sys_tpl: str, tools: list[dict], docs: list[str], user_msg: str) -> list[dict]:# Static segments: template, tool definitions, and documents. Keep order fixed and do not mix in dynamic fields.static = sys_tpl + "\n" + json.dumps(tools, ensure_ascii=False, sort_keys=True)static += "\n" + "\n".join(docs)# Dynamic segment: place it only at the end to avoid breaking prefix consistency.return [{"role": "system", "content": static},{"role": "user", "content": user_msg},]
Details such as sort_keys=True matter: stable serialization is a prerequisite for cache hits.
5.2 Context engineering: do not leave unnecessary tokens
Uncached input is billed at full price, so sending less is more fundamental than getting cache hits. Useful combinations include:
- Retrieve only what is needed: inject top-k relevant snippets rather than an entire document.
- Compress context: apply rolling summaries to long conversation histories, keeping conclusions and dropping intermediate steps.
- Structured pruning: after parsing HTML, logs, or JSON, retain only fields required for the task.
- Hard budgets: set input token limits by request type; if exceeded, downgrade to segmented processing.
5.3 Model-tier routing
Use a small, fast model for intent recognition, format validation, content extraction, and initial scoring. Escalate to the flagship model only when complex reasoning is required. Make routing configurable rather than hard-coded so it can be tuned based on production evaluation results:
routing:- name: classifytier: small # classification, extraction, field normalizationmax_input: 2000- name: reasontier: flagship # multi-step reasoning, code generation, long-form writingfallback: [medium]budget:daily_token_cap: 80_000_000alert_ratio: 0.8
5.4 Constrain output length
Because output tokens are usually several times more expensive than input tokens, verbose answers multiply cost. Three actions have immediate effect: specify word or field limits in the prompt; use structured output modes such as JSON Schema to reduce boilerplate and filler; split explanatory output from result output into separate calls, using a smaller model for the explanatory part.
5.5 Batch processing and asynchronous execution
Many inference services offer discounted pricing for non-real-time batch calls. For offline workloads—batch labeling, offline summarization, log consolidation, and dataset generation—use a message queue for peak shaving and queuing, and execute them through non-real-time channels. Keep only requests that truly need low latency in the synchronous online path. Combined with elastic scaling on a container platform, this allows more generous compute capacity to be used during off-peak periods.
5.6 Build or fine-tune high-frequency subtasks
For subtasks with concentrated traffic and fixed patterns—such as intent classification, field extraction, and templated rewriting—use accumulated call logs to fine-tune models or train smaller task-specific models. Long-term cost is often an order of magnitude lower than calling a general-purpose large model. The prerequisite is proper data desensitization and evaluation-set regression testing to avoid quality regressions.
6. Cost observability: no instrumentation, no cost reduction
Measure before optimizing. Record structured logs at the gateway layer for every call, with at least the following fields:
{"trace_id": "...","biz_scene": "invoice_extract","model_tier": "flagship","input_tokens": 3120,"output_tokens": 410,"cache_hit_tokens": 2450,"latency_ms": 1870,"task_success": true,"cost": 0.0331}
Build four core metrics on top of these logs:
- Cache hit rate (
cache_hit_tokens / input_tokens): a direct measure of prefix-engineering quality. - Average token consumption per task: an efficiency baseline for cross-version comparison; alert immediately if it jumps after a model or prompt change.
- Cost per unit of business output: cost per 1,000 requests, cost per active user, cost per ticket; this is the metric that can be reported to management.
- Downgrade trigger rate and retry rate: retries are a hidden cost driver, and one failed retry can mean a full-price input round.
On cloud platforms, these capabilities are commonly provided through general categories such as logging, monitoring and alerting, object storage, message queues, and container platforms. Combined with budget-threshold alerts, they form a complete cost-governance loop.
7. Conclusion: higher prices are not the problem; cost per task is
Returning to the original debate: a severalfold increase in next-generation unit prices only exposes changes in the cost structure. Arguing about whether the list price is high or whether a package has enough quota does not address the real issue. Three questions need to be answered:
- How much does my business pay per task?
- Can I remove 60% to 70% of that cost through prefix caching, context compression, tiered routing, output constraints, and batch processing?
- Given that quality requirements are met, is there an alternative combination with a lower effective unit price?
For package economics, calculate effective unit price as plan price divided by expected real usage. For model selection, use a Pareto frontier of completion rate versus cost per task. Once these two metrics become standard team processes, any future generation-driven price increase becomes a calculable arithmetic problem rather than a debate based on intuition.
