How to Calculate LLM API Costs: From Token Pricing to Per-Task Cost
LLM price changes routinely dominate developer discussions, yet the debate usually stops at "how many times more expensive" and rarely answers the more important question: how much does it actually cost to complete the same batch of tasks in your own application? This article breaks down how model pricing is constructed, introduces a reusable measurement method centered on "cost per successful task," and shows how load structure, cache hit rate, task success rate, and routing strategy determine the real bill. It covers input/output pricing asymmetry, cache-hit discounting, context-length tiers and add-ons, the hidden costs of retries and fallbacks in long-context and agent workloads, a comparison of subscription versus pay-as-you-go billing, and four layers of engineering-level cost control, including model routing, context reduction, output constraints, and budget circuit breakers.
1. Introduction: Why Price Changes Spark Debate
Price adjustments to large language models almost instantly become a focus of attention in the developer community. When a new generation of a model is released, if its capabilities improve markedly but the per-token prices for input, output, and cache hits are raised several-fold over the previous generation, controversy is nearly inevitable. The discussion usually stays at the level of “how many times more expensive” or “cheaper than X or not,” yet rarely answers a more important question: in my own workload, how much does it cost to complete the same batch of tasks?
The root of the problem is the mismatch in how things are compared. The unit price is only the “scale marking” of a pricing system; the real expense depends on the shape of your workload, including the input-to-output token ratio, context length, cache hit situation, and the number of turns and retries. The same price sheet, applied to short question-answering and to long-document agent tasks, can produce costs that differ by an order of magnitude.
This article first breaks down how model pricing is constructed, then offers a reusable estimation method, and finally lands on engineering-level cost-control techniques. With this, readers can turn the subjective feeling of “expensive or not” into a measurable, optimizable metric.
The takeaway (TL;DR)
- Unit prices cannot be compared directly; first compute “cost per successful task = total spend over the period / number of successfully completed tasks.”
- Output prices are usually higher than input prices, so output-heavy tasks are dominated by the output side of the bill.
- Cache-hit prices are far below the input price, and the cost of long-context work is determined almost entirely by the hit rate.
- Failed retries and fallback handling are hidden expenses; a task with a 90% success rate consumes roughly 1.1× the tokens of the ideal case.
- Cost reduction comes from engineering techniques: tiered routing, stable prefixes, retrieval instead of stuffing in everything, and budget circuit breakers.
2. Key Dimensions of Model Pricing
2.1 Input and Output: An Asymmetric Cost
Mainstream inference APIs bill by token and price input and output separately. The reason is that the two are computed differently: the input stage can process the entire context in parallel (called “prefill”), while the output stage must generate tokens autoregressively, one at a time. As a result, the output price is usually significantly higher than the input price.
This asymmetry determines where optimization should focus: the longer the output, the more its impact on the total bill is amplified. Output-heavy tasks such as summarization, rewriting, and code generation can be dominated by the output side of the bill even when the input is short.
2.2 Cache Hits: Selling Repeated Computation at a Discount
The cache-hit price is usually only a fraction of the input price, making it the most easily overlooked variable in today’s pricing. The principle is that when the model processes a request, it generates a set of key-value pairs (KV Cache) for each token in the input, to be reused by the attention computation of subsequent tokens. When the prefix of a new request is identical to that of a previous request, the prefill computation for that prefix can be reused directly.
Pricing therefore distinguishes three kinds of input/output tokens:
- Uncached input: billed at the standard input price;
- Cache-hit input: billed at a significantly lower cache price;
- Output: billed at the output price.
It is worth stressing that the conclusion “cache hits only apply to reusable stable prefixes” is itself constrained by the provider’s policies. The cache lifetime, the minimum cacheable length, whether caching is enabled automatically, and the discount ratio on hits all vary across vendors, and a hit does not constitute any service guarantee. In multi-turn conversations, fixed system instructions, and long-document question-answering, prefixes are naturally highly repetitive and the hit rate is often substantial; in scenarios where every request has completely different content, the hit rate approaches zero. Refer to the official documentation for the exact rules.
2.3 Easily Overlooked Tiers and Add-Ons
Many providers do not charge a single price for all requests. Common additional dimensions include:
- Context-length tiers: input prices in the very-long-context bracket may be higher;
- Invocation mode and latency: asynchronous or batch jobs may get discounts, while real-time calls are usually billed at the standard price;
- Additional capabilities: value-added features such as web retrieval, file parsing, and voice may be billed separately.
When choosing a provider, rely on the pricing rules in the official documentation and factor tier boundaries into your estimates.
2.4 The Basic Cost Formula
Ignoring tiers and add-on fees, the cost of a single call can be written as:
cost per call = (uncached input tokens × input price+ cache-hit tokens × cache price+ output tokens × output price) / 10^6
Input tokens must be split into “uncached” and “hit” portions and billed separately; otherwise the same batch of tokens would be counted twice, once at the standard price and again at the cache price. Prices here are expressed per million tokens. For ease of discussion, this article uses “price units (pu)” to represent relative prices, without reference to any real tariffs. The examples below use these assumptions:
| Pricing item | Assumed price (pu / million tokens) |
|---|---|
| Input (uncached) | 20 |
| Cache hit | 1 |
| Output | 100 |
| Light model input (uncached) | 2 |
| Light model cache hit | 0.1 |
| Light model output | 8 |
3. From Unit Price to “Cost per Successful Task”
3.1 Three Common Pitfalls
| Common practice | The problem | A better alternative |
|---|---|---|
| Comparing only the per-million-token price | Ignores task structure; output-heavy tasks are underestimated and long-input tasks are overestimated | Replay real traffic and record the distribution of input/output/cache-hit tokens |
| Comparing someone else’s metered usage against your own subscription | The two use different accounting bases, and price shifts dynamically with usage tiers and tier rules | Convert everything to “cost per successful task” before comparing |
| Counting only the cost of successful samples | Ignores the wasted cost of retries, timeouts, and failures | Include failed tasks in the denominator and evaluate the true cost with fault tolerance |
3.2 Definition: Cost per Successful Task
It is recommended to use “cost per successful task“ as the core metric:
cost per successful task = total spend over the period / number of successfully completed tasks
Here the total spend must include all calls, including retries, fallbacks to a backup model, and sub-calls generated by tool invocations; success is judged by a business-verifiable standard, such as the output passing format validation, unit tests, or manual spot checks.
3.3 Differences Across Three Typical Scenarios
- Long-document question-answering: The input is dominated by the document body, and a single request can consume a large amount of input tokens; the cache hit rate depends on whether the document is referenced repeatedly.
- Content writing: The output share is high, so a price increase hits the output side hardest, but a task is usually completed in a single call.
- Agent tasks: The task is decomposed into a multi-turn loop of “plan → call tool → read result → re-plan,” where every turn must carry the full historical context; token consumption can reach several times, or even tens of times, that of a simple conversation.
3.4 Frequently Asked Questions
Q: After a model raises its prices, how can I quickly tell whether it actually became more expensive?
A: Replay a segment of real traffic, record the distribution of the three token types and the task success rate, and plug them into the formula in 2.4 to compute the old and new “cost per successful task.” It is common for the unit price to rise while the cost stays flat, and vice versa.
Q: Why can cache hits reduce costs so much? Will they always apply?
A: A hit means the prefix does not need to be prefilled again. But a hit depends on the prefix being exactly identical, and is affected by the cache lifetime, the minimum length, and the provider’s policies, so budget for “a possible hit” rather than a “guaranteed hit.”
Q: How can long-context tasks be made cheaper?
A: Three paths: use retrieval to fetch only the relevant fragments instead of stuffing in everything; pin stable documents at the prefix of the prompt to reuse the cache; move non-real-time batch tasks to an asynchronous/batch channel.
Q: How should I choose between a subscription plan and pay-as-you-go?
A: See the comparison table in Section 5; the core judgment is whether the workload is stable and predictable.
3.5 An Estimation Code Example
The following code compares task costs across different models and cache hit rates under a unified pricing model (all illustrative assumptions, not the real tariffs of any vendor):
# Pricing config: units of pu / million tokens, illustrative only, not any vendor's real tariffsPRICING = {"flagship": {"input": 20, "cache": 1, "output": 100},"light": {"input": 2, "cache": 0.1, "output": 8},}# Task profile: an agent task as an examplePROFILE = {"turns": 15, # total turns, including planning, tool calls, and summarization"avg_input": 12000, # average input tokens per turn (including history and tool returns)"avg_output": 400, # average output tokens per turn"cache_ratio": 0.6, # share of the input that hits the cache}def estimate_cost(profile, model):price = PRICING[model]billable = 0.0for _ in range(profile["turns"]):hit = profile["avg_input"] * profile["cache_ratio"]billable += (hit * price["cache"]+ (profile["avg_input"] - hit) * price["input"]+ profile["avg_output"] * price["output"])return billable / 1e6for model in PRICING:cost = estimate_cost(PROFILE, model)print(f"{model:8s} cost per task: {cost:.4f} pu")
The output is:
flagship cost per task: 2.1480 pulight cost per task: 0.2028 pu
Manual check: each turn has 7200 hit tokens, 4800 uncached tokens, and 400 output tokens. The flagship model costs 7200×1 + 4800×20 + 400×100 = 143200 pu·tokens per turn, or 2.148 pu over 15 turns; the light model costs 7200×0.1 + 4800×2 + 400×8 = 13520 per turn, or 0.2028 pu over 15 turns.
As you can see, the cost per task of the flagship model is about 10.6× that of the light model. This multiple differs from the input price ratio (10×) and does not equal the price gap of any single dimension either; it is jointly determined by the composition of the three token types—uncached input, cache hits, and output. Only by plugging the real workload into the model can the impact of a price change be evaluated accurately.
4. Hidden Costs in Long-Context and Agent Scenarios
4.1 The Cache Hit Rate Determines the Real Price of Long Contexts
In a long-document scenario, suppose each request carries 80,000 tokens of document plus 500 tokens of output, so the output share is under 1%. Using the assumed prices from earlier:
- Cache miss: 80000×20 + 500×100 = 1,650,000, about 1.65 pu per call;
- With a 90% cache hit rate: 72000×1 + 8000×20 + 500×100 = 282,000, about 0.282 pu per call.
That is roughly a 5.9× difference. If the system analyzes the same material repeatedly, whether the cache hits almost decides whether the cost is acceptable.
4.2 Three Common Reasons for Cache Invalidation
Caches typically match only exactly identical prefixes. The following situations cause tokens that could have hit to be billed again at the standard price:
- Unstable system prompts: dynamically inserting timestamps, request IDs, or randomized examples on every request;
- Changing tool and example order: tool definitions serialized from a set or hash structure, with a different order each time;
- Poor content placement: documents concatenated at the end of the prompt, after the prefix has already accumulated a lot of changing content.
In addition, vendors differ in their policies on cache lifetime, minimum cacheable length, and whether caching is enabled automatically; confirm these through the official documentation.
4.3 Cache-Friendly Message Organization
A reliable approach is “stable content first, dynamic content last”:
import jsondef build_messages(system: str, tools: list, history: list, docs: list, query: str):"""Put stable content first and dynamic content last to improve prefix cache hit rate"""messages = [{"role": "system", "content": system}]# Serialize tool definitions stably to avoid prefix changes from ordering jitterif tools:tool_text = json.dumps(tools, ensure_ascii=False, sort_keys=True)messages.append({"role": "system", "content": f"Available tools: {tool_text}"})# Documents injected at the start of the session go in the prefix so later turns can hit againif docs:doc_text = "\n\n".join(f"[Doc {i + 1}] {d}" for i, d in enumerate(docs))messages.append({"role": "system", "content": f"Reference material:\n{doc_text}"})# Conversation history grows naturally, so place it as the dynamic part at the backmessages.extend(history)messages.append({"role": "user", "content": query})return messages
Note that some inference APIs rewrite system messages on the server side or insert their own content. Even so, organizing with “fixed content first” still helps maximize the reusable prefix. Refer to the API documentation for the specific caching mechanism.
4.4 Output and Retries: Two Underestimated Expenses
- Output: As noted earlier, the output price is higher. By constraining the output format (e.g., a JSON Schema), setting a reasonable maximum output length, and splitting the flow into “draft → refine,” you can avoid unnecessary generation.
- Retries: Retries triggered by timeouts, rate limiting, and validation failures produce extra calls. A task with a 90% success rate that retries once on failure consumes about 1.1× the tokens of the ideal case (1/0.9 ≈ 1.11). If a more expensive backup model is used as a fallback after a failure, the cost rises further.
5. Subscription Plans vs. Pay-as-You-Go: How to Compare
5.1 The Nature of the Two Billing Models
A subscription plan is essentially a “prepaid discount”: the user commits a fixed spend per period in exchange for a lower effective unit price and quota tilting. By comparison, pay-as-you-go settles by actual consumption, with no idle waste but also no volume discount.
When comparing the two, the key is not the plan price itself, but the effective usage it can support.
5.2 The Effective Unit Price of a Subscription
subscription effective unit price = plan cost / actually usable tokens within the period
If the plan has rate or period caps, the actually usable amount can be lower than the advertised figure. For example, a plan designed around a period cap, if the actual workload only “just fills” it, can have an effective unit price nearly identical to pay-as-you-go, and the advantage of subscribing is limited.
5.3 Recommendations
| Workload characteristic | More suitable model | Reasoning |
|---|---|---|
| Stable, predictable usage with long-term batch jobs | Subscription or reserved throughput | Amortizes the unit price, locks in capacity |
| Pronounced peaks and troughs, mostly interactive requests | Pay-as-you-go | Avoids paying for idle capacity |
| Short-term validation, traffic not yet settled | Pay-as-you-go, with budget caps | Controls the cost of trial and error |
| Mixed scenarios | Combine both | Baseline load on subscription, overflow billed on demand |
Whatever the model, you should set usage caps or budget alerts to avoid unexpected spending from abnormal calls.
6. Four Layers of Engineering Cost Reduction
6.1 Routing Layer: Tier by Task Difficulty
Not every task needs the most capable model. A lightweight model can handle the task first, escalating to the flagship model only when confidence is insufficient or validation fails:
QUALITY_GATE = 0.85 # confidence threshold, to be calibrated against business evaluation resultsdef route(task, light_model, flagship_model, scorer):result = light_model(task)if scorer(result) >= QUALITY_GATE:return resultreturn flagship_model(task) # fallback escalation, triggered for only a small fraction of tasks
The key to tiering is calibrating QUALITY_GATE: too low a threshold lets quality problems leak downstream, while too high a threshold degrades into “use the flagship model for everything.” It is recommended to first sweep the threshold on an offline evaluation set to find the balance point between quality and cost.
6.2 Context Layer: Control What Each Request Carries
- Retrieval instead of everything: use vector or keyword retrieval to fetch the relevant fragments instead of stuffing in the whole document;
- History compression: summarize early conversation turns, keeping decisions and conclusions while discarding process content;
- Structured tool returns: trim tool results to the necessary fields, avoiding passing back full logs or the raw text of web pages.
6.3 Output Layer: Reduce Wasteful Generation
- Constrain output structure with a JSON Schema and enum values to reduce retries caused by format errors;
- Set a reasonable maximum output length for generation tasks;
- For fixed template portions, use local fill-in and let the model generate only the parts that change.
6.4 Budget Layer: Make Cost Observable and Controllable
The prerequisite for cost governance is visibility. It is recommended to log, for each call, the model name, input/output/cache-hit token counts, latency, and task result, and to aggregate by scenario:
- The average token composition per task type;
- The trend of the cache hit rate;
- The quantile distribution of cost per task.
Because prompts and tool returns often contain user text and business data, the cost logs themselves also need governance: log only token counts and hash fingerprints, and do not store full text; if retention is truly required, mask fields (phone numbers, emails, ID documents, keys, etc.) first, and set different retention periods according to data classification. Logs and monitoring metrics should be access-controlled by project/scenario and follow the principle of least privilege, so that cost reports do not become a data side-channel that bypasses business authentication.
Based on these metrics, you can set a daily budget circuit breaker:
from collections import defaultdictclass BudgetGuard:"""Daily budget guard with per-scenario accounting; trips when the threshold is exceeded"""def __init__(self, limits: dict):self.limits = limitsself.spent = defaultdict(float)def charge(self, scene: str, cost: float):if self.spent[scene] + cost > self.limits[scene]:raise RuntimeError(f"Scene {scene} has exceeded the daily budget; circuit broken")self.spent[scene] += costdef summary(self):return {scene: {"used": spent, "limit": self.spent[scene]}for scene, spent in self.spent.items()}
When running in the cloud, these metrics can be fed into a log service for unified collection, and monitoring alerts can notify you promptly when the cache hit rate drops sharply or the cost per task rises abnormally. Model serving itself can use the capabilities offered by mainstream model service platforms, combined with your own workload, to complete cost accounting and governance.
7. A Model Selection Comparison Checklist
When prices change, the most rational response is not to wait and see, but to evaluate candidate options against a fixed set of dimensions:
| Dimension | How to evaluate |
|---|---|
| Task quality | Build an evaluation set from real business, covering typical and edge cases |
| Cost per successful task | Estimate under a unified pricing model, including retry and failure costs |
| Cache support | Whether caching is automatic, minimum length, lifetime, and hit discount |
| Long-context capability | Quality and price tiers at the target length |
| Performance and stability | Time to first token, throughput, rate-limiting policy, and availability commitments |
| Ecosystem and migration cost | API compatibility and the reusability of prompts and evaluation assets |
8. Conclusion and Action Checklist
A several-fold increase in model prices does not necessarily mean costs spiral out of control; flat or falling prices do not mean no optimization is needed either. What really determines the bill is the shape of your workload, the cache hit rate, the task success rate, and the routing strategy.
Implementation checklist
- You already meter calls per scenario: uncached input, cache hits, output, and failed retries are counted separately;
- You have computed the “cost per successful task” under a unified pricing model and track its quantiles;
- Your prompts have been reworked into “stable prefix + dynamic suffix,” and tool definitions are stably serialized;
- You have configured routing thresholds and quality gates, and regularly regression-test and recalibrate them;
- You have set daily budget circuit breakers, usage alerts, and a log-masking policy.
By building an estimation model around “cost per successful task,” together with tiered routing, cache-friendly assembly, and budget circuit breakers, developers can maintain dual control over cost and quality in a market where capabilities iterate quickly. When a generation of models adjusts its prices, rather than rushing to argue about expensive versus cheap, ask first: can it complete my task more reliably at an acceptable cost?
