New Model APIs, Same Price: How to Calculate the Real Cost Per Task
Model vendors keep raising capability while holding API prices flat, but a stable unit price doesn't guarantee a stable bill—a newer model can consume more output tokens per task and quietly push up end-to-end costs. This article breaks down how token-based billing actually works, explains why composite capability scores are useful for shortlisting but unreliable for cost decisions, and shows how to translate per-token pricing into a meaningful cost per completed task. It includes sample Python for tracking task-level metrics, a look at output-token bloat and cache hit rates as cost levers, and the near-quadratic cost growth of long-context Agents—plus budget guardrails to keep it in check. Readers will come away with a reusable cost-accounting method and a practical checklist for making rational model-selection decisions.
1. Introduction: The Unit Price Didn’t Change, but Your Bill Might
Every time a large-model provider ships a new version, it usually publishes two kinds of information: changes in overall capability (often citing a third-party composite intelligence benchmark) and API pricing—tiered unit prices for input, output, and cache-hit input.
A recent, representative upgrade illustrates the pattern:
- Capability side: a new-generation model scored 60 on a public composite intelligence benchmark, up from 53 for the previous generation in the same series. It now ties another model in the same tier, while the 60-point tier clearly pulls ahead of the previous generation.
- Price side: input, output, and cache-hit input all hold at the previous generation’s unit prices. The vendor cites a per-task cost of about $0.68—roughly 20% below another same-score model (about $0.84) and more than 40% below a higher-priced model (about $1.23).
- Hidden risk: the same benchmark data shows the new-generation model consumes about 20% more output tokens per task on average than the previous generation. The unit price hasn’t risen, but doing the same work now means “writing” noticeably more—so the end-to-end bill may not actually stay flat.
These figures come from vendor announcements and third-party benchmarks with different calibers and samples, so they shouldn’t be taken directly as a purchasing decision. But they point to the same engineering question: as models keep getting upgraded, how do you put “capability” and “cost” on the same ledger?
Rather than focus on any specific product, this article extracts a general problem: how to build a reusable method for accounting for the cost of a single task, and to make rational model-selection decisions on that basis. Readers will gain four things:
- an understanding of how token-based billing really works;
- how to correctly interpret composite capability scores;
- cost-accounting metrics, sample code, and key cost-control points for Agent scenarios;
- a practical, actionable selection checklist.
2. Breaking Down Token Billing: Where the Bill Actually Comes From
Most general-purpose LLM APIs bill by token. Raw text is first split into tokens by a tokenizer—for Chinese, roughly one character maps to one or two tokens. The price sheet looks simple, but the actual bill is affected by at least these dimensions:
- Input and output are priced separately: output is usually several times the price of input, a consequence of the per-token autoregressive generation process.
- Cached input and cache writes: a fixed prefix can be cached, and the cache-hit portion is billed at a discounted rate (here about 20% of the regular input unit price, an ~80% discount). Some services also charge a cache-write fee or impose a minimum cacheable-length threshold.
- Context-length tiers: the longer the input, the closer attention computation grows to quadratic, so many services price or rate-limit very long inputs separately.
- Ancillary token consumption: tool-call arguments and return values, chain-of-thought or the internal reasoning tokens of reasoning models (usually counted as output), multimodal input, and retried requests all consume extra tokens.
- Rate limits and quotas: rate limits determine how many tasks you can complete per unit time, which in turn affects how fixed costs (compute, labor, cloud resources) are amortized.
Abstracted into a formula:
task cost = uncached input × P_in + cache-hit input × P_cache + output × P_out + cache write × P_write
where P_in, P_cache, P_out, and P_write are the unit prices for each token type, per the pricing published by the specific service. Two core conclusions follow:
- Output tokens are the single most cost-sensitive item. In the earlier example, P_out is about 3.1× P_in, so small swings in output volume can noticeably change the bill.
- Total input tokens are not necessarily equal to “uncached input.” The cache hit rate directly determines the billing discount on the bulk of the input.
3. Reading Composite Capability Scores: Useful, but Not Directly
A composite capability score is typically aggregated by a third-party evaluation body from multiple benchmarks, covering multidisciplinary knowledge, math and scientific reasoning, code generation, and so on. It’s a useful reference for horizontal shortlisting, but it can’t serve directly as the basis for a cost decision, for four reasons:
- Different statistical calibers. The “7 points” from 53 to 60 in the example is the difference of an index after weighted aggregation across many benchmarks—not the same thing as the improvement in any single capability. When comparing across generations, note whether the evaluation version is consistent.
- The synthesis method determines what the score means. Different models have different weak spots. One may excel at math reasoning, another at long-context handling—but your business might be 70% structured extraction and 30% code completion. A one-point edge in the total score may have nothing to do with the tasks you care about.
- Benchmarks may be saturated or contaminated. Once public benchmarks are widely used in training, their discriminative power drops. A more reliable approach is to re-check with an internal evaluation set that reflects your business.
- Scores contain no cost information at all. They don’t distinguish “answered correctly on the first try” from “answered correctly after repeated corrections,” and they don’t reflect latency, stability, or availability. That’s how you end up with the situation in the example: same-score models can differ in per-task cost by around 20%.
The right way to use them is a two-step “shortlist + re-check”:
- Shortlist: use composite capability scores to narrow the candidate pool—for example, only considering models in the top tier.
- Re-check: compare quality, cost, and latency across candidates using your own business samples, and plot a quality–cost frontier.
4. Cost Per Task: Turning “Unit Price” into “Money per Completed Job”
Many industry evaluations treat “cost per task” as a metric alongside the composite capability score, for a simple reason: the unit price is the price of each token, but what an enterprise actually buys is “the result of a completed job.” For the same task, different models may consume very different numbers of tokens; only when converted to task granularity are different models comparable.
In practice, cost per task needs to be refined into a set of metrics:
- Average cost per task: total cost divided by the number of tasks completed successfully. Costs from failures and retries must be included in the numerator.
- Cost distribution: look at P50 and P90, not just the average. A few long-context tasks or tasks requiring multiple tool calls often account for most of the cost.
- Cache hit rate: the share of cache-hit input in total input.
- Task pass rate and rework rate: the share passing on the first attempt, and the number of additional calls needed to meet acceptance criteria.
- Cost per unit of effective output: for example, the cost per 1,000 lines of code that pass review, or per 100 correctly extracted fields.
The following sample code shows how to compute these metrics from call logs:
from collections import defaultdict# Sample prices, all in USD per million tokens; use the price list published by the service you pick# tokens_per_dollar is a conversion factor, calibrated by dividing total bill by total tokensPRICE = {"in": 1.40, "cache": 0.26, "out": 4.40, "tokens_per_dollar": 1.0}def task_cost(usage: dict) -> float:"""Compute the cost of a single call from its token usage. Example usage:{"in": 1200, "cache": 8000, "out": 650}"""return (usage.get("in", 0) * PRICE["in"]+ usage.get("cache", 0) * PRICE["cache"]+ usage.get("out", 0) * PRICE["out"]) / 1_000_000 * PRICE["tokens_per_dollar"]def stats(records):"""Each record in records has task_id, usage, success (whether the task met the bar), retry_of (may be empty)"""cost, ok_cost, tokens, outputs, cache_hit, success = (defaultdict(float) for _ in range(6))retries = defaultdict(int)for r in records:tid = r["task_id"]c, u = task_cost(r["usage"]), r["usage"]cost[tid] += cok_cost[tid] += c if r["success"] else 0tokens[tid] += u.get("in", 0) + u.get("cache", 0)outputs[tid] += u.get("out", 0)cache_hit[tid] += u.get("cache", 0)success[tid] += int(r["success"])if r.get("retry_of"):retries[tid] += 1 # all cost from retries is charged to the original taskdone = [t for t in cost if success[t] > 0]hit_rate = sum(cache_hit.values()) / max(sum(tokens.values()), 1)return {"tasks": len(cost),"completed_tasks": len(done),"total_cost": round(sum(cost.values()), 4),"avg_cost_per_task": round(sum(cost.values()) / max(len(done), 1), 4),"first_pass_rate": round(sum(success[t] == 1 for t in cost) / len(cost), 3),"cache_hit_rate": round(hit_rate, 3),"avg_output_tokens": round(sum(outputs.values()) / len(cost), 1),"wasted_cost_failed": round(sum(cost[t] - ok_cost[t] for t in cost), 4),}
The sample code embodies three accounting principles:
- First-pass rate reflects stability. The higher the first-pass rate, the fewer repeated calls are needed for corrections.
- Wasted cost on failed tasks reflects the price of failure. It measures the money spent for nothing due to failures and retries. When the failure rate rises, even a cheaper single call may not pay off.
- Cache hit rate reflects architectural efficiency. It reflects the efficiency of architecture and prompt design, not the model’s inherent capability.
5. Output Token Bloat: The Underestimated Cost Amplifier
The new version consuming about 20% more output tokens to complete the same task is common in capability-upgrade iterations: the model “thinks in more detail and writes more,” and tool-call arguments, code comments, and even internal reasoning all add to output length.
A quick calculation: assume a task with 8,000 input tokens and 2,000 output tokens.
- Before the upgrade, the cost per task is about $0.0200;
- After the upgrade, output becomes 2,400 tokens, and the cost rises to about $0.0218—up roughly 9%.
The reason: the cost increase equals the output-volume increase multiplied by “output cost’s share of the total.” Here output cost accounts for 44%, and 20% × 44% ≈ 9%.
In long-context, multi-round scenarios, output’s share is often higher, so the same token increase gets amplified further. But longer output isn’t necessarily a bad thing:
- When rework drops, total cost may actually fall. One model outputs 2,000 tokens per round and needs 3 rounds to meet the bar, for a total of 6,000 output tokens; another outputs 2,400 tokens per round and meets it in 1 round, for a total of 2,400 output tokens. In this case you should compute the “total cost to meet the bar,” not the “single-round cost.”
- Available control measures:
- Constrain output structure through the system prompt, e.g., “return JSON only, no explanation”;
- Set a reasonable maximum output tokens;
- Validate structured output, and on failure return only the error message so the model makes a local correction instead of regenerating the whole thing;
- Route simple tasks to a cheaper small model;
- Apply semantic caching to high-frequency, stable requests.
- Watch for artifact billing. Some providers bill artifact content separately, which noticeably affects code-generation scenarios.
Back to the two-model comparison: the ~20% per-task cost difference—whether that gap holds depends on each model’s actual task-token profile. It’s just an average under one specific evaluation caliber and can’t be extrapolated to all businesses.
6. Long Context and Agents: Cost Grows Roughly Quadratically with Rounds and Context
Long-horizon Agent tasks are the fastest-changing part of the cost structure today. A typical tool-call loop looks like this:
Round 1: system prompt + tool definitions + task description → model decides to call tool ARound 2: all of Round 1 + tool A's return value → model decides to call tool BRound 3: all of Round 2 + tool B's return value → model decides to write a file...Round n: all of the above history + new observation → task completes or budget cap is hit
Each round resends all prior history. Let L be the initial prompt length, Δ the new content per round, and n the number of rounds, then:
- Cumulative input tokens ≈ n·L + Δ·n(n−1)/2, growing quadratically with the number of rounds;
- Tool return values are often large files or entire log sections that quickly crowd the context, causing “context rot”: as irrelevant content piles up, the model’s ability to follow instructions and keep reasoning coherent both degrade;
- Latency accumulates too: each round’s time adds up, and since P_out is higher than P_in, long tasks are more sensitive on both price and latency.
Given these characteristics, you can adopt the following control strategies:
- Prefix caching. Put the system prompt, tool definitions, and long-lived knowledge first and keep them byte-for-byte stable, and put conversation history, timestamps, and retrieval results later, to raise the cache hit rate.
- Context compression. Every few rounds, summarize the history into “goal, completed steps, key findings, to-dos,” replacing raw dialogue with a structured state.
- Tool return-value governance. Truncate, paginate, or summarize return values, sending back only what the model needs for its next decision. When necessary, store raw data in object storage and keep only a reference URL in the context.
- Budget guardrails. Set a maximum number of steps, maximum tokens, and a spending cap for each task, degrading to human intervention once exceeded. You can enforce this uniformly in the LLM call client:
class BudgetExceeded(Exception):passclass BudgetGuard:"""Plug into the LLM call client to track and enforce a per-task budget"""def __init__(self, max_tokens: int, max_cost: float):self.max_tokens, self.max_cost = max_tokens, max_costself.tokens = self.cost = 0def charge(self, usage: dict) -> None:self.tokens += usage.get("in", 0) + usage.get("cache", 0) + usage.get("out", 0)self.cost += task_cost(usage)if self.tokens > self.max_tokens or self.cost > self.max_cost:raise BudgetExceeded(f"budget exceeded: tokens={self.tokens}, cost={self.cost:.4f}")# Usage: guarded = BudgetGuard(max_tokens=200_000, max_cost=0.10)# guarded.charge(usage) # run after each model call returns
- Async and queues. Use a message queue to smooth peaks and valleys, and route non-real-time tasks to a batch channel to complete them at a lower unit price.
7. A Capability–Cost Decision Framework in Four Steps
Step 1: Profile the tasks. Layer requests by business share—for example, simple Q&A, complex reasoning, code generation, document processing, and multi-round tool calls. Then extract 100–300 samples per layer that reflect the real distribution to form an internal evaluation set.
Step 2: Build a dual-axis “quality × cost” evaluation. Run all candidate models through the same task set, logging tokens, cost, latency, pass/fail, and retry count per item, then compute average cost per task, P90 cost, and first-pass rate. For the decision, refer to the general situations in the table below and set weights based on your own task mix:
| Task profile | Primary concern | Secondary concern | Typical optimizations |
|---|---|---|---|
| High-frequency, short input and output | Latency, throughput | Unit price | Small-model routing, semantic cache, batch API |
| Long-document understanding | Input unit price, long-context tier | Cache hit, extraction accuracy | Cache reuse, segmented summarization, retrieval augmentation |
| Code generation and review | Output quality, first-pass rate | Output unit price, artifact billing | Output-format constraints, automated unit-test validation |
| Multi-round Agent | Total cost per task, P90 latency | Stability, budget control | Caching, context compression, tool return-value governance |
Step 3: Build a regression mechanism. After a model upgrade, prompt change, or shift in business traffic, re-run the task set and compare quality–cost metrics. An iteration where quality holds and cost drops is real cost reduction.
Step 4: Make cost observable. Log the model version, per-category token counts, cost, latency, and task outcome for every call, and wire it into monitoring, alerting, and logging services. Only by breaking down cost by business line, scenario, and user can you locate where the money goes.
Additional considerations:
- Stability: watch rate-limit policies, service availability, and version sunset cycles, so you don’t take on business-interruption risk to save a few cents.
- Data compliance: when sensitive data is involved, prefer services that support not training on your data and offer enterprise-grade data-protection terms.
- Pricing transparency: whether billing is clear and whether prices stay stable long-term is also part of long-term cost.
- Implementation path: on cloud platforms, you can build the application chain from general cloud products such as model services, object storage, a message queue, a container platform, a logging service, and monitoring/alerting. Keep the model-call layer abstracted behind a stable interface so you can quickly run regression checks when upgrading or switching.
8. Conclusion: From Competing on Capability to Competing on Efficiency
Judging by the trend of “unit price flat, capability climbing,” the capability gap between top models is narrowing, and the focus of competition is shifting from “who is stronger” to “at the same level of intelligence, who can complete more tasks at lower cost and with greater stability.”
For enterprise developers, that’s actually good news: negotiating room comes from optimization, not from waiting for price cuts. Grasp three main threads and you can turn model upgrades into real gains:
- First, replace the token unit price with task-level metrics. Cost per task, first-pass rate, and P90 cost are the numbers that tie directly to business outcomes.
- Second, watch output bloat and cache hit rate. These are the most easily overlooked and often the most effective cost levers.
- Third, build budget guardrails for Agents. Cost grows roughly quadratically with rounds and context, and an unbounded long task can spin out of control at any time.
In the end, model selection shouldn’t be a one-off ranking game of “pick whoever scores higher,” but an ongoing engineering practice: let your own data do the talking, so every upgrade is backed by evidence.
