LLM API Cost Accounting: Why Cost per Task Matters More Than Token Price

    New model generations keep getting more capable, yet their API prices often stay flat at the previous generation's tiers. That looks like free upside, but it can lead developers to an overly optimistic conclusion. Token price per million tokens is only the denominator: what actually determines a bill is how many tokens a complete task consumes. This article explains why cost per task is the metric worth tracking, the four variables that inflate or deflate it — output token weight, prompt cache hit discounts, retries and failed calls, and context growth across multiple turns — how to build a reproducible accounting model with sample Python code, and five vendor-neutral engineering practices for keeping per-task spend predictable.

    1. From price per token to cost per task: a shift in perspective

    A pattern worth watching has emerged in the industry: new generations of large models show a clear jump in general reasoning and capability, but the API pricing they are offered at has not moved up. Input and output still sit in the same tiers as the previous generation. On the surface, that is a straightforward win — more capability at the same price. But if you stop at “how much per million tokens,” it is easy to walk away with an overly optimistic conclusion.

    The reason is that token price is only the denominator in the equation. What actually determines a business’s spend is the total number of tokens consumed to complete one full task. When a model becomes stronger and more willing to “say more and think more,” the unit price stays flat while the same job may consume more output tokens — pushing the real bill up. So judging whether a model is worth the money requires stepping out of the unit-price view and back to cost per task, a metric much closer to how the business actually operates.

    This article follows that thread: how to measure cost per task properly, which variables inflate or shrink it, and how to use engineering practices to keep it within a predictable range.

    2. Why cost per task matters more than token price

    Imagine two models, A and B, priced almost identically. Model A is terse and usually returns a usable answer on the first pass. Model B is talkative and produces longer reasoning and explanations for the same job. Compared on price alone, they look even. Measured against a real invoice, after one hundred tasks B may cost noticeably more.

    That exposes the key logic:

    • Unit price sets the price of each unit of information; cost per task sets the real expense of getting something done.
    • Output tokens are typically priced higher than input tokens, and reasoning-heavy work, code generation, and multi-turn agent calls are exactly where output tokens pile up.
    • A more meaningful way to compare intelligence per dollar is to introduce a metric like cost per task: run the same representative task set through each model, measure total consumption for each, then divide by the number of tasks to get a comparable per-task cost.

    The value of this view is that it merges two separate questions — “how capable is the model” and “can we afford it” — into one comparable number. When two models score similarly on general capability, the one with the lower cost per task usually has the advantage in engineering adoption and large-scale deployment.

    3. Four variables that inflate or shrink cost

    To actually control the bill, you need to understand what drives cost per task. These four variables are the ones most often overlooked, and their impact runs deep.

    3.1 The weight of output tokens

    Input and output are usually billed in different tiers, with output priced higher. As a result, the more a model outputs, the faster cost grows. A newer version of a model can raise output volume per task simply by reasoning more thoroughly and explaining more completely — and even with an unchanged unit price, total cost climbs. When assessing cost, track input and output tokens separately rather than looking only at the combined total.

    3.2 The discount effect of cache hits

    Most mature large-model services give a meaningful discount on cache hits. The mechanism: for repeated, long, stable context — system prompts, fixed documents, snippets of a long codebase — the serving side can reuse the part it has already computed and price that portion of the input substantially lower. Whether you hit the cache reliably directly determines cost in long-context scenarios. A well-designed fixed prefix, paired with a stable call order, often produces considerable savings.

    3.3 Retries and failed-call overhead

    Network jitter, rate limiting, and truncated responses all trigger retries. Every failed retry can consume the input tokens again, and may even produce output. Without sensible backoff and idempotency, a rising failure rate quietly inflates total cost.

    3.4 Context and multi-turn amplification

    In long-running agents or multi-turn conversations, each turn typically carries the history along. The more turns, the faster input tokens accumulate. Unless you trim, summarize, or structurally compress the history, cost per task grows roughly linearly — or faster — with conversation length.

    4. How to measure cost per task properly

    Rather than being led around by a price sheet, build your own accounting model. The sample code below shows how to price input, output, and cache hits separately, then roll everything up into a cost-per-task figure:

    1. from dataclasses import dataclass, field
    2. @dataclass
    3. class ModelPricing:
    4. # Unit: price per 1M tokens (placeholder values; replace with real tiers)
    5. input_per_mtok: float
    6. output_per_mtok: float
    7. cached_input_per_mtok: float # Cached input is typically cheaper
    8. @dataclass
    9. class TaskUsage:
    10. input_tokens: int = 0
    11. output_tokens: int = 0
    12. cached_input_tokens: int = 0 # Portion of input served from cache (usually a subset of input)
    13. retries: int = 0 # Number of failed retries, tracked to observe overhead
    14. def estimate_task_cost(pricing: ModelPricing, usage: TaskUsage) -> dict:
    15. # Cached tokens are billed at the discount rate; the rest of the input at standard rate
    16. billable_normal_input = max(0, usage.input_tokens - usage.cached_input_tokens)
    17. in_cost = billable_normal_input / 1_000_000 * pricing.input_per_mtok
    18. cached_cost = usage.cached_input_tokens / 1_000_000 * pricing.cached_input_per_mtok
    19. out_cost = usage.output_tokens / 1_000_000 * pricing.output_per_mtok
    20. return {
    21. "input_cost": round(in_cost, 6),
    22. "cached_input_cost": round(cached_cost, 6),
    23. "output_cost": round(out_cost, 6),
    24. "total_cost": round(in_cost + cached_cost + out_cost, 6),
    25. "retries": usage.retries,
    26. }
    27. def aggregate_cost_per_task(pricing: ModelPricing, usages: list[TaskUsage]) -> float:
    28. if not usages:
    29. return 0.0
    30. total = sum(estimate_task_cost(pricing, u)["total_cost"] for u in usages)
    31. # Note: to account for retry overhead, add the actual billed cost per retry here
    32. return total / len(usages)

    The key points:

    1. Price the parts separately: compute input, cached input, and output independently, or you cannot see the cost structure.
    2. Land on task granularity: sum usage across a batch of representative tasks, then divide by the task count for a comparable per-task cost.
    3. Account for overhead: to stay close to reality, add the actual billed cost of failed retries to the numerator so you do not understate spend.
    4. Use a reproducible evaluation set: compare models on a fixed set of “gold standard” tasks, otherwise the conclusion means little.

    5. Five engineering practices to bring cost down

    Once the accounting logic is clear, optimization has a direction. None of the practices below depend on a particular vendor; they are industry-standard approaches.

    • Keep reusable context stable and put it first: place system prompts, fixed knowledge, and long documents at the front of the request and leave them unchanged to raise the cache hit rate and earn the discount.
    • Control output length: explicit output format constraints, structured field requirements, and length caps cut down on unnecessary verbose responses.
    • Trim history: in multi-turn scenarios, use summarization, retrieval augmentation, or structural compression instead of carrying everything forward, which suppresses input growth per turn.
    • Tiered routing: send simple tasks to a lighter model or cheaper approach and reserve the stronger model for complex work, avoiding the waste of over-provisioned capability.
    • Robust retry strategy: use exponential backoff, idempotency keys, and sensible timeouts to cut needless repeat consumption; deduplicate repeatable requests locally.

    Combined, these practices typically reduce cost per task substantially without sacrificing delivery quality.

    6. Conclusion: the second half of the race is the cost of a unit of intelligence

    Looking back across the whole technical arc, the large-model industry is shifting its center of gravity. The first half was about “can it, and how strong is it,” measured by capability scores. Now, as top models converge into a similar tier, the real dividing line is moving toward “for the same intelligence, who can deliver it at lower cost.”

    For developers and enterprises, that means model evaluation cannot stop at the price sheet. Building an accounting and optimization system centered on cost per task — folding output tokens, cache hits, retry overhead, and context growth into one unified model — is what makes model selection rational, bills controllable, and large-scale deployment sustainable. Future cost efficiency will not be decided by whose numbers look best, but by who can push the real cost of getting one thing done lower. That is the engineering capability worth investing in next.