Why Task Cost Can Rise After LLM Unit Prices Fall

Unit Price and Task Cost Are Different Metrics

Model APIs are usually billed by token, often as a price per million input tokens or per million output tokens. A price table is easy to read, but it describes only the price of one unit of usage. It does not describe what it costs to finish a piece of work.

What teams actually care about is how much it costs to complete a task. A task may include multiple turns, tool results written back into the prompt, retries after failure, and self-checks added to protect quality. Each of those steps consumes tokens, so unit price is only one term in the cost formula.

Single-task cost can be written as:

Single-task cost = Σ (input tokens × input unit price + output tokens × output unit price) / 10⁶

Cost normalized by successful tasks better reflects the tradeoff between quality and spend:

Cost per successful task = total cost / number of successful tasks

A cheaper model can still be more expensive per success if a high failure rate forces many retries. A more expensive model with a higher first-pass success rate can cost less per completed task.

Four Variables That Drive Single-Task Cost

1. Unit price. This is the easiest variable to compare, and the easiest to over-weight. A lower price only means the same number of tokens costs less. It does not mean the task needs fewer tokens.

2. Output token count. Output is usually more expensive than input, and output length affects quality in very different ways. Overly long output raises cost and latency, and it adds parsing work downstream.

3. Number of calls. Multi-step workflows, tool calls, retries, and corrections all increase the call count. One user request can map to several model calls, so cost has to be measured across the full path.

4. Context length. If each turn repeats prior context, input tokens accumulate with every round. This shows up clearly in long conversations and retrieval-augmented setups.

The four variables interact. A model that writes more detailed intermediate reasoning produces longer output, but if that reasoning cuts retries, total cost may not rise. Evaluation has to watch all four variables, not just one of them.

Why Reasoning Models Tend to Produce Longer Output

Stronger reasoning models often generate intermediate reasoning before the final answer. That content can include breaking the problem down, listing candidate approaches, checking steps, and revising earlier choices. Deeper reasoning usually means more intermediate text.

That creates a structural tradeoff. Asking a model to “think harder and check more branches” usually lengthens the output. Asking it to “be concise” can lower accuracy on hard problems. In many settings, those two goals cannot both be maximized.

Engineering should make that tradeoff explicit instead of assuming longer output is always more valuable. For simple Q&A, format conversion, classification, and labeling, extra reasoning rarely pays off. For multi-step planning, difficult code fixes, and hard data analysis, extra reasoning can cut the failure rate substantially.

Evaluate Cost on Real Task Samples

Public leaderboards and vendor claims are a weak guide to how a model will behave on your own workload. A more reliable approach is an internal evaluation set, built as follows.

Step 1: Build the sample set. Draw representative tasks from production logs, covering easy, medium, and hard cases. Each class should include typical successes and typical failures.

Step 2: Fix the evaluation conditions. Run every candidate model with the same prompts, the same tool definitions, and the same timeout and retry policy. Otherwise the comparison cannot be attributed to the model itself.

Step 3: Record tokens and outcomes for every call. The example below aggregates single-task cost into cost per successful task.

from dataclasses import dataclass
@dataclass
class CallLog:
    task_id: str
    input_tokens: int
    output_tokens: int
    succeeded: bool
def cost_per_success(logs, price_in_per_m, price_out_per_m):
    """
    Compute cost per successful task.
    price_in_per_m: price per million input tokens
    price_out_per_m: price per million output tokens
    """
    total_cost = 0.0
    success_tasks = set()
    all_tasks = set()
    for log in logs:
        all_tasks.add(log.task_id)
        total_cost += (
            log.input_tokens / 1_000_000 * price_in_per_m
            + log.output_tokens / 1_000_000 * price_out_per_m
        )
        if log.succeeded:
            success_tasks.add(log.task_id)
    if not success_tasks:
        return float("inf")
    return total_cost / len(success_tasks)

The function counts successes by unique task ID, so multiple successful calls for the same task are not counted more than once.

Step 4: Compare the full ledger. Useful dimensions include cost per successful task, average latency, first-pass success rate, and time spent on human correction. Only the combination of these metrics shows whether a model is actually cheaper.

A Worked Example with Hypothetical Numbers

The figures below are illustrative. They show the calculation method only and do not represent any real product.

Assume model A charges 10 currency units per million output tokens, averages 100K output tokens per task, and has an 80% first-pass success rate. Model B is 20% cheaper on output, at 8 currency units per million tokens, but averages 200K output tokens per task and has a 95% first-pass success rate.

• Model A output cost per task: 100K × 10 / 10⁶ = 1.00; cost per successful task: 1.00 / 0.80 = 1.25
• Model B output cost per task: 200K × 8 / 10⁶ = 1.60; cost per successful task: 1.60 / 0.95 ≈ 1.68

In this hypothetical, model B’s unit price is 20% lower, yet single-task cost is 60% higher, and cost per successful task is still higher. If the success-rate gap were smaller, the conclusion could reverse. Comparisons have to be based on measured data.

Engineering Practices

Set an output budget. Cap maximum output tokens by task type, and log an alert when a call exceeds the threshold. Set the budget from percentiles of historical usage, not from a guess.

Route by difficulty. Send simple tasks to a cheaper model or a rules engine, and escalate hard tasks to a stronger model. Routing can depend on task type, input length, or the result of an upstream classifier.

Cut repeated context. Cache stable system prompts and knowledge snippets so they are not billed again on every turn. Summarize conversation history to keep input size under control.

Constrain the output format. Ask the model to return a fixed structure, such as JSON or a specified delimiter. That reduces irrelevant output and makes parsing easier.

Monitor cost. Write token counts, latency, and outcomes for every call to a logging service, and aggregate by task type, model version, and date. Alert on period-over-period changes in cost per successful task so a version upgrade that shifts cost is visible quickly.

How to Read Capability Results

First, model versions change often, and behavior under the same name can change with the version. Re-run the evaluation set after every upgrade.

Second, public benchmarks and production work are not the same. Benchmark items are usually well structured and clearly bounded. Real tasks include vague requirements and noisy data.

Third, some services apply different capability ceilings and usage limits by use case or by model tier. When you choose a model, confirm the actual capability and quota of the version you will run, not only the marketing description.

Fourth, both cost and quality are time-bound. Prices, limits, and results change. Conclusions should be reviewed on a regular schedule.

Model Selection Checklist

• Did you test on real task samples, rather than reading only the price table?
• Did you measure cost per successful task, including retries and human correction?
• Did you record the distribution of input and output tokens, not only the averages?
• Did you design tiered routing for tasks of different difficulty?
• Did you set output budgets, caching, and cost alerts?
• Did you re-run the evaluation set after the model version changed?

Once those questions are answered, a team can judge cost-effectiveness from its own data. Unit price is only the starting point. The ledger for completed work is what decides the choice.