Choosing LLMs by Task Cost, Not Unit Price

Unit price is not the same as cost

Large language models are usually billed by token, with separate rates for input and output. Output is typically more expensive. When a new version drops its list price, teams often assume it is cheaper. That judgment looks at only one variable.

The figures below are hypothetical and do not match any real pricing. Suppose a task reads 100,000 input tokens and generates 10,000 output tokens. If model A charges 1 per million input tokens and 4 per million output tokens, one call costs:

100000 × 1 ÷ 1,000,000 + 10000 × 4 ÷ 1,000,000 = 0.14

The same task on a higher-priced model B might cost 0.45 per call. On a single call, A looks more than twice as cheap. In production, one-call cost is only the starting point. You also need to know whether the task finishes in one attempt.

Measure value with cost per successful task

Cost per successful task has three parts:

1. Cost of one call: driven by input tokens, output tokens, and the matching unit prices.
2. First-pass rate: the probability that the first result is usable, measured on an evaluation set.
3. Rework cost: tokens and time spent on retries, human edits, and extra calls after a failure.

If a failure means another independent call, the expected cost of one successful task is:

expected cost = cost of one call ÷ first-pass rate

If the pass rate rises from 0.80 to 0.95 at the same per-call cost, expected cost falls by about 16% (the ratio of 1 ÷ 0.80 to 1 ÷ 0.95 is about 0.84). Quality gains that cut rework can therefore offset a higher unit price.

Two caveats. First, independence often fails in real work: repeating the same input that already failed may not help. Second, some models emit extra reasoning tokens; follow the provider’s official documentation and include those tokens in the estimate.

Choose a tier by task difficulty

Do not pick a model only by name or version. Classify difficulty and risk first, then match a tier. A common split:

Task type	Typical traits	Suggested tier	Why
Copy editing, format conversion, summaries	Clear rules, short context	Small	Quality gaps across tiers are small; cost matters more
Simple scripts, single-file edits	Basic logic; errors are easy to spot	Small or mid	A higher mid-tier pass rate can cover the price gap
Multi-file code changes, complex analysis	Cross-context reasoning	Mid	Balance of quality and cost
Long pipelines, multi-subtask orchestration	Many steps; errors accumulate	Large, with a budget cap	Fewer mid-run failures and less rework

This only works if you label tasks. Tag common jobs, measure each class on each tier, and converge on default routing rules.

Controlling cost on long jobs

For long jobs, price is not the only risk. Each step’s output becomes the next step’s input, so context grows and token use compounds. Common controls:

• Split subtasks: cap each subtask’s context and pass only what it needs.
• Set ceilings: max output tokens and max calls per subtask.
• Reuse intermediates: cache reusable artifacts so the same material is not reread and reprocessed.
• Keep a ledger: record cost, latency, and status per subtask to find waste.
• Global budget: pause when a threshold is hit and ask a human whether to continue.

Parallel scheduling can cut wall-clock time but also multiplies token spend. Teams should choose speed versus cost based on business priority.

Build your own evaluation baseline

Public leaderboards measure general skill and may not match your workload. A more reliable approach:

1. Sample 50–200 real tasks covering typical and edge cases.
2. Freeze the set and the scoring rules. Scoring can be automated tests or two independent human raters compared.
3. For each candidate tier, record pass rate, average input and output tokens, latency, and human rework count.
4. Plug the numbers into the expected-cost formula and choose against your quality bar.

The following snippet shows how to compute expected cost per tier. Replace the parameters with your own data:

from dataclasses import dataclass
@dataclass
class ModelTier:
    name: str
    input_price: float   # price per million input tokens; same currency as the budget
    output_price: float  # price per million output tokens
    pass_rate: float     # first-pass rate on the eval set, between 0 and 1
def single_call_cost(tier: ModelTier, input_tokens: int, output_tokens: int) -> float:
    return (input_tokens * tier.input_price + output_tokens * tier.output_price) / 1_000_000
def expected_cost_per_success(tier: ModelTier, input_tokens: int, output_tokens: int) -> float:
    if tier.pass_rate <= 0:
        raise ValueError(f"{tier.name} pass rate must be greater than 0")
    return single_call_cost(tier, input_tokens, output_tokens) / tier.pass_rate
if __name__ == "__main__":
    # Hypothetical values for demonstration only, not real prices or test results
    tiers = [
        ModelTier("small", input_price=1.0, output_price=4.0, pass_rate=0.80),
        ModelTier("mid", input_price=3.0, output_price=15.0, pass_rate=0.90),
        ModelTier("large", input_price=8.0, output_price=40.0, pass_rate=0.95),
    ]
    for tier in tiers:
        cost = expected_cost_per_success(tier, input_tokens=100_000, output_tokens=10_000)
        print(f"{tier.name}: expected cost {cost:.3f}")

With those assumptions, the small tier has the lowest expected cost, about 0.175; mid is about 0.500; large is about 1.263. The ranking depends on pass rate. If the small tier’s pass rate falls to 0.40, its expected cost rises to 0.35 and the order changes. Selection must rest on your own data and be remeasured regularly.

Tiered routing and escalation

Once evaluation data is solid, default to the cheapest tier that meets the quality bar for that task class, and escalate only when the first pass fails or the job is high risk. Keep a budget cap on long pipelines so spend cannot run away.