Choose LLMs by Total Task Cost, Not Token Price

Separate the two ledgers

When teams connect large language models, price lists are easy to over-weight: a drop in input or output unit price is treated as “cheaper.” For developers and owners, the number to manage is the total cost of completing a clearly defined task, not the sticker price of a single call.

That total has at least four parts:

• Read cost: System prompts, specs, codebase snippets, history, and retrieval results are billed as input.
• Write cost: Final answers, intermediate traces, and tool arguments are usually more expensive per token than input.
• Failed retries: Wrong answers, invalid formats, or failing tests send the same context again.
• Orchestration overhead: Multi-agent splits, state passing, and human review raise both time and call count.

A higher unit-price model can still be cheaper if it takes fewer detours and fewer reworks. A cheaper model can cost more if it retries often and contexts keep growing. First principle: account by task, not by ranking the price list.

Write the formula on a single call

The numbers below are illustrative, not vendor quotes. Let (P{in}) and (P{out}) be input and output unit prices, (T{in}) and (T{out}) tokens read and written per attempt, and (N) the average attempts until acceptance:

[
C \approx N \times (T{in}\cdot P{in} + T{out}\cdot P{out}) + C_{orch}
]

(C_{orch}) covers scheduling, queues, logs, and sampling review that never fully appear on the token invoice.

Example scale: a code change reads 100k tokens and writes 10k. If a high-capability tier is about twice the unit price of a workhorse tier, one call costs about twice as much. If the workhorse needs three attempts to pass tests and the high-capability model passes once, token spend can reverse. The comparison is valid only under the same acceptance bar: tests pass, output is parseable, key facts can be checked.

A “percent price cut” describes rate change, not bill change. If rates fall 20% but output length or retries rise 30%, the bill still grows.

Why “stronger” can be cheaper

On complex work, waste is usually not “thinking deeper.” It is:

1. Unclear goals, so plans keep changing and long context is reread every round.
2. Tool output is not digested; logs are stuffed back into the next turn.
3. One generation is too long, then rewritten in full.
4. Fuzzy subtask boundaries, so agents duplicate work.

Stronger models often pay for themselves by making fewer of these mistakes: earlier convergence, shorter useful output, fewer empty tool loops. Vendor “cost per task” curves are worth reading as hypotheses, not as buying criteria. An internal eval set, fixed prompts, and logged pass rates and tokens track your bill better than external leaderboards.

The reverse also holds: on simple tasks, high-capability models do not shorten output in proportion. Weekly-note polish, format conversion, and small single-file edits have similar length and few retries; extra unit price is mostly premium. Use a light tier or a rules script.

Tier by the job, not “strongest by default”

Split work into three bands and write them as team defaults.

Light: short, checkable, reversible.
Email drafts, meeting notes, style tweaks, a couple of unit tests, tables to fixed JSON. Acceptance is clear and mistakes are cheap. Default a light model; add templates and validators before paying for a larger model.

Workhorse: most engineering and knowledge work.
Edits across a few files, API notes, first-pass incident hypotheses, mid-length retrieval Q&A. Complete prompts, retrieval, and tests before upgrading the model. Many “the model is bad” cases are incomplete context or acceptance criteria never written into the prompt.

High-capability: long horizon, strong coupling, expensive rework.
Work that splits into many subtasks, runs for hours, and must keep intermediate state consistent—large refactor planning and execution, multi-module protocol alignment, drafts that need strict formality or high-risk decision support. Paying for a higher first-pass rate is reasonable, but set budget caps and human gates so agents do not burn tokens while spinning.

If a platform suggests starting daily work on the workhorse and switching up for hard jobs, treat that as a reasonable engineering default. Encode it in a gateway or router, not in ad-hoc human choice.

Long-running agents: cost is loss of control, not unit price

Continuously scheduling many subtasks from an empty tree for hours is where model gaps show—and where bills blow up. Controls are brand-agnostic:

• Stoppable task graphs. Each subtask has done criteria, timeouts, and max retries; failure bubbles up instead of infinite re-split.
• Trimmed context. Pass conclusions, diffs, and failure summaries, not full traces every time.
• Least privilege. Separate write, command, and external access so one bad call cannot expand damage.
• Recoverable checkpoints. Persist long jobs by stage; resume from a checkpoint instead of paying input from scratch.
• Hard budgets. Token or dollar caps per task; pause and hand off to a human at the limit.

Demos that “boot a prototype” or “draft a formal proof” show an upper bound on planning and tools. They should not become the default route for everyday scripts. Use them as capability probes, not default routing.

How to read evals without being misled by a single score

When you look at external evals, read at least four columns, not accuracy alone:

Dimension	Question to ask
Task definition	Close to your real tickets, or public Q&A?
Settings	Reasoning strength, temperature, tools aligned with production?
Cost accounting	Single call, or cumulative cost until pass?
Variance	Enough samples, or failures averaged away?

When scores are close, a 2× gap in cumulative cost matters more for budget than a few tenths of a point of accuracy. If pass rates differ a lot, token savings can be eaten by production incidents. Knowledge exams, coding benches, and remote-work tasks measure different skills; one score should not set company-wide routing.

Vendor self-tests are directional. If reproduction conditions are incomplete, treat them as reference only. Ten to thirty internal tickets as a fixed set, run weekly, logging pass rate, P95 latency, average tokens, and human edit lines, is more stable than chasing leaderboards.

Ship it: routing, observability, prompt hygiene

Routing. Use task labels instead of letting users pick “strongest.” Examples: format, edit_small, debug, plan_large. Map labels to tiers, allow human override, and audit overrides.

Observability. Persist at least: task id, model tier, input tokens, output tokens, retry flag, acceptance, latency. Aggregate by task id so you can answer “what did this job cost.” Account totals cannot tell you whether to downshift or fix the prompt.

Prompt hygiene. Keep stable specs on the system side and bound their length; inject only relevant retrieval; ask for a change plan and verification steps before a patch; emit machine-readable structure and, on validation failure, resend only the error locus, not the whole corpus.

Illustrative cost rollup (pseudocode; units are yours):

def task_cost(attempts):
    total = 0
    for a in attempts:
        total += a.input_tokens * PRICE_IN[a.tier]
        total += a.output_tokens * PRICE_OUT[a.tier]
    return total
# Selection: among tiers that meet the pass-rate bar, take min task_cost

Keep the price table in config, updated with the contract, not hardcoded. Compare tiers on the same attempts and the same pass standard, or the comparison is invalid.

Practical takeaways for individuals and teams

Individuals: weekly notes, slide edits, public lookup, short scripts—use a sufficient tier. Making the most expensive tier the default usually means paying a high unit price for low-failure-cost work.

Engineering teams: lock acceptance, tests, and context trimming before debating an upgrade. The case for upgrade is that pass-rate or rework-time gains cover the unit-price gap, proven on task-level bills.

Procurement and platform: contracts should cover rates, rate limits, and how long context is actually billed (cache hits, batches, tool results). Those rules change (T_{in}) and (N) more than a 20% rate cut.

One line: the price list answers the rate; the task bill answers value. Start daily work on a sufficient tier, reserve high-capability models for long, hard, expensive-to-rework jobs, and keep correcting with pass rate and cumulative tokens.