Choosing LLM Models by Cost-Performance: Reasoning Effort Tiers and Token Cost

1. Start With the Applicability Boundary

When many teams select a model, the first instinct is to look at leaderboards: composite intelligence index, coding benchmarks, whether math scores set a new record. Leaderboard scores are useful, but they usually answer only “can the model get this right?” They do not simultaneously answer “how much does one attempt cost?”, “how long does the user wait?”, or “will retries double the bill?”

For token- or call-metered LLM APIs, agentic tasks, long-chain coding tasks, document processing, and multi-turn Q&A, the combination that actually matters is three quantities: task success rate, end-to-end latency, and cost per unit of task. The method in this article mainly applies to online APIs billed per token or per call, and to tasks that call models repeatedly while orchestrating tools or sub-agents. For self-hosted deployments, fixed license fees, or dedicated resource pools, the cost structure is different and these formulas do not transfer directly.

Common questions in this space include: why does the bill go up after the unit price drops? How should reasoning effort tiers be chosen? How do you evaluate token cost? The rest of this article answers them with a computable cost model.

2. The Cost Model: Unit Price Is Only Half, Token Volume Is the Other Half

Metered LLM calls decompose into a fairly simple formula:

Cost per call = input tokens × input price
              + output tokens × output price
              + optional fees (caching, retrieval, tool calls, plugins, audit storage, etc.)
Cost per task = sum of all turn costs + cost of failed retries

Many teams focus only on “unit price” and overlook the other key variable in this formula: token volume. When reasoning effort increases, the model generates more intermediate reasoning content. Whether that content is billed, and how, must be taken from the vendor’s official billing documentation. Vendors differ in how they treat “chain of thought,” “intermediate reasoning,” and “visible answer”: some count intermediate reasoning as output tokens, some bill only the final visible output, and some offer separate discounts for cache hits or batch requests. You therefore cannot apply the assumed numbers in this article to any real product; the numbers here illustrate the calculation mechanism only.

The table below uses hypothetical values to show how reasoning effort affects output cost. Note that it covers output token cost only and excludes input tokens, caching, retrieval, tool calls, failed retries, and other add-on fees.

Reasoning effort tier	Avg. output tokens/task	Output price (per 1M tokens, assumed)	Avg. output cost per task (output only)
Low	40K	40	1.60
High (default)	120K	40	4.80
Maximum	274K	40	10.96

Suppose a vendor cuts the unit price by 20% across the board. If the task stays at the high tier, cost drops. But if you raise the tier from “high” to “maximum” in pursuit of accuracy, output tokens more than double and output cost can still rise substantially. In other words, a price cut lowers the overall height of the curve, while raising the tier changes where you sit on that curve.

Beyond reasoning effort, two amplification factors are easy to miss.

The first is the cost of failed retries. At high reasoning effort, failed samples have often already consumed a large number of tokens. If the failure rate is around 10% and each failed task is retried only once, the added cost of retry calls is roughly 10% of the original task cost. But the actual increase in the total bill also depends on the failure rate, the input-to-output ratio, whether failed samples are longer, whether retries escalate to a higher tier, and whether context is fed in again.

The second is context accumulation cost. In multi-turn agentic tasks, history messages, tool returns, retrieval results, and sub-agent conclusions are carried into subsequent requests repeatedly. Without compression, trimming, or summarization of context, input tokens grow roughly linearly with the number of turns. The cost structure of a long task can shift from “output-dominated” to “input-dominated,” and comparing output unit prices alone will then lead you astray.

3. What Reasoning Effort Actually Changes: Marginal Returns on Test-Time Compute

Vendors use different names for reasoning effort: thinking intensity, effort level, budget tier, reasoning budget, and so on. Their essence is similar: they control how much extra computation the model invests before answering. This knob has some stable characteristics.

1. Accuracy rises with tier, but with diminishing returns. Low tiers are usually enough for structured tasks such as classification, extraction, format conversion, and simple Q&A. Competition math, complex code refactoring, cross-file defect localization, and solution reasoning are the tasks that more often benefit from higher tiers.

2. Latency usually grows faster than accuracy. The maximum tier significantly increases both time-to-first-byte and total completion time. Interactive scenarios, real-time customer support replies, and mobile apps typically cannot absorb that much inference wait.

3. The default tier reflects the vendor’s cost-performance judgment, but still needs measurement. The default is normally set where accuracy and cost are reasonably balanced. When the default tier already meets your business acceptance bar, and moving up the tiers clearly increases output tokens, latency, or failure rate, the default tier is often a better deal than the maximum tier. This is not automatic, though; it should still be validated against your own eval set.

4. Not every task needs a high tier. For classification, extraction, short-text rewriting, and single-file patches, the accuracy gap between the low and maximum tiers can fall within noise, while the cost gap can be several times, or more than ten times, larger.

The engineering conclusion is clear: reasoning effort should be a per-request parameter, not a constant hard-coded in your code. Different tasks, different users, and different budget windows should use different tiers.

4. How to Measure Unit Cost: From Leaderboard Scores to Business Metrics

Leaderboard scores answer only “can it get this right?” A production selection decision also needs cost, latency, and success rate in the same table. At minimum, define these metrics.

• Success@1 rate: the share of tasks a model passes on the first attempt over your own eval set. Be careful not to confuse it with pass rates after best-of-n, multiple voting rounds, or human filtering.
• Cost per successful task: total spend divided by the number of successful tasks. Expressed as a ratio, it can be written as successful tasks per unit of currency, i.e., “successes per dollar.” The two forms must not be mixed.
• P95 end-to-end latency: a real experience metric that includes inference time, queueing time, tool call time, and network time. For batch offline tasks, use “average wall-clock time per task” instead.
• Token budget per successful task: used for capacity planning, quota requests, and peak load testing.

More important than the metrics themselves is calibration. Public leaderboards and real business distributions often diverge widely. Build an in-house eval set of 200 to 500 items covering real input lengths, real difficulty, and real failure modes. Sample it in layers by business scenario, for example short-context Q&A, cross-file code refactoring, long-document extraction, multi-turn tasks with tools, structured output, and hard defect localization. Look at the curve for each layer separately; do not average all scenarios into one composite score.

A practical approach is: fix the eval set, fix the prompts, fix the temperature, and scan only the reasoning tier. Run the same batch of samples tier by tier from low to high and record four numbers:

accuracy / average output tokens / cost per task / P95 latency

Plot the four numbers as curves and look for the “knee.” The knee typically appears where the accuracy curve starts to flatten but the cost curve is still climbing steeply. It is not fixed in absolute terms: it shifts with business tolerance for error, budget ceilings, and latency requirements. High-value tasks can set the knee, or slightly to the right of it, as the default tier, while bulk low-value tasks should stay as close to the low tier as possible.

5. Selection and Tiered Routing: Replace Single-Model Selection With a Portfolio

The biggest cost-performance gains usually do not come from “picking the single strongest model,” but from routing by task difficulty. A typical three-tier routing structure looks like this:

Request arrives
  ├─ Simple tasks: classification, extraction, intent detection, format validation  → low tier
  ├─ Medium tasks: code generation, document summarization, structured generation → default tier
  └─ Complex tasks: cross-file refactoring, hard defect localization, solution reasoning → high tier + budget guardrails

Four supporting techniques go with this.

1. Cascade fallback. First produce an answer with the low or default tier, along with a confidence signal. When confidence falls below a threshold, format validation fails, unit tests fail, or a business rule is violated, escalate to a higher tier. Escalation thresholds should not be set by gut feel; derive them from the confidence distribution, the validator’s false-positive rate, task value, and cost of failure, and validate with a small-traffic canary.

2. Front-load result validation. Code tasks can use compilation, unit tests, static checks, and diff review as objective gates. Data tasks can use schema validation, business rule checks, and uniqueness checks. Lowering the tier is only appropriate when the validator is reliable.

3. Context governance. In long tasks, history messages should not stack indefinitely. Apply summarization and compression, sliding windows, retrieval augmentation, tool-return truncation, structured summaries, and parent-child agent isolation. This directly reduces input-side token cost and also cuts interference from irrelevant information.

4. Caching and batching. If the prompt prefix is stable, enable cache hits. Offline batch jobs can use batch endpoints in exchange for a lower unit price or looser latency. Cache hits reduce repeated input cost and batching relieves interactive latency pressure, but both must be tracked separately on the cost dashboard.

6. Budget Guardrails: Turn Cost Into a Controllable Variable

The risk in agentic tasks is not that they are expensive, but that they are unpredictably expensive. Budget guardrails belong in code, not in manual monitoring. A minimal working example:

import time
class BudgetExceeded(Exception):
    pass
class BudgetGuard:
    def __init__(self, max_cost, max_output_tokens, max_steps, deadline_s):
        self.max_cost = max_cost
        self.max_out = max_output_tokens
        self.max_steps = max_steps
        self.steps = 0
        self.deadline = time.time() + deadline_s
        self.spent = 0.0
        self.out_used = 0
    def check(self):
        if self.spent >= self.max_cost:
            raise BudgetExceeded("cost")
        if self.out_used >= self.max_out:
            raise BudgetExceeded("tokens")
        if self.steps >= self.max_steps:
            raise BudgetExceeded("steps")
        if time.time() > self.deadline:
            raise BudgetExceeded("deadline")
    def record(self, in_tok, out_tok, price_in, price_out):
        self.spent += in_tok * price_in + out_tok * price_out
        self.out_used += out_tok
        self.steps += 1
guard = BudgetGuard(
    max_cost=1.0,
    max_output_tokens=200000,
    max_steps=3,
    deadline_s=60
)
try:
    for i in range(10):
        guard.check()
        guard.record(in_tok=1000, out_tok=500, price_in=0.000001, price_out=0.000004)
        print(f"Call {i + 1} completed, spent {guard.spent:.6f}")
except BudgetExceeded as e:
    print("Budget guard triggered:", e)

When designing guardrails, satisfy these four requirements at the same time.

1. All four gates active simultaneously: cost, output tokens, steps, and time. Any single gate can be bypassed by an anomalous loop. For example, some tasks produce few output tokens but carry an enormous input context, or take few steps but run for a very long time.

2. Degrade on limit breach rather than aborting. When the budget runs out, fall back to a lower tier and return “partial results + list of unfinished items + suggested next steps.” That is more valuable than an outright failure, especially for long-chain tasks.

3. Checkpoints must be resumable. Write intermediate artifacts at each key checkpoint to persistent storage, such as object storage, a database, or a task table. After a restart, resume from the most recent checkpoint and avoid paying twice.

4. Leave a trace throughout. Log the model, tier, input tokens, output tokens, latency, validation result, and escalation reason for every call. Feed those logs into monitoring, alerting, and a cost dashboard, and only then does cost become attributable.

7. Long-Running Autonomous Tasks: More Capability, More Governance

Having a model start from an empty directory, orchestrate multiple sub-agents, and run for days to build a complete piece of software — demos like that show the upper bound of what long-running autonomous execution can do. From an engineering perspective, what they really expose is not “whether the model can write code,” but orchestration, budgeting, acceptance, and permission governance.

First, cost amplifies non-linearly. The number of sub-agents, the number of rounds, and repeated context injection multiply together, and total cost can come out an order of magnitude above the initial estimate. Validate on a small scale before launch — 5% of samples, a one-hour window, or a single module — then extrapolate from measured data instead of turning everything on at once.

Second, failure modes can be silent drift. In long tasks, goal drift, duplicated work, agents overwriting each other, and error-fix loops often raise no error at all and show up only as degraded final quality. You need periodic human review gates, objective acceptance criteria, and a rollback-capable versioning mechanism.

Third, permissions must follow least privilege. Tasks that autonomously write code, call tools, and modify files should run in an isolated sandbox with only the necessary directories, network scope, and credentials. Access to production environments, customer data, and unredacted secrets should be prohibited. This is the baseline protection for automation whose behavior cannot be fully predicted.

Fourth, observability is a prerequisite. Task orchestration, queue smoothing, log aggregation, metrics collection, alerting, and audit trails form the common infrastructure for long-running tasks. You can compose container services, message queues, object storage, log services, and monitoring and alerting products on a general cloud platform to build a cost dashboard and audit chain, rather than depending on a single proprietary component.

8. Rollout Checklist and Monthly Review

A suggested rollout sequence:

1. Build an in-house eval set of 200 to 500 items, layered by business scenario, with fixed prompts, parameters, and evaluation scripts.
2. Scan the reasoning tiers and produce four curves — accuracy, average output tokens, cost per task, P95 latency — then locate the knee.
3. Use cost per successful task as the core comparison metric instead of unit price alone. If a ratio is needed, express it as successes per dollar.
4. Default to the vendor’s default tier; escalate only on validation failure, for high-value tasks, or on complex chains.
5. Enable the four-gate budget guardrails: cost, output tokens, steps, time; also turn on checkpoint recovery and full call logging.
6. Establish monthly review metrics: call share per tier, accuracy per tier, cost per task per tier, escalation hit rate, guardrail trigger count, cache hit rate, cost per successful task, and failed-retry share.

One-sentence summary: unit price determines where the price curve sits, reasoning effort determines which point on that curve you occupy, and routing, validation, and guardrails determine whether you can stay near the cost-performance knee over the long term. The selection goal is not “use the highest tier,” but “get acceptable results for the least money.”