Don't Choose LLMs by Benchmarks Alone: Reasoning Tiers and Token Cost

Token usage per task, not list price, determines whether a model is cheaper

When choosing a model, many teams look at leaderboard scores first, then at the price per million tokens. Both numbers are useful, but neither answers the real question: how much does it cost to finish one job? Newer models often change three things at once: capability, how many tokens they consume at inference time, and unit price. The combination of all three sets the cost of a single task.

This article uses the launch of a recent reasoning model as context for how to evaluate those changes, and it outlines selection steps you can run yourself. Figures below are examples or vendor-claimed results. Before you switch models, verify against official pricing pages and samples from your own workloads.

A lower unit price does not mean a cheaper task

Reasoning models often generate internal thinking before the final answer, and that content is billed as output tokens. The longer the model thinks, the more output tokens you pay for. Even if the price per million tokens drops, total cost can still rise if output volume per task grows a lot.

Suppose output unit price falls 20% while output tokens for the same task rise 132%. A simplified cost formula is:

task cost ≈ (input tokens × input price + output tokens × output price) / 1,000,000

A 132% increase in output tokens means 2.32× the previous output volume (1 + 1.32). Unit price becomes 0.8×. Multiplying:

2.32 × 0.8 ≈ 1.86×

Output cost alone is about 1.86× the original, before any change in input tokens. You must look at token volume and unit price together; using only one of them leads to the wrong conclusion.

How to choose a reasoning tier

Many reasoning models expose several reasoning-strength tiers. Higher tiers usually think longer, use more tokens, and produce better results, but quality and cost rarely scale linearly. Some workloads already meet the quality bar at a mid tier; going higher only adds spend.

In practice, compare three cases:

1. Different tiers of the same model: is the score gain worth the extra cost?
2. A mid tier on a new model versus a high tier on the previous generation: which is better value?
3. Score gaps across models at the same cost.

Run these comparisons on your own tasks, not by copying leaderboard numbers. Benchmark item mix often differs from production work.

For most production jobs—code completion, document summarization, format conversion, and routine Q&A—the default or a lower tier is often enough. High tiers are more justified for complex multi-step planning, cross-file refactors, and architecture design.

The following is routing pseudocode to illustrate the idea. Use the actual parameters from your platform’s documentation:

def route_task(task, budget_limit=0.5):
    # Pseudocode: pick a reasoning tier from task complexity and cap per-task spend
    level = {"high": "max", "medium": "high"}.get(task.complexity, "low")
    try:
        result = model.call(task, reasoning=level, timeout=120)
    except TimeoutError:
        # Drop one tier on timeout so the request does not block for long
        result = model.call(task, reasoning="low", timeout=60)
    cost = result.usage.total_cost
    log_usage(task.id, level, result.usage.output_tokens, cost)
    if cost > budget_limit:
        alert(f"Task {task.id} exceeded budget: {cost:.2f}")
    return result

Three points matter: route by complexity, degrade after timeout, and log tokens and cost per task.

Coding: measure marginal cost with concrete metrics

Coding is one of the most competitive model use cases and one of the easiest places to pick the wrong tier. Score gains at the highest tier often come with a clear cost increase.

Ask a specific question: how much extra does each task cost per additional point of score? Can that extra spend be offset by saved human time? If generated code still needs heavy review and rework, chasing scores can reduce overall efficiency. In high-safety or highly automated pipelines, a higher first-pass rate can cut review and rollback cost.

Teams should track two metrics for coding: first-pass rate (share of code that passes tests with no edits) and average cost per passing task. Together they say more than scores alone.

Multi-agent demos: separate claimed capability from verified results

The launch included a multi-agent demo: starting from an empty directory, a main agent split work, sub-agents wrote code, tested, and reviewed, and the system produced a small OS prototype. That shows a possible direction for multi-agent collaboration, but it is a vendor demo and has not been independently reproduced. Repeatability, total cost, and failure rate still need separate verification.

Teams planning production multi-agent systems can use these engineering checks:

1. Task decomposition: the main agent’s split should be inspectable so subtasks do not conflict.
2. Continuous verification: compile and test each sub-agent’s output immediately and record failures.
3. State management: persist progress and context for long jobs; do not rely only on a single context window.
4. Independent review: keep a review agent separate from generation; keep humans on critical modules.
5. Cost and safety bounds: set per-task budget caps, permission scopes, and sandboxes so agents cannot act outside policy.

Containers, message queues, logging, and alerting help you build orchestration and observability around that loop.

A five-step selection framework

You can turn the above into an evaluation process:

1. Map the workload: measure complexity mix and which tasks truly need high tiers.
2. Measure cost per task: run real samples, record input and output token distributions, and cover at least one full business cycle.
3. Compare value: plot score versus per-task cost for each model and tier and find the better combinations.
4. Check quality stability: run the same task multiple times and watch variance; do not trust a single run.
5. Design fallback and review: set tier, budget, and human review by risk so spend stays bounded.

Sample size matters. With only a few dozen samples, average cost can swing widely. Use a large enough sample, compute a confidence interval on the mean, then decide whether to switch.

Closing

Newer models keep raising reasoning quality, but cost is harder to read from list price, leaderboard scores, or demo videos. What matters is data from your own work: tokens per task type, quality gain per tier, and the cost of failure.

Put task routing and cost logging in place first, then choose the model and tier. Selection becomes evidence-based, and later model updates are easier to compare.