Reasoning Effort Tiers and Cost Accounting: A Practical Guide to LLM Selection

1. First, a common misconception: a lower unit price ≠ a lower cost

The billing structure of reasoning models makes “unit price” a metric with a very small denominator. Real spend is determined jointly by three variables:

1. Input token volume: the running total of prompts, retrieved passages, conversation history, and tool return values;
2. Output token volume: the final answer plus the intermediate tokens generated during the “thinking/reasoning” phase — most vendors bill the latter as output tokens;
3. Unit price and caching strategy: whether a prefix cache is hit and whether batch-processing discounts apply.

When a model trades longer internal reasoning for a higher task success rate, output tokens can balloon non-linearly, easily swallowing the savings from a lower unit price. The industry has repeatedly seen the “unit price down 20%, per-task cost doubled” pattern for exactly this reason: procurement decisions look at the unit price, but the financial outcome is per task.

Formalized, this can be written directly as a cost-estimation function:

from dataclasses import dataclass
@dataclass
class PriceTable:                 # unit: currency / million tokens
    input_price: float
    output_price: float
    cached_input_price: float | None = None   # input unit price on a cache hit, usually much lower
def estimate_cost(input_tokens: int,
                  output_tokens: int,
                  thinking_tokens: int = 0,       # reasoning-stage tokens, usually billed as output
                  cached_input_tokens: int = 0,
                  price: PriceTable | None = None) -> dict:
    fresh_input = max(input_tokens - cached_input_tokens, 0)
    billed_output = output_tokens + thinking_tokens
    cost = fresh_input * price.input_price / 1e6
    cost += billed_output * price.output_price / 1e6
    if price.cached_input_price is not None:
        cost += cached_input_tokens * price.cached_input_price / 1e6
    return {
        "cost": round(cost, 6),
        "billed_output_tokens": billed_output,
        "output_ratio": round(billed_output / max(input_tokens, 1), 2),
    }

The engineering takeaway is direct: whenever output_ratio is significantly greater than 1, you must compare models on “cost per task” rather than “price per million tokens.” Otherwise your selection conclusion will be systematically biased toward the more verbose models.

2. Reasoning effort tiers: a cost–quality curve you must model

Mainstream reasoning models generally offer graded reasoning effort (for example, low/medium/high/very high). This is essentially a cap on the model’s “thinking budget.” Three easily overlooked patterns apply here:

• Effort and token consumption are roughly superlinear: doubling a tier often multiplies reasoning tokens several times over, rather than increasing them linearly;
• Quality gains are subject to diminishing returns: moving from the lowest tier to a mid-to-high tier usually yields the steepest improvement; maxing it out adds only a few points while the cost multiplies;
• The inflection point shifts between task families: structured extraction and intent classification saturate at very low tiers, whereas multi-step code edits and long agent chains only see meaningful gains at high tiers.

There is therefore no “globally optimal tier” — only an “optimal tier per task family.” A sample measurement framework for locating the inflection point looks like this (the numbers are illustrative assumed data and do not represent any specific product):

Reasoning Effort	Relative Output Tokens	Task Success Rate	Per-Task Cost Index	Marginal Gain
Low	1.0×	78%	1.0	—
Medium	2.6×	88%	2.4	+10pt / +1.4 cost
High	5.4×	93%	4.8	+5pt / +2.4 cost
Very High	9.1×	94%	8.3	+1pt / +3.5 cost

The conclusion is clear: the default tier often sits at “medium” or “high,” and blindly maxing out every request is the most common waste. A reusable default policy is to have a low tier handle the bulk of traffic, and to reserve mid-to-high tiers for failed or high-value samples.

3. Replace leaderboard scores with an internal evaluation set

Third-party comprehensive benchmarks are an effective filter, but they measure a general average that will almost certainly be misaligned with your business distribution. Production selection needs a self-built, reproducible evaluation pipeline:

1. Stratified-sampled golden set: partitioned by business task family (such as Q&A, extraction, generation, code edits, and multi-turn tool calls), with 100–300 items per stratum, covering both typical and difficult samples;
2. Metrics beyond just accuracy: also record pass@1, human acceptance rate, retry rate, P50/P95 latency, per-task token distribution, and per-task cost;
3. Treat effort as an experimental dimension: run the same model across tiers to obtain the curve above, rather than taking a single configuration;
4. Fix randomness and versions: pin the prompt templates, temperature, and context-trimming strategy; rerun the baseline after any single change.

The core structure of an evaluation runner is “one call = one complete record”:

def evaluate(model, effort: str, samples: list[Sample]) -> Report:
    records = []
    for s in samples:
        t0 = time.perf_counter()
        resp = client.chat(model=model, effort=effort, messages=s.prompt)
        latency = time.perf_counter() - t0
        usage = resp.usage                      # unified usage structure
        cost = estimate_cost(
            input_tokens=usage.input_tokens,
            output_tokens=usage.output_tokens,
            thinking_tokens=getattr(usage, "thinking_tokens", 0),
            cached_input_tokens=getattr(usage, "cached_input_tokens", 0),
            price=PRICE[model],
        )
        records.append({
            "sample_id": s.id, "tier": s.tier, "effort": effort,
            "score": s.judge(resp.text), "latency_s": round(latency, 3),
            "cost_usd": cost["cost"], "output_tokens": cost["billed_output_tokens"],
        })
    return Report.aggregate(records)            # per-tier aggregation + confidence intervals

A companion recommendation: run regression against the same golden set in a staging environment, and review every model or prompt change on a two-dimensional cost–quality plot rather than on accuracy alone.

4. Four categories of cost-optimization engineering

1) Cascading and routing

Process first with a lower tier or smaller model, and escalate only when confidence is insufficient or validation fails. This is the highest-leverage move, because it pushes the expensive-tier call ratio down to the long-tail samples.

TIERS = ("low", "medium", "high")
def run_cascade(task, validator, budget_usd: float):
    spent = 0.0
    for effort in TIERS:
        resp = call_model(task, effort=effort)
        spent += resp.cost_usd
        if validator.accept(resp) or spent >= budget_usd:
            return resp, {"tier": effort, "spent": spent}
    return resp, {"tier": "high", "spent": spent, "degraded": True}

A required component: a programmable validator (JSON Schema validation, unit tests, rule assertions) that decides whether escalation happens; a cascade without a validator is just random escalation.

2) Context engineering: cut input tokens

After reranking retrieval results, keep only the top-K and the key passages; summarize and compress conversation history; trim fields from tool return values; and place stable system prompts and tool definitions at the prefix position to maximize cache hits. Input tokens usually account for the bulk of the total, so trimming has a higher leverage ratio than adjusting the effort tier.

3) Output constraints

Force structured output (JSON Schema / tool-call arguments), explicitly cap fields and list lengths, and for “analysis-type” tasks require the conclusion first and the supporting evidence after. Constraining output reduces both token usage and downstream parsing cost at the same time.

4) Budget guardrails and circuit breaking

Apply unified controls at the gateway layer to prevent any single point from running out of control:

@dataclass
class Guardrail:
    max_tokens_per_request: int = 32_000
    max_thinking_tokens: int = 8_000
    max_cost_per_task_usd: float = 1.0
    timeout_s: float = 120.0
def guarded_call(task, g: Guardrail):
    if estimate_cost(0, g.max_tokens_per_request, g.max_thinking_tokens,
                     price=PRICE[task.model])["cost"] > g.max_cost_per_task_usd:
        return fallback(task)          # fall back to a smaller model / queue / human handoff
    return call_model(task, effort=task.effort, timeout=g.timeout_s,
                      max_tokens=g.max_tokens_per_request)

5. Observability and unit-economics accounting

Any cost governance without usage data is just an estimate. It’s recommended to collect the following fields uniformly (one record per line, written to a logging service and shipped to a data warehouse):

• Identity fields: request_id, task_id, tenant_id, business-scenario tag, model, effort, whether a cache was hit;
• Usage fields: input_tokens, output_tokens, thinking_tokens, cached_input_tokens, retry count, cascade hop count;
• Performance fields: queue time, time-to-first-token, end-to-end P50/P95;
• Cost fields: cost_usd converted from unit prices, plus cost per document / monthly cost per user expressed in business terms.

On top of this, configure three types of alerts: cost spikes (month-over-month and year-over-year thresholds), effort drift (an abnormal rise in the high-tier share), and cache-hit-rate drops (often caused by a prompt-template change that invalidates the prefix). When implementing this system on Baidu AI Cloud, you can directly reuse standard capabilities such as the general logging service, monitoring and alerting, object storage (for archiving golden sets and evaluation reports), and a containerized inference gateway, without building the full infrastructure stack yourself; on the model side, use a unified inference access layer for multi-model abstraction to avoid binding business code to a single vendor.

6. Selection decision checklist

1. Horizontal comparison must use per-task cost under the same task set, the same tier, and the same prompt template;
2. Measure the inflection point first, then set the default tier; the cost share of the default tier usually determines the total bill;
3. Drive cascades with validators, and confine high-cost calls to genuinely difficult samples;
4. Build token budgets, timeouts, and cost circuit breakers into the gateway, rather than leaving them to voluntary compliance in the application layer;
5. Review latency by P95, and don’t let averages hide the long tail;
6. Keep a multi-model abstraction layer and a swappable routing strategy, leaving room for the next round of price and tier changes.

In one sentence: the progress of this generation of reasoning models is essentially “trading a controllable compute budget for a higher task success rate.” Whoever spends that budget on the right samples and the right tiers gets both quality and cost; conversely, teams that blindly max out the effort will find they are paying for an average while only a long tail actually benefits.