LLM Inference Tiers and True Cost: How to Calculate the Real Cost of Model Selection

1. The symptom: lower unit price, higher bill

Every time a flagship model is updated, two seemingly contradictory evaluations appear. Some people focus only on the price list and assume that lower per-token prices mean a better deal. Others publish actual bills and point out that the same tasks now cost several times more. Both observations can be correct. The disagreement comes from how cost is calculated.

Large language model service pricing is usually composed of three parts:

1. Token unit price: Input and output are priced separately, usually per million tokens. The output price is often several times higher than the input price.
2. Tokens consumed per call: This includes system prompts, historical context, retrieved content, the model’s internal reasoning process, and the final answer.
3. Total number of calls required to complete one business task: Complex tasks often require multiple tool calls and result validation.

Unit price only determines the first part. Today’s flagship models commonly trade additional reasoning computation for better quality: they are allowed to generate longer chain-of-thought or reasoning traces before producing an answer. As a result, reasoning tokens can increase sharply. Even if the output token price drops by 20%, the cost of a single task may still double.

A simple example illustrates this:

Assume Model A: input at $8 per million tokens, output at $40 per million tokens
Task: approximately 20K input tokens, 120K output tokens including reasoning and the final answer
Cost ≈ 20K × 8 / 1e6 + 120K × 40 / 1e6 ≈ 0.16 + 4.80 ≈ $4.96
If the highest reasoning setting is enabled for the same task, output expands to 275K tokens:
Cost ≈ 0.16 + 11.00 ≈ $11.16, or about 2.2x the original cost

The conclusion is straightforward: for reasoning-oriented models, the dominant cost variable is output tokens, not the per-token price. If model selection only compares unit prices, the result will almost certainly be wrong.

2. Turn cost into an auditable model

Teams should standardize a cost estimation function internally so that every solution comparison uses the same calculation method:

def estimate_cost(in_tok: int, out_tok: int, cached_tok: int = 0,
                  price_in: float = 8.0, price_out: float = 40.0,
                  cache_discount: float = 0.9) -> float:
    """
    Prices are in USD per million tokens; cache_discount is the discount applied after a cache hit.
    Returns: cost per call in USD.
    """
    fresh_in = max(in_tok - cached_tok, 0)
    cached_in = cached_tok * price_in * (1 - cache_discount)
    in_cost = (fresh_in * price_in + cached_in) / 1e6
    out_cost = out_tok * price_out / 1e6
    return round(in_cost + out_cost, 4)

A more important metric is the second layer: cost per successful task.

Cost per successful task = total spend / number of tasks that pass validation
                         = cost per call / first-pass success rate

This metric explains a counterintuitive situation: a lower reasoning tier may be cheaper per call, but if the first-pass success rate is low and the task requires an average of three retries, the real cost may be higher than a higher reasoning tier that gets the answer right the first time. Cost evaluation must be tied to quality. Comparing unit prices without considering success rate has little decision-making value.

3. Choosing reasoning tiers: most tasks do not need the maximum setting

Most reasoning-oriented models offer multiple reasoning effort levels, such as fast, standard, deep, or maximum. These levels usually differ in the upper limit of reasoning tokens and the breadth of search or exploration. In practice, three principles should be followed:

• Match the tier to task difficulty: Use lower tiers for extraction, classification, format conversion, and simple rewriting. Use medium or high tiers for multi-step code repair, long-document reasoning, mathematical proof, and complex agent orchestration.
• Set the default to a medium tier: In many industry evaluations, the overall intelligence level of a medium tier can already exceed the maximum tier of the previous flagship generation, while costing less. Treat the maximum tier as an “upgrade path” rather than the default.
• Calibrate tiers using success rate: For each tier, track first-pass success rate and cost per successful task, then select the lowest-cost tier that meets the quality bar. If the medium tier already achieves a 95% first-pass success rate, the marginal benefit of moving to a higher tier often does not justify the additional cost.

4. How to read benchmark scores correctly

Third-party composite evaluation indexes and code capability benchmarks are useful for model selection, but four cautions are important. Otherwise, the conclusions can be misleading:

1. Scores and cost must be evaluated together: A single score has little decision value by itself. Plot an “effect versus cost” scatter chart and look for the Pareto frontier. Only solutions on the frontier are reasonable choices under current conditions.
2. Pay attention to differences caused by the same metric across tiers: The score, latency, and cost of the same model can differ by several times across reasoning tiers. When citing scores, always specify the tier.
3. Beware of mismatches between benchmark data and business traffic distribution: Public benchmarks tend to favor standardized and difficult problems, while most production traffic consists of medium-difficulty requests. A lead on benchmarks does not necessarily mean a lead on business metrics.
4. Focus on latency and throughput: If two solutions have the same score but one doubles latency, the experience difference in interactive applications can be much larger than the difference shown by paper cost alone.

5. Build a private evaluation set: the only reliable basis for decisions

External leaderboards can be used for initial screening, but the final decision must rely on a private evaluation set that matches the business traffic distribution.

How to build it

• Sample from real online traffic by difficulty and domain. A size of 200 to 500 examples is usually enough to obtain stable conclusions.
• Prepare automatically verifiable validation logic for each sample, such as unit tests, field assertions, regular expression checks, or human double-blind scoring.
• Manage the set by version, record collection time and source, and periodically add newly appearing long-tail scenarios.
• Keep the evaluation set isolated. Avoid letting evaluation samples enter any form of training data or prompt optimization data; otherwise, scores may be artificially inflated.

Metrics that must be collected

Dimension	Metrics
Quality	First-pass success rate, fix rate, factual consistency, format compliance rate
Cost	Cost per call, cost per successful task, average daily cost per 1,000 requests
Performance	P50/P95 time to first token, end-to-end latency, throughput
Stability	Rate limiting rate, timeout rate, degradation trigger rate

6. Four engineering approaches to reduce cost

1. Tiered routing and cascading calls

Use smaller models for simple requests and escalate only when the response fails or confidence is insufficient:

ROUTE = [
    {"model": "small",    "max_out": 4_000,  "effort": "low"},
    {"model": "mid",      "max_out": 16_000, "effort": "medium"},
    {"model": "frontier", "max_out": 64_000, "effort": "high"},
]
def solve(task, quality_bar=0.95, budget=5.0):
    spent, ctx = 0.0, task
    for step in ROUTE:
        r = call(ctx, model=step["model"], max_tokens=step["max_out"],
                 reasoning=step["effort"])
        spent += r.cost
        if r.verified and r.confidence >= quality_bar:
            return r, spent
        if spent >= budget:              # Budget circuit breaker to avoid cost amplification from cascading.
            break
        ctx = ctx.with_feedback(r.errors)  # Feed failure information back into the context.
    return escalate_to_human(ctx), spent

The risk of cascading is “cost accumulation.” Therefore, each request must have a budget limit and a maximum escalation depth. If the budget is exceeded, the system should degrade gracefully or hand the task to a human.

2. Prefix caching and context reuse

System prompts, tool definitions, knowledge-base fragments, and long documents are stable prefixes that repeat heavily across requests. Placing stable content at the beginning of the prompt and enabling caching can significantly reduce input costs and time to first token. The engineering constraint is that dynamic content, such as timestamps, user IDs, and random IDs, must never be placed inside a cached prefix; otherwise, the cache hit rate will drop to zero.

3. Context compression and retrieval instead of full-context injection

If a retrieval fragment can solve the problem, do not insert an entire document into the context. If a summary can replace the full history, do not carry the complete history. In multi-turn conversations, rolling summaries of history and retaining only key fields from tool-call results can often compress input tokens by more than half.

4. Output-side control

Explicitly constrain output length. Use JSON Schema to limit fields for structured tasks, require “minimal patch changes” rather than full-file rewrites for code tasks, and limit extraction tasks to list items. At the same time, use batch processing, asynchronous execution, and streaming output to improve throughput and user experience.

7. Observability and cost governance

Cost reduction requires visibility. It is recommended to log the following fields from the inference call chain and build an attributable cost dashboard: model and reasoning tier, input/output/cached token counts, per-call and cumulative cost, latency distribution, validation results, retry counts, and degradation reasons. On top of this foundation, configure budget threshold alerts and tenant-level quotas, and set monitoring rules for abnormal spikes, such as the average reasoning tokens for a certain request type doubling.

If you use Baidu AI Cloud’s model inference services, compute resources, object storage, logging, and monitoring capabilities, it is also advisable to feed the same cost accounting into a unified observability system so that “every dollar can be traced to a specific model, tier, and business scenario.” Organizational adoption of cost governance can follow FinOps principles: allocate budgets by business line or scenario, review cost per successful task monthly, and include both savings and quality metrics in the evaluation.

8. Model selection checklist

Before introducing or upgrading any reasoning model, check the following questions:

1. Is the price comparison based on per-million-token unit price, or on cost per successful task?
2. Are different reasoning tiers and default values configured for requests of different difficulty?
3. Is there a private evaluation set that matches online traffic distribution, with the model tier clearly specified?
4. Has an effect-versus-cost Pareto chart been produced instead of relying only on a single score?
5. Are prefix caching, context compression, and retrieval instead of full-context injection enabled?
6. Are request-level budget limits, maximum escalation depth, and degradation paths defined?
7. Are there observability dashboards for tokens and cost, as well as quotas and alerts?

Once this calculation framework is fixed, the question “is the new model worth it?” is no longer an emotional debate. It becomes a data-driven arithmetic problem: under the same task distribution and the same quality threshold, whichever model has lower cost per successful task and acceptable latency should enter production.