How to Calculate Real LLM Costs: Reasoning Effort Levels and Cost-Effectiveness

1. A typical misjudgment: unit price drops, but the bill doubles

When a new flagship model is released, “20% lower API pricing” is often the most visible headline. Directly converting that into “my cost will also drop by 20%” is almost certainly wrong.

A common pattern in the industry is: the unit price gets a 20% discount, the same evaluation set is run, but the average cost per question actually doubles. The reason is not mysterious: the extra money comes from reasoning tokens (reasoning / thinking tokens). New-generation models tend to think before answering by default. For the same question, output can grow from around one hundred thousand tokens to nearly three hundred thousand tokens, with most of the increase being the thinking process. Meanwhile, mainstream platforms generally bill tokens generated during reasoning as output tokens. As a result:

unit price × 0.8, usage × 2.3, total cost ≈ 1.8×

This kind of accounting can lead to completely different conclusions when each tier is examined separately. Using the public measurement approach of a third-party evaluation organization as a reference, here is the same coding benchmark across different reasoning effort levels. The table below uses illustrative data to explain the calculation method; it does not represent official numbers for any specific product.

Effort level	Benchmark score	Average output tokens per question	Cost per question (illustrative)	Cost per point (illustrative)
Low	54	~60K	3.1	0.057
Medium	56	~90K	3.8	0.068
High	62	~150K	9.1	0.147
Extra-high	62	~220K	14.8	0.239
Maximum	65	~380K	26.5	0.408
Previous generation · Maximum (reference)	59	~120K	~10	0.169

Three conclusions can be used directly for decision-making:

1. Only the top one or two tiers are expensive. The high and extra-high tiers achieve exactly the same score, but extra-high costs about 63% more. That tier has little engineering value.
2. The medium tier is often where cross-generational gains are largest. In the previous generation, the same tier was about 43 points at 5.6 cost per point; in the new generation, it is about 56 points at 3.8 cost per point, achieving both “stronger” and “cheaper.” This is also why many intelligent coding assistants and conversational applications set medium as the default tier.
3. The high tier is a cost-performance inflection point. Compared with the previous-generation maximum-tier reference, the score is slightly higher and the cost is comparable or even lower. Raising the tier further by roughly three times only buys about 3 more points, which is a classic region of collapsing marginal returns.

2. The right unit of measurement: cost per task, not price per million tokens

Unit price is for procurement; cost is for architecture. At minimum, engineering teams should establish three layers of measurement:

from dataclasses import dataclass
# Illustrative pricing: per million tokens
PRICE = {"input": 8.0, "output": 40.0}
@dataclass
class Usage:
    prompt_tokens: int
    completion_tokens: int
    reasoning_tokens: int = 0      # Most platforms bill this at the output price
    cached_prompt_tokens: int = 0  # Prompt-cache hits are usually discounted
def task_cost(u: Usage, price=PRICE) -> float:
    billed_out = u.completion_tokens + u.reasoning_tokens
    fresh_in = u.prompt_tokens - u.cached_prompt_tokens
    return (
        fresh_in * price["input"]
        + u.cached_prompt_tokens * price["input"] * 0.1   # Cache-hit discount; adjust to actual platform rules
        + billed_out * price["output"]
    ) / 1_000_000

Based on this, three derived metrics are truly useful for decision-making:

• Cost per task: the average cost to complete one end-to-end business unit, including retries and failed tasks.
• Cost per point: cost per task divided by benchmark score. It is much more intuitive than absolute score for comparing the cost-effectiveness of different tiers.
• Cost per accepted output: divide by accepted output volume rather than number of attempts. If a higher tier raises the first-pass rate from 60% to 75%, the hidden savings from reduced rework may offset part of the token premium. This point is invisible in a token-only accounting.

3. Reasoning effort is not a linear knob

Reasoning effort levels (low / medium / high / xhigh / max, or an equivalent thinking-budget cap) control how many reasoning tokens the model is allowed to consume before giving an answer. Their practical behavior has three notable characteristics:

• Diminishing returns and a plateau. Moving from low to medium is often a jump from “cannot do it” to “can do it”; moving from medium to high is a stable but limited improvement; the maximum tier often solves only long-tail hard problems at the cost of the entire bill. The fact that “high” and “extra-high” score the same in the table is a direct manifestation of this plateau.
• Cost grows faster than score. Reasoning-token consumption is superlinear: the harder the problem, the longer the model thinks, and long context can raise the input cost of each reasoning step at the same time.
• Effort and output caps must be constrained together. Raising only the effort level without setting a maximum output-token limit can easily lead to “runaway thinking,” producing long-tail outliers in per-request cost. In practice, configure both maximum output length and per-request budget.

Therefore, the correct way to select a tier is not uniform across all traffic, but to tier requests by task difficulty: bucket requests by predictable difficulty, send simple tasks to low, medium tasks to medium, and allow only requests judged to be difficult to rise to high.

4. What to look for in benchmark tables: four comparability traps

Vendor comparison tables at launch usually need to be read selectively. Not because the data are fake, but because the measurement conditions are often not comparable:

1. Effort levels are not aligned. If one vendor labels its model as “extra-high reasoning” while comparing against a competitor’s “medium effort,” it is effectively comparing against the competitor’s cost-saving mode. The first thing to check in such a table is the effort-level annotation.
2. Self-reported and third-party results are mixed. Some metrics have footnotes saying “internal testing” and lack independent reproduction conditions, such as prompts, sample size, scoring scripts, and temperature parameters. Such numbers can only be treated as directional references.
3. Benchmark saturation and contamination. Once public benchmarks are widely used, training-data contamination and question saturation can inflate scores. In cross-generation comparisons, “the score went up” may simply mean the benchmark is no longer valid.
4. Cost is not normalized. A table that reports only pass rate but not cost per task cannot support selection. At minimum, it should be possible to derive “the cost required to reach the same score.”

A reasonable attitude toward such tables is: the direction is credible, but absolute values are not. If a vendor’s self-calculated accounting is directionally consistent with third-party estimates, it can serve as a preliminary basis. The final conclusion must return to your own business evaluation set.

5. Bring evaluation into your own business: a three-layer evaluation system

External leaderboards can only filter the candidate set; internal evaluation is what truly determines the tier. A three-layer structure is recommended:

• Layer 1: deterministic cases (automatically scorable). Tasks with clear correct answers, such as function-level code generation, SQL rewriting, configuration conversion, and log-field extraction. Use unit tests, compilation pass rates, or execution-result comparison for scoring. This can be fully automated and is suitable as a regression gate.
• Layer 2: judge-model scoring (LLM-as-judge). For tasks without a unique answer, such as refactoring plans, document summaries, and root-cause analysis. Fix a scoring rubric, and manually calibrate the judge itself to avoid position bias and length preference.
• Layer 3: human spot checks. Sample real business requests at a certain ratio, and record first-pass rate, number of rework rounds, and average duration. This layer is the most expensive, but it is the only one that can quantify the “cost of human acceptance.”

The core of an evaluation framework is not the score, but producing quality metrics and cost metrics in the same run:

def evaluate(cases, effort):
    passed, cost, latencies = 0, 0.0, []
    for c in cases:
        t0 = time.time()
        resp = llm.complete(prompt=c.prompt, effort=effort,
                            max_output_tokens=cap_for(effort))
        latencies.append(time.time() - t0)
        cost += task_cost(resp.usage)
        passed += int(c.checker(resp.text))
    n = len(cases)
    return {
        "effort": effort,
        "pass_rate": passed / n,
        "cost_per_task": cost / n,
        "cost_per_point": (cost / n) / max(passed / n, 1e-6),
        "p95_latency": sorted(latencies)[int(n * 0.95) - 1],
    }

For evaluation-set size, 200–500 tasks that cover the real distribution are usually enough to distinguish tier differences. The key is to version everything: prompts, model versions, effort levels, and evaluation-set hashes should be recorded together. Otherwise, week-over-week comparisons are meaningless.

6. Cost governance: routing, caching, and guardrails

Choosing the right tier is only the first step. Once usage scales, engineering governance is required.

1) Cascading and routing. First try a low-cost tier, validate with an automatically scorable checker, and escalate only when validation fails. This is the most effective way to push down average cost:

LADDER = ["low", "medium", "high"]
def solve_with_escalation(prompt, checker, budget):
    spent = 0.0
    for effort in LADDER:
        resp = llm.complete(prompt=prompt, effort=effort)
        spent += task_cost(resp.usage)
        if checker(resp.text):
            return resp.text, effort, spent
        if spent > budget:
            break
    return None, None, spent   # Escalate to manual handling or a fallback process

Routing signals can be earlier than checker failure: task-type classification, input length, whether the task involves cross-file changes, and the historical first-pass rate of similar tasks can all be used to predict whether a higher tier is needed.

2) Caching and context governance. Stable prefixes such as system prompts, repository summaries, and tool definitions should use prompt caching. The cached portion is usually billed at a significant discount. At the same time, use retrieval to inject context on demand, and avoid stuffing the entire repository into the prompt. In long-context scenarios, input tokens often exceed output tokens, making them a cost item that cannot be ignored.

3) Batch and asynchronous processing. Offline batch tasks, such as bulk code review or log classification, should use asynchronous batch interfaces. These usually have clear price discounts and can also shift compute usage to off-peak periods.

4) Budget guardrails and observability. Tag costs by team, project, or user. Record prompt, completion, and reasoning token usage as well as failure rates. Connect to logging services and monitoring alerts, and set daily budgets, per-request budgets, and rate-limit thresholds. On platforms such as Baidu AI Cloud, this is often implemented through a combination of general capabilities: object storage for archived call logs, a logging service for structured search, and monitoring/alerting for cost and latency dashboards. There is no need to build the entire infrastructure stack from scratch. Without this layer of observability, cost overruns are often discovered only when the monthly bill arrives.

7. Two real-world constraints often hidden by benchmark tables

• Long-horizon tasks have a completely different cost magnitude. Demonstrations such as “a multi-agent system working in parallel for days to complete an operating-system-level project” or “a long-form formal proof of a major theorem” involve hundreds or thousands of agents, wall-clock time measured in days, and a budget of a different order of magnitude. They demonstrate the capability ceiling and are not the same problem as “opening a session in a coding assistant to fix one bug.” When evaluating everyday productivity, such cases should not be used as a cost reference.
• Safety-sensitive requests may be blocked by policy. Requests involving penetration testing, exploit-chain construction, or binary vulnerability analysis may be intercepted by a platform-side safety classifier or require additional authorization. For teams with compliance requirements, this directly affects workflow design: such capabilities need to be supplemented in a local or private environment, and formal authorization and audit processes should be used, rather than hoping to bypass policy by adjusting prompts.

8. Selection checklist

1. Compare cost per task rather than unit price, and count reasoning tokens on the output side.
2. Run internal evaluations across at least low / medium / high tiers, and choose the tier with the lowest cost per point as the default.
3. Set the default tier to low-to-medium; open the high tier only on demand for difficult tasks, and configure an output cap.
4. When reading external leaderboards, first align effort levels, distinguish self-reported from third-party results, and confirm whether cost normalization is included.
5. Build a versioned internal evaluation set with at least 200 tasks, and include cost metrics alongside quality metrics in regression testing.
6. Deploy three cost-reduction mechanisms: cascading routing, prompt caching, and asynchronous batch processing.
7. Add token-level observability and budget guardrails, with attribution by business dimension.
8. Include the cost of human acceptance in the total accounting, and use it to judge whether a more expensive tier is really cheaper.

In one sentence: The answer to model selection is not in the benchmark tables shown at a launch event, but at the inflection point of the “score–cost” curve on your own evaluation set. Find that inflection point, set it as the default tier, and leave the rest to routing and guardrails.