Inference Budget and Cost: The Real Price of LLM Performance Gains

1. Why Calculate Per-Task Cost

LLM APIs are usually billed per million tokens, with separate prices for input and output. Output is generally priced higher. Many teams compare price tables during model selection but overlook a key variable: how many tokens are actually consumed to complete the same task.

Reasoning-enhanced models generate a large amount of intermediate reasoning content before producing the final answer. That content is also billed as output tokens, so deeper model reasoning leads to higher cost. A lower unit price only means each token is cheaper; it does not mean the total cost of completing a task is lower.

2. How Unit Price, Token Count, and Total Cost Relate

Per-task cost can be broken down into the following formula:

Per-task cost = input tokens × input price + output tokens (including reasoning tokens) × output price

The function below calculates cost using this formula, with prices expressed per million tokens:

def task_cost(input_tokens, output_tokens, price_in_per_m, price_out_per_m):
    """Calculate the cost of a single task. Prices are per million tokens."""
    return (input_tokens * price_in_per_m + output_tokens * price_out_per_m) / 1_000_000

The numbers below are only for demonstrating the calculation method and do not represent actual data for any specific model. Assume an input price of $3 per million tokens and an output price of $15 per million tokens. For a task with 50,000 input tokens and 100,000 output tokens, the cost is 0.15 + 1.50 = $1.65. If reasoning intensity is reduced and output drops to 30,000 tokens, the cost falls to 0.15 + 0.45 = $0.60.

Cost change is only half of the picture. The other half is task success rate. Suppose the low tier costs $0.60 per task with a 60% pass rate, while the high tier costs $1.65 with a 70% pass rate. Cost per successful task is about $1.00 for the low tier and about $2.36 for the high tier. Therefore, looking only at per-task cost, the low tier is cheaper; when the cost of failed retries is considered, the gap between the two can widen or narrow, and the decision should be based on the actual pass rate.

3. How Reasoning Intensity Affects Results

Reasoning intensity usually corresponds to an available reasoning token budget. Within that budget, the model can perform multi-step derivation, self-checking, and plan backtracking.

The benefit tends to diminish as the budget increases:

• Simple tasks (format conversion, short-text classification, routine summarization) are almost unaffected by the reasoning budget, and most additional tokens are redundant.
• Medium tasks (single-function implementation, routine data processing) usually gain stable improvements from moderate reasoning.
• Complex tasks (cross-module refactoring, multi-step logical reasoning, long-chain troubleshooting) are more likely to fail when reasoning is insufficient, and the benefit becomes clear when the budget is increased.

An excessively high budget also causes problems. The model may produce a large amount of unnecessary reasoning, increasing response latency and making the cost curve rise steeply. Therefore, reasoning intensity should be selected by task type rather than always set to the maximum.

4. Define Metrics for Cost-Effectiveness

When evaluating a model or configuration, it is recommended to observe the following metrics together:

1. Pass rate or quality score: calculated according to task acceptance criteria, such as unit test pass rate, structured output validation pass rate, or mean human rating.
2. Per-task cost: calculated from input and output token usage and unit prices.
3. Cost per successful task: per-task cost divided by pass rate. It directly reflects how much it costs to produce one usable result and is the most suitable metric for decision-making.
4. Latency percentiles: focus on P50 and P95, especially the latter, which determines user wait time in the slowest scenarios.
5. Output token distribution: in addition to the mean, also examine long-tail values such as P90. Averages can easily hide cost spikes caused by individual extremely long outputs.

5. Select Reasoning Tiers by Task

Using the highest reasoning intensity uniformly wastes budget and slows responses. Tasks can be tiered according to complexity and the cost of errors:

Task type	Main characteristics	Recommended reasoning tier
Format conversion, summarization, simple Q&A	Few steps, relatively high error tolerance	Low
Routine code completion, single-function implementation	Medium complexity, automatically verifiable	Medium
Cross-module refactoring, difficult defect localization	Multi-step reasoning, requires repeated verification	High
Security audit, critical algorithm design	High cost of errors	Highest, with human review

In engineering practice, you can first select an initial tier by task type, then decide whether to upgrade based on validation results. The example below illustrates this approach: assign a tier by type first, then escalate level by level after validation failures.

DEFAULT_TIER = {"summarize": "low", "codegen": "medium", "refactor": "high"}
UPGRADE = {"low": "medium", "medium": "high", "high": "high"}
def pick_tier(task_type, retry_count=0):
    tier = DEFAULT_TIER.get(task_type, "medium")
    for _ in range(retry_count):
        tier = UPGRADE[tier]
    return tier

The upgrade policy needs a cap. If the highest tier still fails, the task should be routed to human handling rather than retried indefinitely.

6. Build an Internal Evaluation Benchmark

Public leaderboards reflect general capabilities and may not align with a team’s own business. An internal benchmark makes model-selection conclusions reproducible. The main steps are:

1. Collect real task samples and stratify them by type. Samples should cover both common and difficult tasks, with proportions as close as possible to the online distribution.
2. Define acceptance criteria that can be automatically judged, such as unit tests, JSON Schema validation, or key-field matching. For tasks that cannot be automatically judged, use a consistent human scoring rubric.
3. Fix evaluation conditions, including model version, system prompt, sampling parameters, and evaluation date. If conditions are not fixed, results cannot be compared across runs.
4. Record input and output tokens, latency, and pass/fail status for each call.
5. Run the same task multiple times and take the average, because model outputs are stochastic and a single result is not sufficient to draw conclusions.

The example below summarizes the performance of different configurations on the same set of tasks:

import statistics
def evaluate(config_name, tasks, run_fn, cost_fn):
    """Evaluate a configuration on a set of tasks and return pass rate, cost, and long-tail output metrics."""
    records = []
    for task in tasks:
        output, in_tok, out_tok, latency = run_fn(task)
        records.append({
            "passed": bool(task["check"](output)),
            "cost": cost_fn(in_tok, out_tok),
            "out_tok": out_tok,
            "latency": latency,
        })
    pass_rate = sum(r["passed"] for r in records) / len(records)
    avg_cost = statistics.mean(r["cost"] for r in records)
    out_sorted = sorted(r["out_tok"] for r in records)
    p90_out = out_sorted[min(len(out_sorted) - 1, int(len(out_sorted) * 0.9))]
    return {
        "config": config_name,
        "pass_rate": round(pass_rate, 3),
        "avg_cost": round(avg_cost, 4),
        "cost_per_success": round(avg_cost / pass_rate, 4) if pass_rate else float("inf"),
        "p90_out_tokens": p90_out,
    }

Evaluation results should be used as the basis for model selection and rerun regularly. After model versions or billing rules change, the evaluation needs to be repeated.

7. Engineering Implementation Points

• Set a maximum output token limit to prevent runaway single calls.
• Define clear policies for timeouts and retries. Retries incur additional cost and must be included in cost statistics.
• Cache results for identical inputs to avoid duplicate billing.
• Feed token usage, per-task cost, and pass rate into logging services and monitoring alerts, and set daily budget thresholds that notify when exceeded.
• Do not log secrets or user privacy data; sensitive fields need to be redacted.

8. Conclusion

Inference cost is a core variable in LLM applications. A lower unit price does not mean lower total cost, and higher reasoning intensity does not necessarily produce positive returns. In practice, teams should do three things: use cost per successful task as the decision metric; select reasoning tiers by task complexity and error cost; and continuously validate with an internal benchmark that reflects the business. Only by comparing quality, cost, and latency in the same table can a team find a sustainable balance between performance and expense.