# LLM Inference Cost vs. Performance: How to Balance the Trade-off

1. A Lower Unit Price Doesn’t Mean a Lower Bill

LLM API pricing is typically split into input tokens and output tokens, with output usually priced higher. The cost of a single call can be estimated as:

Cost per call = Input tokens × Input price + Output tokens × Output price

The most commonly overlooked variable in this formula is the number of output tokens. Before producing a final answer, reasoning models generate large amounts of intermediate reasoning content—and that content is billed as output. So even if the listed price per million tokens drops, your total bill can still rise if the model generates significantly more tokens for the same task.

Here’s an example: suppose the output price falls by 20%, but average output per task grows from 100K tokens to 270K. The output cost per task changes by a factor of 0.8 × 2.7 = 2.16—an increase of roughly 116%. The savings from the price cut are more than wiped out by the growth in output volume.

The takeaway: judge costs by actual spend per task, not by the price list.

2. Inference Depth Is a Tunable Cost Parameter

Many reasoning models let you adjust the reasoning budget, commonly through low, medium, and high levels. Higher levels invest more computation in intermediate steps and usually perform better on hard tasks such as mathematical derivations, complex code changes, and multi-step planning. But quality gains don’t scale linearly with cost. When a task is simple, bumping up the level yields limited improvement while costs rise proportionally.

Treat inference depth as an engineering parameter rather than leaving it maxed out by default:

• For straightforward tasks like classification, summarization, and format conversion, use the low level or disable extra reasoning;
• For tasks requiring multi-step reasoning, such as code generation and troubleshooting, use medium or high levels;
• Reserve the highest level for a small number of critical, high-value tasks, paired with human review.

3. Measure Models by Cost per Success

Comparing per-call prices or average costs alone leads to misleading conclusions. A better metric is cost per success—the average spend to produce one valid result:

Cost per success = Average cost per task ÷ Task success rate

Example (illustrative data only): Configuration A costs $9 per task with a 60% success rate, giving a cost per success of $15. Configuration B costs $10 per task with a 70% success rate, giving roughly $14.3. B’s per-call cost is higher, yet it spends less per valid result.

When benchmarking, track four dimensions together:

• Quality: task pass rate, human ratings, or automated test results;
• Cost: itemized counts of input, output, and reasoning tokens;
• Latency: time to first token and total response time, both of which affect user experience;
• Stability: variance in results across repeated runs of the same task.

4. Cost Estimation Example Code

The script below estimates budgets under different configurations before deployment. Prices are placeholders—replace them with your actual rates and usage data.

from dataclasses import dataclass
@dataclass
class PricingConfig:
    input_price_per_m: float   # Price per million input tokens
    output_price_per_m: float  # Price per million output tokens
def estimate_cost(input_tokens: int, output_tokens: int, cfg: PricingConfig) -> float:
    """Estimate the cost of a single call."""
    return (
        input_tokens * cfg.input_price_per_m
        + output_tokens * cfg.output_price_per_m
    ) / 1_000_000
def cost_per_success(avg_cost: float, success_rate: float) -> float:
    """Compute cost per success; success_rate must be in the (0, 1] range."""
    if not 0 < success_rate <= 1:
        raise ValueError("success_rate must be in the (0, 1] range")
    return avg_cost / success_rate
if __name__ == "__main__":
    # Placeholder prices, for demonstration only
    cfg = PricingConfig(input_price_per_m=2.0, output_price_per_m=10.0)
    cost = estimate_cost(input_tokens=20_000, output_tokens=150_000, cfg=cfg)
    print(f"Cost per call: {cost:.2f}")
    print(f"Cost per success (70% success rate): {cost_per_success(cost, 0.7):.2f}")

5. Six Practices for Production

1. Tiered task routing: Choose models and reasoning levels by task complexity; route simple tasks through the low-cost path.
2. Set budget caps: Define maximum output tokens and timeouts per call to prevent unusually long reasoning runs from inflating your bill.
3. Trim prompts: Remove repeated context and redundant examples to reduce input tokens and help the model get to the task faster.
4. Cache results: Reuse previous results for identical inputs, especially for Q&A and template generation workloads.
5. Itemized monitoring: Log input, output, and reasoning tokens for every call, and aggregate by business line and feature module to spot cost anomalies.
6. Canary comparisons: A/B test new models or reasoning levels on a small traffic slice first, and use cost per success as the criterion for full rollout.

6. Selection Advice and Evaluation Caveats

Individual developers usually work with tighter budgets and need finer-grained control. Reserve high reasoning levels for hard problems, and hand routine work—autocompletion, renaming, sample generation—to low-cost configurations. For teams, bring cost into requirements reviews, with product and engineering jointly setting an acceptable cost-per-task ceiling for each feature.

Technical upgrades don’t automatically reduce costs. The real question is whether an upgrade lowers cost per success, or solves problems that were previously unsolvable within an acceptable budget.

Public benchmark results are affected by test sets, prompt templates, sampling parameters, and inference settings. Before applying them to your own business, re-evaluate with real production data and confirm the conclusions hold in your scenario.