LLM Model Selection Guide: Inference Effort Levels and Real-World Costs

1. First, Correct a Common Misreading: Lower Token Prices Do Not Guarantee Lower Bills

When vendors announce new model prices, the headline token price is often the most visible metric—and the most easily misinterpreted. Real spend is not determined by price per token alone. It is determined by the following formula:

Cost per task = (input tokens × input price + output tokens × output price) / 1,000,000
             + retry cost + tool-call cost + storage and logging cost

Two easily overlooked points matter here.

First, the “thinking process” of reasoning models is billed as output tokens. Intermediate reasoning, self-checks, and comparison of alternatives generated before the final answer are billed on the output side, and output prices are usually much higher than input prices. If a model becomes “smarter” mainly by thinking for more rounds, the bill can rise much faster than token-price reductions suggest.

Second, multi-turn conversations and tool calls repeatedly bill input tokens. Conversation history, code context, and tool-returned logs and file contents accumulate across turns. If prefix caching is not hit, the same context can be re-billed on every request.

For example, in a commonly cited public-benchmark dataset: when a recent model generation runs at maximum reasoning effort, average output tokens per task increase from about 118K to about 274K, an increase of about 132%. At the same time, API prices are reduced by about 20% (input around $8 per million tokens, output around $40 per million tokens). Yet the average cost per task rises from about $12.99 to about $26.50, nearly doubling. The breakdown is clear: 274K output tokens at $40 per million tokens costs about $10.96, only around 40% of the $26.50 total. The rest comes from repeated multi-turn context billing, tool-returned content, and failed retries. Budgeting based only on token price will almost certainly underestimate real cost.

2. What Inference Effort Levels Actually Adjust

Mainstream reasoning models usually expose several inference effort levels from low to high (common names include minimal / low / medium / high / xhigh / max), with the default often set to a medium-high tier. Changing the effort level is not the same as switching to a different model. It adjusts limits on several internal resources:

• Thinking-token budget: how much intermediate reasoning the model may generate before answering;
• Search and sampling width: the number of internal candidate approaches and the strictness of filtering;
• Verification and revision rounds: the maximum number of self-checks, tests, rollbacks, or retries.

This leads to an important engineering conclusion: inference effort is a performance-cost curve, not an on/off switch. Using the benchmark data above: at maximum effort, the model scores about 65 on a composite intelligence index. The default higher tier scores 62—only 3 points lower—but the average cost per task drops from about $26.50 to about $9.08, a reduction of more than 60%. In other words, a lower-effort setting on a newer generation can still beat the maximum-effort setting of a previous generation (the previous generation scores about 49 at maximum effort and costs about $12.99 per task; a comparable competitor scores about 59 at maximum effort and costs about $10.04). The cost-performance inflection point on this curve is where model selection should focus.

Diminishing returns are especially obvious with reasoning models: moving from low to medium usually captures most of the benefit; moving from medium to maximum often produces meaningful gains only on a small set of difficult tasks, while cost can multiply.

3. Turn “Cost-Effectiveness” into a Calculable Metric

Token-level unit prices do not reflect business value. A better metric is Cost Per Successful Task (CPST):

CPST = total spend (including retries and tool calls) / number of successfully completed tasks
     = average cost per task / task success rate

This metric exposes traps such as “a cheap model succeeds only twice out of ten runs.” Three observation dimensions should be tracked alongside it:

1. Capability: task success rate, first-pass success rate, average number of correction rounds;
2. Cost: average input/output tokens, cache hit rate, average cost per task;
3. Stability: score variance across repeated samples, interruption rate for long-running tasks.

For evaluation, a stratified shadow evaluation is recommended: sample real business tasks to build an internal benchmark of at least 200 tasks, stratified by difficulty (easy / medium / hard), sample each effort level at least three times to estimate variance, and keep all variables fixed except temperature and random seed. Then plot a three-point evaluation chart of effort level, success rate, and CPST, and use the inflection-point effort level as the production default.

A directly usable cost estimation tool:

def estimate_cost(input_tokens, output_tokens, retries=1,
                  price_in=8.0, price_out=40.0, cache_hit_rate=0.0):
    """Estimate the cost per task using per-million-token pricing.
    Cached portions are billed at 10% of the input price (simplified)."""
    effective_in = input_tokens * (1 - cache_hit_rate * 0.9)
    base = (effective_in * price_in + output_tokens * price_out) / 1e6
    return base * retries

4. Four Engineering Levers for Reducing Cost

Tiered routing. Do not use the same effort level for every request. Use rules, a lightweight classifier, or model-reported confidence to make an initial judgment, and escalate only difficult samples:

def route(task):
    if task.est_difficulty == "low":
        return dict(effort="low",    max_tokens=4096)
    if task.est_difficulty == "mid":
        return dict(effort="high",   max_tokens=16384)
    if task.budget_left <= 0:
        return dict(effort="high")          # Fall back when budget is exhausted
    return dict(effort="max", max_tokens=65536)

In production, medium- and low-difficulty requests usually make up the majority. Routing can produce a larger overall cost reduction than switching models alone.

Context governance. Enable prefix caching and keep prompts in a stable order (system prompt → long-term context → current-turn input) to maximize cache hit rate. Use retrieval instead of injecting all code or documentation. Truncate and summarize tool-returned results. Compress long conversations with rolling summaries.

Asynchronous processing and batching. Route offline tasks through batch APIs (most platforms offer price discounts for batching). Keep the online path limited to necessary requests. Use message queues to smooth peaks and avoid retry amplification during traffic spikes.

Result reuse. Use semantic caching to serve similar requests, artifact caching to avoid duplicate builds, and incremental computation to process only changed portions.

5. How to Read Leaderboard Scores

Public leaderboards are a starting point for filtering candidates, not the basis for final decisions. There are three reasons:

• A single score does not include cost. Two models scoring 60 may cost $3 and $30 per task respectively; their business meaning is completely different. Rankings that omit token consumption and inference effort have limited value.
• Distribution mismatch. The task distribution in public benchmarks often differs substantially from real business distribution, and there is a risk of training-data contamination.
• Variance is hidden. Reasoning models can show sampling variance of several percentage points on difficult tasks. A single benchmark run is not enough to support a conclusion.

The right approach is: use leaderboards as a candidate pool, make the final decision with your own evaluation set, and require any externally claimed “SOTA” result to also disclose the inference effort level, average token consumption, and average cost per task.

6. Evaluation Focus for Coding and Agentic Tasks

Evaluation of coding tasks is shifting from “single-function completion accuracy” to “end-to-end task completion rate.” The benchmark data discussed above also supports this point: on a coding capability benchmark, the model achieves its best score of about 49.1% at maximum reasoning effort, while the higher tier scores about 45.3%, roughly equal to a comparable model’s maximum-effort score of about 45.5%, but at about half the cost. In another agentic coding benchmark, maximum effort also sets a new record of about 60%, but at a significantly higher cost.

The conclusion is clear: the value of maximum effort is concentrated in a small number of difficult tasks. Routing those tasks separately is more economical than raising the effort level for all traffic.

Evaluation metrics should include at least: compile/build pass rate, unit-test and integration-test pass rate, regression introduction rate, number of human interventions, average number of patch rounds, and cost per task.

7. Long-Horizon Autonomous Engineering: The Real Barrier Is Orchestration Architecture

One vendor demonstrated the following validation: starting from an empty directory, it coordinated more than one thousand sub-agents working continuously for about eight days, autonomously completing coding, testing, and code review, and ultimately produced a runnable operating system containing a kernel, drivers, file system, graphical interface, login, and installer.

The engineering value of such a demonstration is not whether the output can be directly put into production. It is that it validates the feasibility of long-horizon, large-scale agent orchestration. To bring similar capability into a production environment, single-model capability is only half the picture; the other half is architecture:

• Task graph decomposition: break goals into a DAG of tasks with dependencies, and advance them in topological order through a scheduler rather than forcing a single agent to handle everything linearly;
• Externalized state: write plans, progress, and decision records into external storage and version them, supporting checkpoints and rollback, so the context window is not the only memory;
• Isolated sandboxes: run each sub-agent in an isolated container or lightweight virtual machine, with least privilege, no network access or whitelisted network access, to prevent mistakes from spreading;
• Event-driven scheduling: use message queues to distribute tasks and collect results, and use a workflow engine to handle retries and timeouts;
• Quality gates: enforce staged checkpoints such as compile → unit tests → integration tests → performance → security scans; fail the pipeline and do not allow progress while any gate remains red;
• Observability: use centralized logging to collect execution traces, monitor alerts to track failure rates and queue backlog, and attribute cost to tasks, agents, and stages;
• Budget guardrails: set hard limits and circuit breakers for tokens, duration, and spend; automatically degrade or pause near thresholds and request human confirmation.

These capabilities belong to general-purpose cloud capability categories: container platforms provide isolated runtimes, object storage holds code, artifacts, and trace logs, message queues handle task distribution, and logging and monitoring services support observability and cost control. Baidu AI Cloud can also provide support through a combination of similar general-purpose capabilities. Refer to the official documentation for exact specifications and billing.

8. Deployment Checklist

1. Use CPST instead of token unit price as the primary model-selection metric.
2. Plot an “effort level — success rate — cost” curve for each business category and identify the inflection-point effort level.
3. Use a medium-high effort level by default, and route only difficult tasks to maximum effort.
4. Enable prefix caching, keep prompt prefixes stable, and monitor cache hit rate.
5. Force truncation and summarization of tool-returned results to control context growth.
6. Send offline tasks through batch processing, and apply both concurrency and budget rate limiting to online traffic.
7. Build an internal evaluation set of at least 200 tasks, stratified by difficulty, with multiple samples per tier.
8. Attribute cost to business lines and task types, and review effort-level configuration monthly.

9. Conclusion

Three sentences summarize this methodology: evaluate performance together with the effort level, evaluate cost per task rather than per token, and select models by cost per successful task. New-generation models improve performance by increasing reasoning tokens; this path delivers real benefits, but the costs are equally real. Most production traffic, however, does not need maximum effort. Treating effort level as an adjustable dial rather than a fixed property—and using routing, caching, and budget guardrails to keep cost near the inflection point—is the way to achieve truly reasonable cost-performance without sacrificing success rates on critical tasks.