LLM Reasoning Effort Levels: Capability Boundaries and Cost Accounting

Rising evaluation scores, faster output, and lower unit prices are easily read as “stronger and cheaper.” For engineering teams, the real question is different: under default usage, do end-to-end cost, failure rate, and rework time improve together for the same class of tasks? Reasoning models often trade more intermediate thinking for higher accuracy. That approach is frequently effective for complex reasoning and long-horizon coding, but the bill depends on the total input and output consumed by the whole call, not on the list price per million tokens.

Break the Four Metrics Apart

When discussing cost-effectiveness, record at least four types of data together. Do not let a single leaderboard replace production decisions.

1. Capability: Evaluate by task type, not only by a composite index. Code repair, solution design, long-document summarization, and tool calling have completely different failure modes.
2. Latency: Record both time to first token and total completion time. A speed gain that comes from shorter reasoning may be accompanied by a quality drop. If total time is still dragged down by very long output, the interaction experience will not be better.
3. Unit price: Input and output are usually priced separately, and output is often more expensive. A lower unit price only indicates a change in the rate.
4. Cost per task: Count everything for one task: input, output, retries, tool round trips, and human review. If output scale doubles, task cost can still rise even when the unit price falls by 20%. Illustrative calculation: output increases from about 100K tokens to about 200K tokens, and the output unit price drops by 20%; the output cost is still about 1.6 times the original. This only shows the order of magnitude, not a measured quote from any service.

A composite index is useful for observing the upper bound, but it is not suitable as a direct procurement basis. The default tier is often the real distribution for most requests. If a high tier leads on score while the default tier is already close to the previous generation’s high tier, most daily tasks do not need to be set to the strongest reasoning level.

Reasoning Effort Is Not “Max Out”

Reasoning intensity can be understood as the search budget the model is allowed to consume before giving a final answer. A larger budget creates more opportunity to check constraints, enumerate boundaries, and self-correct. It also makes verbose reasoning, repeated explanations, and ineffective tool calls more likely.

Run at least a three-tier comparison instead of looking only at the highest tier:

• Low tier or extra reasoning off: Suitable for format conversion, summaries, classification, and completion from an existing template.
• Default high tier: Use this as the main tier. Compare it with the previous generation’s strongest tier to see whether it is already sufficient. This is the key cost-effectiveness comparison.
• Max tier: Reserve it only for tasks with high failure cost and automatic verifiability, such as contest-style algorithms, cross-module defects, and protocol implementations that require strict proof.

Controlled experiments should fix the prompt, tool permissions, temperature or sampling strategy, timeout, and retry count. For each tier, record success rate, human pass rate, total tokens, wall-clock time, and cost. If the max tier only raises the pass rate from 45% to 49% but nearly doubles cost per task, it should not become the default unless the rework cost for that task is higher.

Coding Scenarios Need Verifiable Results

Coding capability should not be judged only by “how many lines were generated.” A more useful evaluation is: given a repository and tests, can the model make the tests green within a limited number of rounds and leave a reviewable diff?

In practice, split the evaluation into three sets:

• Single-file or single-function repair: Tests localization and minimal changes.
• Cross-module feature: Tests interface contracts, regressions, and documentation consistency.
• Continuous delivery from an empty directory or skeleton repository: Tests whether planning, decomposition, testing, and review can form a closed loop.

Long-horizon autonomous development can demonstrate the upper bound: an agent splits tasks, writes code, runs tests, performs review, and advances a runnable system over multiple days. Such demonstrations show the value of orchestration, persistence, and acceptance gates, but they cannot replace production conclusions. Empty-directory experiments lack legacy constraints, permission boundaries, data compliance, and on-call response. As the number of subagents rises, duplicate work and context inflation can push costs very high. Treat this as a stress test, not a default workflow.

During evaluation, publish both pass rate and cost per task. It is common in the industry for a high tier to refresh accuracy while the default tier matches the previous generation’s high tier at roughly half the cost. Procurement language should be written as: “The default tier covers what percentage of tickets; the high tier is escalated only after validation failure.”

An Executable Accounting Process

1. Sampling: Sample 50 to 200 real tickets, stratified by difficulty. Do not use only public hard problems.
2. Fixed protocol: Use the same system prompt, tool list, maximum rounds, and stop conditions.
3. Run by tier: Run the default tier on the full set. Run the max tier only on the subset where the default tier failed, avoiding meaningless full-set spending.
4. Billing basis: Account separately for input, output, cache hits, tool-side costs, and failed retries.
5. Quality basis: Automated test pass is only the lower bound. Add human spot checks for “whether unrelated changes were introduced” and “whether sensitive information was leaked.”
6. Decision: Calculate the pass-rate increment per additional unit of cost. When the increment is lower than the rework savings, downgrade.

An illustrative routing policy can be expressed as configuration. The fields have generic meanings and are not an interface of any cloud product:

route:
  default_effort: high
  escalate_to_max_when:
    - tests_failed
    - diff_lines_above: 400
    - security_sensitive: true
  max_task_budget_usd: 8
  stop_when:
    - same_error_repeats: 3
    - output_tokens_above: 120000
  cache:
    reuse_repo_context: true

When the budget cap is reached, stop and hand the task back to a human instead of continuing indefinitely. Repeating the same error three times usually indicates missing context or an unavailable tool. Adding more reasoning will only amplify cost.

Engineering Practices to Contain Cost

• Context slimming: Inject only relevant files and failure logs instead of stuffing the entire repository into every round. Use retrieval or dependency-graph pruning, store results in object storage, and read them back on demand.
• Prefix caching: Reuse stable specifications, directory structures, and build instructions as much as possible to reduce repeated input costs.
• Early stop and acceptance: Stop when tests pass. The review agent should look only at the diff and failing cases, not rewrite the full text.
• Model tiering: Use a small model or low reasoning tier for simple subtasks. Use the high tier only for planning and the final patch. Pass structured task tickets between subagents through a message queue to avoid long text back-and-forth.
• Observability: Track tokens, cost, success rate, and timeout by task, user, and repository. Alerts should focus on “week-over-week cost per task” and “max-tier share,” not only call count. The log service only needs to retain summaries of prompts and diffs; raw repositories should not be retained without bounds.
• Human gates: Changes involving permissions, billing, data deletion, and external interfaces must be merged by a human. Autonomous loops are suitable for generating candidate patches, not for bypassing code review.

If an organization already uses cloud model invocation and operations components—for example, Baidu AI Cloud—the observability described above can be implemented through general usage statistics, logs, and alerts, with costs split by project. There is no need to bind evaluation to a specific coding assistant UI. The key is that the gateway layer can distinguish effort tiers, quotas, and failure reasons.

How to Interpret “A New Upper Bound”

A higher upper-bound score means that search is more thorough under a specific question set and a specific reasoning budget. It does not automatically mean:

• default interaction is faster or cheaper;
• all business prompts can reproduce that gain;
• multi-agent work over several days equals deliverable product engineering.

A safer conclusion states the conditions clearly: question set, reasoning tier, whether tools are allowed, whether retries are counted, and the cost basis. If the conditions change, rankings can reverse. For internal communication, use comparison statements such as “default tier versus previous-generation high tier: quality equal or slightly higher, lower cost; max tier: better quality, significantly higher cost.” This is less misleading than citing a single total score.

Implementation Recommendations

Treat the default reasoning tier as the production baseline, treat the max tier as the failure escalation path, and set cost and round limits for every task. Review the max-tier share weekly. If a large number of requests start at the max tier, routing is too broad. If the pass rate after escalating default-tier failures is still very low, the bottleneck is unclear requirements or missing tools, not insufficient reasoning. Capability, speed, and unit price can improve together, but an upgrade is truly worthwhile only when total cost per task, rework time, and risk gates improve together.