How to Set Reasoning Effort for Coding Tasks Without Overpaying

In intelligent coding and software-engineering automation, inference strength can usually be set in tiers. A higher tier often spends more reasoning and tool calls on a single attempt, which raises latency and cost. Many teams default to the cautious choice: one high tier for the whole project. Public evaluation discussions repeatedly show a contrasting result: that project-wide cutoff can cost more without improving the final pass rate.

A more effective approach is to assign the tier per task. On a batch of programming problems that can be judged automatically, run everything once at a lower tier, then rerun only the failures at a high tier. In the comparisons discussed for this pattern, the two-stage policy can finish with a pass rate slightly above “one high-tier attempt per task,” while the average cost per task is clearly lower. The numbers move with the task mix, pricing, and how strict grading is, so they should not be treated as a fixed promise for any product. What they support is a repeatable scheduling rule: easy tasks should not be priced like hard ones, and extra budget belongs on the samples that failed.

Why “high effort all the way” is not the safe default

Treating the tier as a quality knob makes “higher is safer” feel obvious. The bill, though, follows consumption. Most of the extra spend on a high tier goes to tasks the lower tier would already have solved. If easy items dominate the set, an always-high policy repeatedly pays a hard-problem premium on samples that were already solvable.

Relative to a high tier, a low tier often scores lower, but its cost is commonly only a small fraction of the high tier. A mid tier sits closer to the high tier: the score gap narrows further, while cost has already climbed to a large share of the high-tier price. Pushing on to an ultra-high tier usually adds only a small score gain and can multiply spend. The curve flattens as effort rises, so each extra point of pass rate gets expensive quickly.

A two-stage policy also gets another draw. Failures from the low tier are solved again at the high tier; an always-high policy samples each task only once. On coding tasks whose results are stochastic and whose tool paths are not unique, an independent second attempt can recover cases a single high-tier run misses. That helps explain why “low, then high” can post a slightly higher aggregate pass rate than “high once,” while total spend stays lower: the high tier covers the failure subset, not the full set.

This is not a claim that the low tier is always better. It is a claim that, in retryable, auto-graded batches, budget should follow failure risk. If the product must return one answer on a single request and cannot rerun, a high tier or a larger model is still reasonable.

The policy needs an automatic verdict

Low-then-high works only when the system can tell which task failed. Usable checks include unit or integration tests, a successful build, static checks with no new errors, an interface-contract check, or a script that compares output with a reference. The verdict must be stable, repeatable, and aligned with the acceptance criteria. If the tests themselves are wrong, a low tier can “pass” a bad implementation and the high-tier rerun never starts.

If right and wrong can be decided only by reading a diff or holding a business review, the model savings are usually spent on review time. A better fit is to use a higher tier directly on high-risk changes, or to generate candidates at a low tier and send only large, wide-impact diffs to a person—not to escalate every failure mechanically.

Automatic grading also has to handle a few edges:

• Flaky tests: the same patch sometimes passes and sometimes fails, so it is misread as a reason to escalate and the budget is wasted. Isolate intermittent failures first, or require consecutive passes.
• Partial passes: you can escalate only when core tests fail, so style warnings do not buy a high tier.
• Timeouts and environment errors: infrastructure failure is not a model failure. Treating it as one upgrades an environment problem into more expensive inference.
• Evaluation leakage: grading scripts and hidden tests must stay out of the model context, or the pass rate stops meaning anything.

How to read each tier: score, cost, and the bend

Use one high-tier result as a baseline and compare the other tiers qualitatively, rather than memorizing a fixed set of percentages.

Low tier: scores are usually clearly below the high tier, but cost per task is often on the order of about one third of the high tier. It fits a first pass whose job is to clear easy tasks cheaply.

Mid tier: scores move closer to the high tier, and so does cost. If the low tier already fails often, starting at mid can be cleaner than “many low-tier failures, then high,” because each failed first pass is paid for too.

High tier: the default quality anchor for failed samples, high-impact changes, and work that is hard to grade automatically.

Ultra-high tier: versus high, the score lift is often small while cost can reach several times as much. Past the high tier, compare “one more tier on this model” with “a stronger model at one of its lower tiers” on the same batch, instead of assuming the next tier is the move.

Compare four numbers together: final pass rate, average cost per task, tail latency (the slowest slice of requests), and the share of work that needs a person. Pass rate alone invites unlimited escalation; unit price alone hides how many reruns you paid for.

Why a larger model can cost more per token and less per task

A higher list price does not mean a higher cost to finish one task. A stronger model may need fewer round trips, fewer useless tool calls, and a shorter rework path to reach the same goal. If a weaker model keeps rewriting tests and rereading the repository, its tokens and time can exceed one successful solve from the stronger model.

Choose on task cost, not on the sticker price alone. Task cost includes at least the prompt and context, model output, tools and the execution environment, failed retries, and human review. A strong model at low or mid effort can sometimes match a weak model at high effort, which moves budget from “turn this model up” to “switch models and turn effort down.”

Hold the task set, the grader, and the retry cap fixed. Record first-pass rate, average retries, how often the ultra-high tier fires, and total cost. With a small sample, a gap of one or two percentage points can be noise.

What to do when the model is not enough

If the current model still fails steadily at a high tier, do not immediately move all traffic to the largest configuration. A steadier order is:

1. Check that the failure is a capability gap, not a vague prompt, missing context, a bad test, or an environment fault. Filling in repository structure, relevant interface notes, and failure logs is often more effective than raising the tier.
2. On the same model, raise the tier only for the failure subset, and cap retries so the loop cannot run forever.
3. If spend above the high tier is already close to switching to a larger model, compare “current model, ultra-high” with “larger model, low or mid” on the same set. Keep the combination with lower task cost and a pass rate that does not drop, and make that the new default path.
4. For the tail that still fails, add a person or a stronger configuration. Do not let the tail set the tier for everyone.

That order reserves expensive capability for samples where the cheaper path has already failed.

How to apply this in a coding assistant

Do not hard-code the tier as a global constant. Write it as a policy. Set a default by task type, and let the evaluation pipeline override it.

A vendor-neutral sketch:

for task in tasks:
    result = solve(task, effort="low")
    if auto_grade(result) == "pass":
        accept(result)
    else if is_infra_error(result):
        retry_same_effort(task)      # do not escalate
    else:
        result2 = solve(task, effort="high", prior_failure=summary(result))
        accept(result2 if auto_grade(result2) == "pass" else escalate(task))

Implementation notes:

• Default by task type, not by locking a whole repository. Formatting, comment completion, and small renames stay low; cross-module defects, concurrency, and data migrations use a higher tier.
• On a rerun, attach a failure summary (failed test names, assertions, and the key stack) so the high tier does not start from zero. Cap the summary length so unrelated logs do not fill the context.
• Set a maximum number of escalations and a cost ceiling per task. Past the ceiling, mark the task for a person and record it as tail work so prompts or tests can be improved later.
• Track tier outcomes: share passed at the low tier, share recovered after escalation, and share failed at both stages. If the low-tier pass rate is too low, raise the starting tier. If escalation almost never recovers anything, the problem is not inference strength.
• For latency-sensitive interaction, you can keep a fast low-tier reply and escalate failures in the background, letting the user decide whether to take the second result. Batch evaluation does not need that parallelism; a sequential rerun is cheaper.
• Re-run a fixed task set on a schedule. After a model or pricing change, an earlier conclusion that “low is enough” can stop being true.

When not to use low-then-high

Use a higher tier or a stronger model directly when a single interaction cannot be retried; when the cost of a mistake is very high and there is no automatic test; when the task depends on long-horizon planning and a low tier will almost certainly fail; or when failures are already nearly the whole set, so two stages only pay for an extra low-tier pass.

The other way around: if the set can be graded automatically, easy tasks are common, and asynchronous reruns are allowed, assigning effort per task usually beats locking the project to a high tier. Budget should buy unsolved difficulty, not prepaid maximum insurance on every task.

Once the policy is in the evaluation pipeline, the team gets more than a lower average cost per task. It also gets an observable difficulty distribution: which tasks a low tier can deliver, which must be upgraded, and which are already beyond the current model. Later work on prompts, tests, or model choice can follow that distribution instead of turning the knob to maximum on instinct.