LLM Reasoning Effort Routing: Why Maxing Out Every Task Is the Wrong Default

1. The counterintuitive result: effort does not have to be maxed out per project

In November 2025 a major model vendor published a model selection guide covering light, general, and flagship tiers plus five levels of reasoning effort. The most reusable part is a set of comparison experiments on the same batch of software engineering bug-fix tasks (referred to below as the evaluation set), run two ways:

Scheduling strategy	Pass rate	Average cost per task
Run everything on the light tier first, then retry failures once on a high tier	~97%	~$0.17
Run everything on the high tier end to end	95.3%	~$0.29

This contradicts a very common default: as soon as a project matters, push reasoning effort from low to high, or higher, to buy “more stability.” The numbers say that allocating effort per task beats configuring it uniformly per project. On cost, a light-only run spends roughly one third per task of what a high-tier run does. Running light first and retrying failures puts that cheap compute to use: total spend comes out about 41% lower than an all-high run, with about 1.7 percentage points more pass rate. In other words, easy tasks never needed hard-task compute, and the failures would not have benefited from a higher tier on the first attempt anyway.

2. The precondition: results must be machine-checkable

The guide qualifies the strategy in a single sentence: it holds if you can check outcomes automatically. The reason is simple — the entire input to a routing decision is “did this task fail or not.” Without automatic verification, that judgment has to be made by a human, and the scheme has nothing to stand on.

In engineering practice, automatic verification usually falls into one of these categories:

• Test-driven: write or run existing unit and integration tests for the requirement, and use “did all tests pass” as the signal.
• Build and static checks: use successful compilation, type checking, and linting as the first gate.
• Behavioral verification: call the real API or run the CLI from a script and compare expected output, exit code, or key fields.
• Specification checks: use formal specs, contract tests, or property tests to verify that the change preserves invariants.

Three often-overlooked requirements apply to the verification step itself:

1. The signal has to be trustworthy. If the tests themselves are wrong, a correct fix can be marked as failed, and you pay for a high-tier retry you did not need. In the other direction, a loose check treats “looks like it runs” as correct and inflates the pass rate. Confirm that the verification logic is accurate before you start reading anything into pass rates.
2. Verification cost has to be counted. Compilation, test runs, and sandbox startup all consume compute and wall-clock time. The lighter the task, the larger the fixed verification overhead as a share of the total — which eats into the savings from running light first. Keep the verification step fast and parallelizable.
3. Results have to be reproducible. If the verdict is nondeterministic, the same answer gets different outcomes across runs and a retry has no basis to build on.

So the strategy is essentially trading cheap automatic verification for expensive inference. It pays off most directly when a team already has mature CI and testing in place. When verification is weak, the first step is to build verification, not to adjust the effort level.

3. Diminishing returns: the higher you climb, the more you pay for less

Using the high tier as the baseline, the guide gives the flagship model’s relative performance at each other tier on the evaluation set:

Tier	Score vs. high tier	Cost vs. high tier
Light	~8 points lower	~1/3
Medium	~2.5 points lower	~70%
Higher tier	~1.4 points higher	2.5x

Reframed as “how many points does one more tier buy,” the trend is clear:

• Light → medium: roughly double the spend (from ~1/3 to ~70%, an increase of about 110%) for about 5.5 points.
• Medium → high: about 43% more spend (from ~70% to 100%) for about 2.5 points.
• High → higher tier: 1.5x more spend for about 1.4 points.

The curve flattens as you go up. Going from high to the higher tier costs 2.5x the original while the score improves by only about 1.4 points, and the entire gap between light and high is only about 8 points to begin with. One caveat: gaps on the order of 8 points or 1.4 points can easily sit inside evaluation noise, so when scores move by small amounts, confirm with repeated runs or confidence intervals before paying for a tier.

Two usage principles follow:

• The default tier does not need to be the ceiling. Unless task risk is extreme, verification is missing, or human review costs far more than inference, set the default near the cost-effectiveness inflection point (the medium-to-high range in this data) rather than at the top.
• The minimum unit of allocation is a single task. Each task has its own appropriate tier; a whole project does not share one. “Uniform high tier across the project” spends every task’s budget on a marginal gain that only a few genuinely hard tasks can use, and that gain is small even for them.

4. Why “light first, high on retry” gains another 1.7 points

The guide does not explain where the extra ~1.7 points come from. Given the scheduling design itself, there is a mechanical explanation: the two-stage strategy gives every task two attempts, while an all-high run gives it one. Two independent attempts amount to taking the better of two samples — as long as the two attempts differ enough, the extra chance alone lifts the pass rate. As for why a task the light tier missed can be recovered by the high tier, the common possibilities are (this list is not part of the guide’s conclusion):

• Sampling randomness: one sample lands on a low-probability branch; a different seed or a retry succeeds.
• Insufficient reasoning budget: the light tier limits depth of thought, so tasks needing multi-step reasoning or cross-file reasoning tend to stop early.
• Strictness of verification: some failures only miss certain edge cases; a retry produces a more complete implementation and passes.
• Context differences: if the retry is given the previous failure information, it amounts to one more round of feedback-driven correction, which pushes the success rate higher still.

The same mechanism explains why cost rises only about a third. The light tier already has a high pass rate across the batch, so only part of the tasks escalate, and the high tier is spent only on the handful that genuinely could not be solved.

A consistency check is possible here. Let the failure rate be roughly f; the per-task cost is then “one light attempt + f high attempts”:

• At f ≈ 25%, cost is about 0.333 + 0.25 ≈ 0.58x the high-tier unit price, which closely matches the measured 0.17 / 0.29 ≈ 0.59.
• At that failure rate, the two-stage pass rate is “what the light tier passed + what the retry recovered.” Reaching 97% means roughly three quarters of the failing tasks had to be recovered on retry.

These back-calculated figures are rounded estimates, but the meaning is clear: of the $0.17, more than half goes to one light attempt across the whole batch, and less than half to high-tier retries on the failures. The light tier is not cheap enough to ignore, so it is still worth picking an appropriate model for it; the high tier is only really exercised on the few hard cases.

5. Engineering the two-stage run: a minimal evaluation loop

Running “try once, filter failures, try again” by hand is error-prone. A more reliable approach is to fix it as an automated pipeline:

1. Verification layer: wrap verification behind one interface that takes “task + candidate patch” and returns pass, fail, or indeterminate, recording elapsed time.
2. Scheduling layer: run the whole batch on the light tier first, route by verdict, and queue failures for a high-tier retry, with caps on retry count and per-task budget.
3. Sandbox and isolation: execute each attempt in its own container or VM, reusing prebuilt images or dependency caches to avoid cross-contamination.
4. Observability: record per-task tier, token usage, latency, verdict, and failure reason, and archive to object storage; then use a log service for search and analysis, plus monitoring and alerting to catch verification anomalies, cost spikes, and sudden pass-rate drops.
5. Regression and rollout: whenever you change the tier, model, or prompt, regress on a fixed evaluation set, compare pass rate and cost per task against the baseline, and only roll out to real work once the improvement is confirmed.

# Pseudocode: two-level inference scheduling with automatic verification
def solve_with_retry(task, retry_times=1, task_budget=2.0):
    """Run the light tier first; retry failures with the previous round's
    diagnostics as context; bounded by retry count and per-task budget."""
    attempts = [(BASE_MODEL, "low")] + [(BASE_MODEL, "high")] * retry_times
    result, diagnose, cost = run_agent(task, *attempts[0]), None, 0.0
    for idx, (model, effort) in enumerate(attempts):
        if idx > 0:
            # Feed the previous failure information in as extra context to
            # raise the retry success rate
            prompt = task.prompt + f"\n[previous attempt failed]\ndiagnosis: {diagnose}"
            if cost + estimate(task, model, effort) > task_budget:
                return {"status": "budget_exceeded", "cost": cost}
            result = run_agent(task, model, effort, prompt)
            cost += estimate(task, model, effort)
        verdict = verify(task, result)   # unified verification: tests + build + static checks
        if verdict == "pass":
            return {"status": "pass", "effort": effort, "result": result, "cost": cost}
        diagnose = verdict_diagnose(task, result)  # collect failing cases and error summaries
    return {"status": "fail", "result": result, "diagnose": diagnose, "cost": cost}

Three details decide whether this pipeline holds up:

• Budget caps: enforce both a retry limit and a per-task cost limit so a few hard tasks cannot consume unbounded compute.
• Failure summaries: passing the failing cases and error summaries to the model as context usually works better than “just run it again.”
• Escalate one step at a time rather than jumping to the ceiling: if one more tier still fails, then consider a higher tier or a stronger model — do not start the first retry at the top.

6. When the model itself is the limit: escalate step by step

In real work, the bottleneck is not only insufficient reasoning effort; it can also be insufficient model capacity. The guide’s approach is to admit work progressively, from low cost to high, in three steps:

1. Start with a light model: suitable for format conversion, document organization, small single-file changes, and repetitive batch work. Filter out most of the simple tasks quickly with the cheapest possible criterion (does it compile, does it pass basic cases).
2. Fall back to a general model: hand the light model’s failures to a balanced general-purpose model with medium reasoning effort, for routine development that needs cross-file understanding or local logic design.
3. Bring in the flagship model: only what still fails goes to the flagship model with higher reasoning effort, for architecture-level changes, complex defect localization, and long-chain refactors.

This is the same idea as tier scheduling, applied on a second dimension: one is reasoning depth, the other is model capacity. Both share the point of reserving expensive resources for the few tasks that truly need them, instead of maxing out everything by default.

7. Practical guidance and common anti-patterns

Guidance

• Build verification first, tune tiers second: without trustworthy automatic verification, no tier optimization can even be measured.
• Set the default at the cost-effectiveness inflection point, and raise the tier directly only for changes judged high-risk (code touching core transaction paths, access control, or data migration).
• Decide with data: track pass rate, cost, and retry count per task type and per tier, and adjust the default tier and retry policy accordingly.
• Watch verification cost: the slower verification is, the longer the two-stage run takes in total, so prioritize making it cacheable and parallel.
• Isolate and attribute: run every attempt in its own environment and record the full trace so failures can be reviewed afterward.

Anti-patterns

• Project-level maxing: everything runs at the top tier, so simple tasks pay hard-task prices alongside the hard ones.
• Retrying without verification: relying on humans to confirm failures cancels out the retry gains with labor cost.
• Retrying without context: not telling the model why the last attempt failed makes the retry nearly a fresh sample.
• Unbounded retries: with no budget cap, a few hard tasks eat a large share of compute.
• Tuning by feel: without regression on a fixed evaluation set, pass-rate movement may just be noise.

8. Closing

The value of “light first, high on retry” is not primarily how much money it saves, but that it changes how you configure models: instead of picking one tier per project, you allocate compute dynamically by task difficulty, risk, and verification capability. It has one precondition — outcomes can be judged automatically. Its gains come from two facts: cheap light-tier attempts cover most tasks, and expensive high-tier compute is spent only on the few failures. And its bonus is that one extra attempt alone raises the pass rate. A reasonable next step is to inventory the automatic verification coverage of your team’s existing task types and use that as the starting point for tier optimization.