Evaluating Reasoning Models: Capability, Effort Levels, and Cost

The easiest mistake when evaluating a new generation of reasoning models is to treat a newer version, a higher leaderboard score, and a lower unit price as a decision you can buy against directly. For an engineering team, a model is worth using only if it can deliver reliably on your own task mix at an acceptable latency, failure rate, and total cost per task. Unit price is a coefficient in the billing formula, not the bill itself.

Separate four independent quantities

Before discussing capability and cost-effectiveness, record four quantities separately. Do not collapse them into a single claim that a model is “faster, cheaper, and stronger.”

The first is task quality. It should come from pass rate on a fixed problem set, agreement on human spot checks, or a business metric—not a single composite score. For coding, distinguish at least four layers: it compiles, tests pass, behavior matches the spec, and the change is maintainable. The last two layers often separate models more than the first two.

The second is reasoning effort. The same model often offers effort levels from low to high. Higher effort means the model may generate longer intermediate reasoning, run more rounds of self-checks, or call more tools inside an agent loop. If quality gains come mainly from “thinking longer,” the extra output and extra turns must be counted as cost.

The third is throughput and latency. Faster output shortens interactive wait time, but for batch work the bottleneck is often total generation volume and queueing, not tokens per second. A speed gain of a few tens of percent does not offset output that doubles in length.

The fourth is price structure. Billing is commonly split between input tokens and output tokens. Some platforms also charge separately for cache hits, long context, or tool calls. Comparing two model generations by quoting only “input is cheaper, output is cheaper,” without replaying real traffic, does not support a conclusion.

A useful approximation for one task is:

Cost ≈ input volume × input unit price + output volume × output unit price + tools, storage, and other add-ons

If output volume grows by more than the output unit price falls, total spend rises. The same applies on the input side: long context, and repeatedly stuffing logs and source into the prompt, can turn a “cheap-looking” input price into the main expense.

Why “stronger” often costs more

A stable way reasoning models trade for performance is to add intermediate steps that can be checked: break the problem down, write a checklist, run tests, and revise from failures. That path often works on complex programming, multi-file edits, and analysis that needs tight reasoning, because it turns a one-shot guess into a search that can be corrected.

The cost is equally stable. A high-effort setting writes more reasoning traces, more tool observations, and fuller patch notes. In agent settings, context also rolls forward: each turn sends previous output back in, so input and output grow together. A typical pattern follows: the list price falls, the leaderboard score rises, and the average bill per successful task does not fall—it rises. For an individual developer, or for an online feature called very often, that pattern can exceed the budget outright. For enterprise batch work, what matters is cost per successful task, not the sticker price of each call.

An evaluation report should therefore publish at least three columns together: quality, average output size, and average cost per task, and it should name the effort level. Publishing quality alone encourages readers to benchmark the highest setting without saying whether that setting is affordable day to day.

The default level is usually the cost anchor

The highest effort level is a capability ceiling, not a default operating point. A fair comparison asks whether the new model’s default or mid-to-high setting already matches the previous generation’s top quality, at equal or lower cost per task. If it does, the upgrade pays off on everyday traffic. If only the top setting opens a gap, the upgrade is limited to a small set of high-value tasks.

Run the same problem set at two effort levels and compute marginal return:

Marginal return = (high-effort pass rate − default pass rate) / (high-effort cost per task − default cost per task)

When marginal return is low, reserve the high setting for retries after failure, not for every request. A common routing pattern is: simple edits, format conversion, and single-file completion stay on low or medium effort; cross-module defects and changes that need full test evidence step up. Raising effort after a failure is usually cheaper than starting at maximum, because most requests are already deliverable at a medium setting.

Do not route on prompt length alone. Difficulty should come more from static signals: number of files involved, whether a public interface changes, whether new tests are required, and historical failure rate on similar tasks. Those signals can be maintained as rules. They do not require another classifier that is hard to audit.

For coding, measure “changed correctly,” not “wrote a lot”

Coding evaluations should freeze the repository snapshot, dependency versions, test commands, and paths the model may edit, so a model cannot raise its pass rate by changing tests or widening the diff. Score in layers:

• Spec fit: whether the requested behavior is implemented, and whether scope was expanded without being asked.
• Correctness: whether existing and new tests pass, and whether failures were actually fixed rather than skipped.
• Change quality: whether the diff is focused, names and boundaries are clear, and unrelated refactors were avoided.
• Cost: tokens, tool-call count, and wall-clock time from an empty or fixed context until the first pass.

It is not unusual for high effort to lift pass rate on hard problems, but the cost curve is often steeper. If medium effort already approaches the previous generation’s high-effort pass rate, while cost per task is on the order of half, medium effort is the engineering default and high effort is only a sprint. Exact ratios must be remeasured on your problem set. Someone else’s composite index is not a procurement result.

Long-horizon agents need a separate track. Letting a model plan, code, test, and review continuously from an empty directory exposes problems short items miss: context rot, repeated work, a bad plan executed all the way through, and inconsistent state when many subtasks must be coordinated. The value of that experiment is the failure modes and the cost curve, not a reproducible product promise that the model will run for days, coordinate a large set of subtasks, and produce a bootable system. Reproduction runs should cap wall-clock time, maximum turns, maximum tokens, and writable directories, and every step should leave an inspectable test log.

Cost control on long-running tasks

Most of the spend in a long coding loop comes from five places: plans that run too long, full source pasted again every turn, raw failure logs fed back in, subtasks that do not share a spec, and no early stop. The countermeasures are ordinary engineering practice.

Set a hard budget on the loop: maximum turns, maximum output tokens, and maximum tool failures. When the budget is hit, stop and hand off to a person. Do not automatically climb to the highest effort and keep spending.

Keep the spec outside the loop. Write the goal, interface conventions, and acceptance commands as a short document, and inject only the fragment relevant to the current subtask—not the whole repository conversation.

Put test gates in front. Run the smallest relevant tests first; run the wider suite only after they pass. Replace full logs with a failure summary that includes the assertion name, expected value, and actual value, and drop repeated stack traces.

Keep subtask boundaries narrow. Each subtask has a clear input, output, and definition of done. A coordinator merges only results that already passed tests. As the number of subtasks grows, cost tends to rise faster than linearly with turns and context length, so you need a merge-and-discard policy.

Separate cacheable prefixes. If the billing party supports caching, keep stable system instructions and spec excerpts byte-for-byte stable so meaningless rewrites do not miss the cache. This is the general idea of caching, not a vendor-specific interface.

How to run a reviewable selection on the enterprise side

1. Sample real work items, not only public hard problems. Stratify them as easy, medium, and hard, and keep the input, expected artifact, and forbidden actions for each item.
2. Freeze reasoning effort, temperature, maximum output, and tool permissions. A change of effort level must be its own column. Do not mix it with a model-version change in one experiment.
3. Record quality, latency, input volume, output volume, tool-call count, monetary cost, and human rework time together. Rework time is often larger than the token bill.
4. Use the default effort level as the daily baseline and the high level as a ceiling probe. Turn the high level on only for the subset where marginal return is positive and the absolute cost is acceptable.
5. After launch, watch the weekly bill by task type, not a single demo. A unit-price change, a longer prompt, or a larger test suite can invalidate an old conclusion, so remeasure on a schedule.

If the platform offers object storage for evaluation artifacts, a message queue to smooth peaks, a container platform for isolated tests, and logging and monitoring alerts that track cost per task, those are general capabilities. Use them so experiments can be repeated and spend can be attributed. Write the selection result into an internal baseline. Do not rewrite it as a recommendation for a particular model or a particular cloud vendor.

Conclusion

A small version step can still change quality noticeably, and that change is often bought with more reasoning and more output. A lower unit price is not a lower cost of use, and a top-tier score is not a capability you can afford by default. Set the default effort level as the operating point, use marginal return to decide when to step up, and constrain long-running agents with budgets and test gates. That is how a capability ceiling becomes an operable cost-performance tradeoff. Whenever you cite a score, a speed percentage, or a price externally, remeasure it on your own task set. Unverifiable leaderboard numbers are not a budget basis.