How to Choose a Reasoning LLM: True Cost and Capability Limits Behind List Price

When a new model is released, the announcement usually focuses on three claims: a percentage cut in list price, a score on a public benchmark, and a lead over the previous generation on tasks that require long-running parallel search. Each claim can be checked individually, but they cannot be chained into the conclusion “better price-performance” the way release notes often imply. The issue is not that the numbers are false; it is that they use different measurement units. List price is quoted per million tokens, benchmark scores are reported for a specific reasoning setting, and large-scale parallel tasks are measured by end-to-end success rate. Putting them in the same comparison mixes incompatible units.

This article does not target any specific product. It breaks such release information into verifiable structures and provides a reusable method for calculating cost and evaluating capability across model services. All prices and scores in this article are illustrative values used to demonstrate the method; they do not represent actual quotes from any vendor.

1. First align the units: list price is not the bill

Mainstream reasoning model services usually price four components: non-cached input, cache-read input, cache-write input, and output. Cache write is generally charged once as a multiple of the input price, while cache read is usually a small fraction of the input price. The key point is that vendors do not set the cache-read ratio the same way. Even if two discounts sound similar, the resulting absolute unit price can differ by several times, and this component is often the largest in long-session scenarios.

So the first step is not to compare discount percentages, but to normalize every tier to an absolute price per million tokens and then weight it by the actual traffic mix.

A reusable cost model

Assume a platform has the following price structure (illustrative values, input/output in USD per million tokens):

Pricing component	High-capability tier	Balanced tier
Non-cached input	8	4
Cache read (ratio of input price)	2.5% → 0.20	5% → 0.20
Cache write (ratio of input price)	12.5% → 1.00	25% → 1.00
Output	40	20

What is interesting is that the absolute prices of cache read and cache write are designed to be identical in the two tiers. This is not a coincidence: vendors have an incentive to make long-session users perceive no difference in same-price components and concentrate the gap in non-cached input and output.

Writing this model in Python is more reliable than doing the arithmetic by hand in a spreadsheet:

from dataclasses import dataclass
@dataclass(frozen=True)
class PriceCard:
    """Unit: USD / million tokens"""
    input_price: float
    output_price: float
    cache_read_ratio: float      # Ratio relative to non-cached input price
    cache_write_ratio: float     # Ratio relative to non-cached input price
    def session_cost(self, *, input_mtok: float, output_mtok: float,
                     hit_rate: float, write_mtok: float = 0.0) -> float:
        cached = input_mtok * hit_rate
        fresh = input_mtok - cached
        return (
            fresh * self.input_price
            + cached * self.input_price * self.cache_read_ratio
            + write_mtok * self.input_price * self.cache_write_ratio
            + output_mtok * self.output_price
        )
high = PriceCard(8.0, 40.0, 0.025, 0.125)
balanced = PriceCard(4.0, 20.0, 0.05, 0.25)
# A typical long session: 1M cumulative input, 90% cache hit rate, 50K output, ignoring cache writes
c_high = high.session_cost(input_mtok=1.0, output_mtok=0.05, hit_rate=0.9)
c_bal = balanced.session_cost(input_mtok=1.0, output_mtok=0.05, hit_rate=0.9)
print(c_high, c_bal, c_high / c_bal)   # 2.98  1.58  1.886

At a 90% hit rate, the bill gap is about 1.89x. This is slightly smaller than the 2x gap in list price, but it is still far from negligible.

How much can cache hit rate save? Make the variables explicit

A common question is whether raising the cache hit rate from 90% to near 100% can erase the price gap. Plugging the numbers into the model gives the answer: at a 0.95 hit rate, the ratio is about 1.85x; at 1.00, it is about 1.83x. The conclusion is that caching can narrow the gap, but it cannot eliminate it, because most of the price gap lies in the output unit price, and output is not affected by caching. This also explains a common misjudgment: some people argue that a higher-priced tier is worthwhile because “our cache hit rate is high,” but in reality the hit rate only affects the less-than-0.2x portion of the difference.

What can actually change the conclusion is the input-to-output ratio. For workloads that produce very little output, such as code retrieval and summarization with an input-to-output ratio of 40:1, there is almost no cost-advantage scenario for the higher-priced tier. Conversely, if the workload mainly generates long text from short inputs, the output-price gap between the two tiers will dominate the bill.

2. Benchmark scores are not the same measuring stick

The second type of information that is often overinterpreted is benchmark scores. Reasoning models generally provide multiple reasoning-strength settings, commonly named low, medium, high, and very high. The same model can score differently across these settings by several percentage points, and higher settings increase thinking-token cost and latency for each call.

This creates a common mismatch: comparing a flagship model’s highest setting with another model’s fully enabled setting, then concluding that the difference is “only 0.2 points.” The problem is that the default setting used in real calls is usually a middle setting, and users do not normally keep the highest setting enabled just to chase benchmark scores. Using illustrative data: a model may score 49.1% at its highest setting and 45.3% at high, while the default setting used for everyday calls may be around 43%. That gap of about 2.3 percentage points matters much more during model selection than a 0.2-point difference.

When aligning benchmarks, it is best to fix three variables: reasoning setting, sampling and retry policy, and task-set version. If any of these is not fixed, the scores are not comparable.

Three metrics beyond scores

• Average cost per task: convert all calls required to complete a task successfully, including retries, rework, and verification, into a dollar amount rather than using the price of a single call.
• First-pass rate: the percentage of tasks that can be delivered without human intervention. This is the most experience-relevant metric in coding-agent scenarios, and it is also the hardest to infer from public leaderboards.
• End-to-end completion rate: the percentage of pipelines completed within a limited number of steps and budget, used to assess stability on long-horizon tasks.

3. Long-horizon parallel tasks: the largest gap, and the farthest from daily work

The claims most likely to create a large gap in release materials often involve tasks that require large-scale parallel agents to retrieve, produce, and machine-verify results, such as formal proofs. The defining feature of these tasks is that errors can be immediately judged by a compiler or proof checker. Therefore, large-scale parallel sampling can turn “can do it but with low probability” capability into a visible success rate. In an illustrative comparison, the same parallel setup moved a formal task from 26% to 49% when using the high-capability tier.

This result has two technical implications. First, automatically verifiable tasks best reflect the model’s true reasoning ceiling, because verification does not consume model capability. Second, they are the farthest from everyday code writing: in daily development, the correctness of many requirements cannot be automatically determined, and answers that “look right” can hide capability differences. Therefore, directly converting a lead on such tasks into a coding-efficiency improvement is an obvious overreach.

For engineering teams, a more practical approach is to treat these verifiable tasks as offline capability probes. They are suitable for judging the model’s upper bound and shortlisting candidates, but they are not suitable for predicting delivery timelines.

4. Hidden variables in marketing claims

• Choice of opponent: the compared model is often shown at its highest score or fully enabled setting, rather than its default configuration.
• Footnote placement: limiting conditions may not be in the main text but in footnotes or supporting documents, such as “this score requires enabling a certain setting” or “this price applies to the batch API.”
• Missing absolute amounts: saying “20% cheaper” without giving specific unit prices prevents readers from aligning the offer against other platforms.
• Benchmark saturation and contamination: benchmarks near the ceiling contribute little to differentiation, and public test sets may overlap with training data.

The test is simple: for every selling point, ask “under what configuration, task set, and pricing dimension does this hold?” The parts that cannot be answered are the parts that should not be used for decisions.

5. Execution: make model selection a rollback-capable engineering action

Do not make a binary procurement decision. Instead, build a switchable and measurable routing layer at the call layer.

# Pseudocode: route models by task type and remaining budget
def route(task, budget_left_ratio):
    if task.kind in {"retrieval", "summarization", "formatting"}:
        return "balanced"          # Short context, low difficulty
    if task.verifiable and budget_left_ratio > 0.4:
        return "high"              # Automatically verifiable and sufficient budget
    return "balanced"              # Fallback when budget is tight

Supporting engineering measures include: keeping stable prefixes to maximize cache hits by placing system prompts and project context first and avoiding meaningless changes; trimming and summarizing long contexts to reduce input tokens; setting unified step and dollar limits for parallel agents to prevent runaway task costs; and using logging services and monitoring alerts to record token consumption and failure reasons for each task type, forming weekly cost-review curves.

The evaluation process is best done in four steps: extract 100 to 300 real historical tasks to build a golden set; run two candidate configurations in parallel under shadow traffic; compare them on first-pass rate and average cost per task; and start with 10% gray traffic for two weeks before deciding the primary-backup ratio. Keep a rollback switch throughout, and switching cost should be close to zero.

6. Conclusion

A sound model-selection judgment requires separating three types of information: normalize list price into absolute unit prices weighted by traffic structure to obtain a bill multiplier; align benchmark scores after fixing reasoning settings and task sets to obtain a capability multiplier; and use long-horizon parallel-task results only to judge the upper bound, not to predict daily output.

Returning to the opening example: after a 20% price cut, the high-priced tier still costs about 1.9x the balanced tier. The same cache price can only push that number down to just over 1.8x. The 0.2-point benchmark gap comes from full-setting versus full-setting, while the gap under the default everyday setting is larger. The truly significant capability lead appears in automatically verifiable long-horizon tasks, which are exactly the farthest from daily coding. All three claims can be true, but the conclusion that the model has “better price-performance” does not hold. For most teams, the practical approach is to use the high-priced tier for verifiable and high-stakes tasks, route the remaining traffic to the balanced tier, and decide the mix with telemetry and gray-rollout data rather than launch-event numbers.