Reasoning Levels and Billing: How to Evaluate LLM Cost-Effectiveness Scientifically

1. The Symptom: A 0.1 Version Bump, a Full Tier of Capability

A recent minor upgrade of a base model provides a revealing case study. The version number moved only from 5.1 to 5.2. Yet according to the officially published figures, the unit price dropped by roughly 20% ($8 per million input tokens and $40 per million output tokens), while the output throughput ceiling rose by about 30%. On these three numbers alone, anyone making a technology-selection decision would conclude that cost-effectiveness has improved significantly.

However, the cost section of the evaluation report tells a different story. At the high reasoning level, the average cost per task rose from $12.99 to $26.50, close to double.

These two sets of data do not contradict each other. They describe different terms of the same cost formula. What really determines the bill is not the unit price but “unit price × consumption.” For reasoning models, consumption swings sharply with the reasoning level, task difficulty, and number of failed retries. The core question of this article is: how do you calculate this cost curve properly, and use it to make tier-selection decisions and apply engineering governance?

2. Capability First: What the Three Metric Families Tell You

Evaluation scores should not be lumped together as “the model is stronger.” Scores from different dimensions carry completely different business implications.

1. Composite Intelligence Index: Measuring the Floor of General Reasoning

On the composite intelligence index (v5) from a third-party evaluation organization, the new version scored 65, up from 49 for the previous generation, a gain of 16 points. Indices of this kind are usually a weighted aggregation of tasks such as math, science, coding, and instruction following. A higher score indicates that the general reasoning baseline has moved up across the board. This benefits high-frequency short tasks the most, such as customer-support Q&A, document extraction, and structured output. These tasks gain little from long reasoning, but they immediately benefit from stronger instruction following and fewer factual errors.

2. Code-Generation Evaluation: Measuring Engineering Deliverability

On a mainstream code-generation benchmark, the new version reached 49.1% at the highest reasoning level, compared with about 45.5% for the previous flagship model. At a high but not the highest reasoning level, it scored 45.3%. That is essentially on par with the previous flagship at its highest level, at roughly half the cost per task. This comparison is more useful than the SOTA number alone. It shows that the marginal return on capability diminishes quickly: going from the next-highest to the highest level adds only 3.8 percentage points but doubles the cost.

3. Long-Horizon Agentic Tasks: Measuring Whether the Model Can Deliver on Its Own

The most informative results came from an end-to-end test. Starting from an empty directory, the model orchestrated more than a thousand sub-agents and ran continuously for about 8 days. It carried out coding, testing, and code review on its own. In the end, it produced a simplified operating system that runs on a laptop, covering the kernel, drivers, file system, graphical interface, login, and installer.

The capability threshold for this kind of task is completely different from that of conventional benchmarks. It tests four engineering capabilities:

• Task decomposition: breaking the goal into sub-tasks that can run in parallel.
• Context management: keeping critical information from being buried in a long-running conversation.
• Error recovery: locating and rolling back from build failures or failing tests on its own.
• Consistency constraints: keeping thousands of parallel outputs from breaking each other’s interface contracts.

These four capabilities are also the root causes of cost-per-task getting out of control, a point we return to below.

3. The Cost Math: Why a Lower Unit Price Can Mean a Bigger Bill

Start with the data. The key figures from the public evaluation, in USD per task, are summarized below:

Version and reasoning level	Composite intelligence index	Avg. output tokens	Avg. cost per task
Previous generation · highest reasoning level	49	~118K	12.99
New version · second-highest reasoning level	62	Not published	9.08
New version · highest reasoning level	65	~274K	26.50

Output tokens rose from 118K to 274K, an increase of about 132%, determined almost entirely by the reasoning level. But notice something important: even at the output price of $40 per million tokens, the pure output cost of 274K tokens is only about $10.96, leaving a gap of roughly $15.5 to the $26.50 total.

The gap comes from the input side. In agentic calls, every turn resends the full conversation history to the model, so input tokens accumulate:

# Illustrative cost model: single-turn vs. multi-turn agentic calls
IN_PRICE = 8.0 / 1e6    # USD per input token
OUT_PRICE = 40.0 / 1e6  # USD per output token
def single_turn_cost(in_tok: int, out_tok: int) -> float:
    return in_tok * IN_PRICE + out_tok * OUT_PRICE
def agentic_cost(turns: list[tuple[int, int]]) -> float:
    """turns: (input tokens, output tokens) per turn; input already includes conversation history"""
    total_in = sum(t[0] for t in turns)
    total_out = sum(t[1] for t in turns)
    return total_in * IN_PRICE + total_out * OUT_PRICE
# Working backwards: if total cost is $26.50 with 274K output tokens
#   Output cost ≈ 274_000 * 40 / 1e6 ≈ 10.96
#   The remainder ≈ 15.54 falls entirely on the input side
#   Implied cumulative input ≈ 15.54 * 1e6 / 8 ≈ 1.94M tokens
# Conclusion: cumulative input tokens are about 7x the output tokens

This estimate yields a reusable rule of thumb: in long-horizon agentic tasks, the dominant cost item is usually the input, not the output. That makes the optimization priority clear too. The goal is not to trim output tokens, but to reduce the context size of each call.

Four measures you can put into practice right away:

1. Context compression: On each turn, send back only the conclusion and the key diffs, not the full tool logs. Summarize large chunks of file content before injecting them.
2. Prompt caching: Keep stable prefixes such as system prompts, coding conventions, and repository structure in the cache region. Cache hits are typically billed at a significant discount.
3. Sub-task isolation: Give each sub-agent an independent, short context window. Parent agents exchange only structured results (file path + checksum + test outcome).
4. Capped retries: Set hard limits on the number of turns, total tokens, and total duration per task. When a limit is exceeded, fall back or hand off to a human, so tasks don’t end up “burning money in an infinite loop.”

4. The Key Decision Variable: Reasoning Level, Not Model Version

Back to that table. The row that is most useful for decisions is the second-highest reasoning level. It shows a composite intelligence score of 62, above the 59 that the previous flagship achieved at its highest level. Its per-task cost of $9.08 is also lower than that flagship’s $10.04.

This reveals a fact that is often overlooked. The reasoning level is a dial that can be adjusted independently, and its impact is often greater than switching to a different model version. A layered configuration is recommended:

• High-frequency, low-difficulty tasks (classification, extraction, intent recognition, formatted output): the lowest reasoning level. The correctness of these tasks is determined by prompt and constraint design. Adding reasoning budget only pushes up latency and cost.
• General engineering tasks (single-file changes, unit-test generation, routine refactoring): the second-highest reasoning level, which is the cost-performance sweet spot in the table above.
• High-value, wait-tolerant tasks (hard-to-locate defects, cross-module architectural rework, critical-path code review): the highest reasoning level. Accept the roughly 2x cost premium, provided the cost of a single failure on the task is high enough.

The most intuitive way to judge whether an upgrade to a higher level is worth it is “return per unit of cost”:

def marginal_value(base_score: float, base_cost: float,
                   next_score: float, next_cost: float) -> float:
    """Score gain per additional dollar; below a threshold, upgrading is not recommended"""
    if next_cost <= base_cost:
        return float("inf")
    return (next_score - base_score) / (next_cost - base_cost)
# Second-highest → highest level (code evaluation): (49.1 - 45.3) / (26.50 - 9.08) ≈ 0.22 points per dollar

For most teams, an upgrade whose marginal return is below 0.5 points per dollar is better reserved for a very small number of critical tasks.

5. Real Capability Gains: Long-Horizon Autonomous Work and Scientific Problem Solving

Beyond general benchmarks, there are two capability signals that developers should pay attention to.

Long-horizon autonomous development is now in a form that can be handed off. Starting from an empty repository, running for 8 days straight, coordinating more than a thousand parallel sub-agents, and delivering a runnable operating system show that the chain of tool calling, parallel scheduling, and self-checking can maintain consistency over an extended period. For engineering teams, this means that batch, unattended work can now move into pilots. Examples include bulk dependency upgrades, large-scale test-matrix completion, and boilerplate code migration. But they must come with three engineering constraints: sandbox isolation, artifact verification, and the ability to roll back changes. Otherwise, the “correctness” of what the model completes autonomously cannot be verified, and the risk is out of control.

Open scientific problems are being pushed forward in practical terms. The report notes that the model produced conclusions on two open problems related to black holes. One of them was a computation for which a method was proposed back in 2003 but had long remained unfinished because of the scale of compute it required. Results of this kind need to be put in context. The model does not replace a scientist’s judgment. Instead, it applies an existing methodology at a scale of computation that was previously out of reach. The takeaway for engineers is that, for existing problems where “the method is well defined but the computation is too large,” the model’s value lies in filling in the intermediate derivations and automating the execution chain. Such tasks are worth a larger budget for reasoning tokens.

6. Governance: Keeping Costs in Check

Conclusions from a technology-selection exercise have to be turned into engineering mechanisms you can actually run. Otherwise, even the most accurate calculations stay on paper. All four layers of defense below are essential.

Layer 1: Build your own evaluation set. Public benchmarks can tell you whether a model is usable. They should not decide which reasoning level you use. Pull 200–500 labeled samples from your real production logs. Cover high-frequency scenarios and past failure cases. Run the full set whenever you change the model or the prompt. Record accuracy, P95 latency, and cost per sample each time.

Layer 2: Routing and fallback. Use a lightweight classifier or rules to judge task difficulty first, then route each task to the appropriate level. When budget approaches a threshold, downgrade the level automatically. When repeated failures occur after a downgrade, retry at a higher level. The routing layer is the only place where quality and cost can be optimized at the same time.

BUDGET_PER_TASK_USD = 3.0
HARD_TOKENS = 400_000
class BudgetGuard:
    def __init__(self):
        self.spent = 0.0
        self.tokens = 0
    def allow(self, est_cost: float, est_tokens: int,
              level: str) -> str:
        """Return the reasoning level allowed for this call; may trigger a downgrade"""
        if self.tokens + est_tokens > HARD_TOKENS:
            raise RuntimeError("Task token limit exceeded; aborting and handing off to a human")
        if self.spent + est_cost > BUDGET_PER_TASK_USD:
            return "low"
        return level
    def commit(self, in_tok: int, out_tok: int):
        self.tokens += in_tok + out_tok
        self.spent += single_turn_cost(in_tok, out_tok)

Layer 3: Observability. For every task, record input/output tokens, cache hit rate, reasoning level, number of retries, and actual spend. Feed these metrics into your monitoring, alerting, and logging services. Set quotas and threshold alerts per business line and per caller. Without usage attribution at the caller level, no cost optimization is possible. This is the most commonly missing piece in practice.

Layer 4: Architectural decoupling. Model calls should sit behind a unified gateway, decoupled from business code. That way, changing reasoning levels, fallback policies, or caching policies does not touch business logic. It also turns “the model is replaceable” from an aspiration into an engineering reality. Such gateways are typically deployed on a container platform. Object storage holds evaluation sets and call traces. Logging services retain audit records. Monitoring and alerting handle quota and cost thresholds. On Baidu AI Cloud, these general-purpose cloud capabilities can be combined into a complete governance loop.

7. Four Common Misjudgments

• Treating a lower unit price as a lower cost. The unit price is only one term in the formula. In reasoning models, the variance in token consumption is much larger than the swing in unit price.
• Treating SOTA scores as business value. The 3.8 percentage points gained by going from the second-highest to the highest level might only show up in a real business setting as “the occasional failure rate dropping from 8% to 5%.” Convert it into manual rework hours before deciding whether it is worth it.
• Defaulting every task to the highest level. Using the top level for high-frequency short tasks is a classic misallocation of resources. It raises both cost and P95 latency.
• Ignoring the input-side overhead. As shown above, input tokens can be several times the output tokens in long-horizon tasks. Context governance is therefore the top priority for cost optimization.

8. Conclusion

The real lesson of this upgrade is not that “another model has gotten stronger.” It is that the way capability gains are achieved has changed. Performance improvements now come more and more from spending more reasoning tokens, rather than from compressing parameters or optimizing the architecture. That means the methodology for model selection has to change too: instead of comparing unit prices and leaderboard rankings, compare “the cost per task at the target quality level.”

Three practical takeaways:

• Default to the second-highest reasoning level, and reserve the highest level for a small number of high-value tasks.
• Tackle context and caching first, rather than compressing the output.
• Validate quality with your own evaluation set first, then control cost with layered routing and budget guardrails.

If you do these three things, your bill stays under control no matter how the base model evolves, and capability gains turn into real business value.