LLM Inference Benchmarking in Practice: Chain-of-Thought, Token Budgets, and Thinking Efficiency

Why “Score-Only” Benchmarks Are Losing Relevance

LLM iteration cycles are getting faster, but the information contained in leaderboard scores is shrinking. Under identical accuracy figures, different models, modes, and problem types show vastly different costs and behaviors. In one public logical reasoning evaluation, three phenomena appeared within the same test pipeline:

• Output bloat in non-reasoning mode: On complex problems, non-reasoning mode outputs could grow to 20K–30K tokens, far exceeding the 15K-tier limit set for that mode. Under a “zero score if over limit” rule, these runs could not be scored at all, making it impossible to complete the evaluation for that mode.
• Excessively long chain-of-thought in reasoning mode: A small number of hard problems produced outputs exceeding the 80K-tier limit across multiple passes. These were scored as zero, directly dragging down the median score for that mode.
• Identical full scores, vastly different costs: Different models could both achieve full marks on the same problem, yet one consumed more than three times the tokens of the other.

These three observations point to the same conclusion: an evaluation system must include “token budgets” and “thinking behavior” as part of its statistical scope. Reporting only “correct or not” while ignoring “how long it thought and how much it cost” means scores reflect neither true capability boundaries nor support cost decisions on the business side.

Set the Rules Before Discussing Scores: Key Design Considerations for Evaluation Criteria

A credible reasoning benchmark depends more on rule design than on the problems themselves. At minimum, four things must be defined:

1. Tiered output limits. Reasoning mode allows explicit chain-of-thought and should have a significantly higher limit than non-reasoning mode. Even though non-reasoning mode does not display the thinking process, you cannot assume it will always be brief.
2. Uniform and transparent over-limit handling. Scoring over-limit runs as zero is the most common approach. The advantage is consistency and preventing accidental credit for “stumbling onto the right answer halfway through.” The downside is that “thinking too long” can be misclassified as “not knowing how.” Therefore, beyond the score itself, the over-limit rate must be reported — it is a key indicator for distinguishing “capability gaps” from “runaway behavior.”
3. Multiple sampling with median aggregation. Inference results are stochastic, and single-pass answers are noisy. Sampling the same problem multiple times and taking the median significantly reduces the impact of chance on conclusions.
4. Per-problem cost accounting. Record input/output token counts, over-limit occurrences, and latency for every problem. Without this, you cannot answer the question: “What is the actual cost behind a full score?”

Unit price and total cost should also be calculated separately. Capability improvements may give vendors room to adjust pricing, but from the user’s perspective, a higher “per-token price” does not equal a higher “per-task cost”: if chain-of-thought length remains stable, total cost stays manageable. Conversely, if the unit price stays flat but chain-of-thought doubles, actual spending spirals out of control.

Fast Thinking Can Also “Silently Expand”: Rethinking Non-Reasoning Mode

Traditionally, non-reasoning mode has been treated as “fast and cheap,” with costs estimated linearly from averages. But recent observations suggest that some models in non-reasoning mode behave more like adaptive reasoning with a low thinking budget: simple problems can indeed be solved within a few thousand tokens, but once a problem becomes complex, the model may implicitly unfold reasoning inside its output even without an explicit chain-of-thought, rapidly consuming 20K or even 30K tokens.

This has three direct engineering implications:

• Model costs by distribution, not by mean. Upstream applications should budget based on P95/P99 output consumption rather than averages; otherwise, long-tail requests will breach cost guardrails.
• The “non-reasoning = cheap” assumption is no longer reliable. During model selection, measure the actual consumption curves of both modes on your target business problems instead of relying on labels.
• Constraining thinking length has a price. You can compress the reasoning space through system prompts or output limits, but you need to evaluate the loss in accuracy in parallel and find a balance between cost and quality that the business can accept.

Long-Chain Reasoning: The Tradeoff Between Solution-Space Exploration and Thinking Efficiency

Generational differences are particularly visible on multi-solution logic problems:

• Previous-generation models: Overall scores were not low, but on problems requiring “find all solutions,” their exploration of the solution space was insufficient. They tended to miss valid solutions — behaving more like “finding one answer” than “exhausting the answer space.”
• New-generation models: They can enumerate possibilities more thoroughly and find nearly all solutions, with notably higher accuracy. But the cost is that they tend to run at the edge of the thinking budget, consuming more than three times the tokens of the previous generation on complex logic problems.
• Efficiency advantage of frontier models: Given the same full score, their consumption can be as little as one-half or even one-sixth of the former.

This leads to an important judgment: accuracy is only the first gate; thinking efficiency (token cost per unit of accuracy) is the next core competitive dimension. For developers, the habit of recording a “per-problem token bill” is essential — two models with similar leaderboard scores may have total costs of ownership that differ severalfold in real business scenarios.

The Coding Benchmark Gap: From One-Shot to Real-World Agent Projects

Coding capability evaluation also has a clear disconnect:

• One-shot coding can be “inflated.” Standalone functions and small utility tasks can pass with a single generation. Improving these scores is relatively easy, but such tasks no longer represent real development.
• The real difficulty in actual projects is systemic. Multi-file context, implicit code conventions, environment differences, and regression verification — if any link breaks down, the generated code cannot be delivered.
• Debugging capability is the core weakness. Whether a model can locate the problem, modify the code, and re-verify after an error determines whether it is a “code generator” or a “usable engineer.” Recent progress of new-generation models in agentic programming is precisely reflected in multi-turn repair and verification loops on real projects, and they are entering the “usable” range for routine frontend and backend development tasks.

Therefore, agentic programming benchmarks should follow three principles: real repositories, multi-turn tool use, and pass-rate-based scoring — let the model read files, modify code, and run tests, and judge capability by final outcomes rather than single-pass output.

Make Benchmarking an Engineering Practice: A Reusable Scaffold

Benchmarking should not be a manual workshop; it should be engineered and reproducible. Here is a minimal benchmark scaffold:

import statistics
from dataclasses import dataclass
@dataclass
class Task:
    task_id: str
    prompt: str
    checker: object            # Scoring function: output text -> score
    max_output: int = 80_000   # Output token limit
def evaluate(tasks, client, rounds=3):
    """client is a unified model invocation interface"""
    report = []
    for t in tasks:
        scores, tokens, over_limit = [], [], 0
        for _ in range(rounds):
            resp = client.chat(
                messages=[{"role": "user", "content": t.prompt}],
                max_tokens=t.max_output,
            )
            tokens.append(resp.output_tokens)
            if resp.output_tokens >= t.max_output:
                scores.append(0)          # Over-limit runs score zero for consistency
                over_limit += 1
            else:
                scores.append(t.checker(resp.text))
        report.append({
            "task_id": t.task_id,
            "median_score": statistics.median(scores),
            "avg_tokens": sum(tokens) / len(tokens),
            "over_limit_rate": over_limit / rounds,
        })
    return report

Problem sets and raw outputs can be stored in object storage, call records aggregated through a log service, and abnormal fluctuations in over-limit rates captured through monitoring and alerting. These capabilities correspond to common product categories on mainstream cloud platforms (such as Baidu AI Cloud’s object storage, log service, and cloud monitoring), which can be combined as needed without building the entire infrastructure from scratch.

Conclusion: From “Benchmark Scores” to “Auditing”

Based on the above practices, four actionable conclusions can be drawn for developers:

1. Look at costs, not just scores: identical accuracy with token costs differing severalfold translates into completely different business experiences.
2. Model costs separately for the two modes: non-reasoning mode is not inherently cheap; long-tail consumption should be estimated using percentiles.
3. Treat the over-limit rate as a first-class metric: it reflects both capability boundaries and behavioral stability, and is the key to interpreting median scores.
4. Judge coding capability through real-world agent projects: one-shot scores are only an entry threshold; debugging and verification loops are the real dividing line for usability.

LLM benchmarking is shifting from “score chasing” to “auditing”: it must answer not only “how smart is it,” but also “how controllable and how affordable is it.” Building an evaluation system with transparent rules, consistent criteria, and computable costs is a required discipline for every technical team in an era of rapid model iteration.