Evaluating LLM Thinking Budgets and Long-Chain Reasoning: A Practical Guide

    This article explains how to evaluate reasoning LLMs beyond a single accuracy number, covering a three-dimensional framework of capability, cost, and stability. It shows why blurring fast/slow thinking boundaries distorts traditional scoring, how to measure solution completeness with set matching and partial credit, and why thinking efficiency has become the real competitive axis once capability catches up. It also covers realistic coding benchmarks and Agent feedback loops, long-context evaluation focused on long-range reasoning rather than retrieval, and cost-control strategies including mode routing, budget guardrails, tiered fallback, cache reuse, observability, and an abstraction layer. A deployment checklist and reproducible evaluation pipelines round out the practical guidance.

    1. Shifting the Evaluation Lens: From “Correct Rate” to “Cost of a Correct Answer”

    For a long time, accuracy was the core metric for evaluating a large model. But once reasoning models introduced an explicit “thinking” mechanism, correct rate alone is no longer enough to support engineering decisions. Given the same problem, model A may produce the correct answer with 3K tokens while model B needs 25K tokens for the same correct answer. In business terms the cost differs by nearly an order of magnitude, yet a traditional leaderboard shows none of this.

    A more reasonable evaluation framework should therefore cover at least three dimensions:

    • Capability: correctness, completeness across multiple solutions, and completion of long-horizon tasks;
    • Cost: output token consumption, end-to-end latency, and the minimum thinking budget required to reach a target accuracy;
    • Stability: variance across repeated runs, truncation rate, and format-compliance rate.

    The latest generation of domestic open-source models has indeed taken a step up in capability: their behavior on multi-step logic and long-horizon tasks is closer to how humans solve problems, bringing their overall reasoning-mode level back into the top tier among domestic models. But the cost structure has shifted as well—unit prices have risen somewhat, and non-reasoning mode increasingly behaves like “an adaptive reasoning model with a compressed thinking budget,” with output length no longer controllable. These changes must be reflected in evaluation methods, or model-selection conclusions will be distorted.

    2. The Boundary Between Fast and Slow Thinking Is Blurring, and Reusing Old Scoring Rules Distorts Results

    In early models, the behavior of non-reasoning mode (fast thinking) was clear-cut: it output answers directly, with stable token consumption. The non-reasoning mode of the newer generation, however, shows a distinct tendency toward “implicit thinking”:

    • Simple questions: still converge within a few thousand tokens;
    • Medium-complexity questions: output swells to around 10K tokens;
    • Complex questions: output can reach 20K–30K, far exceeding the 15K limit set for it.

    This raises a methodological problem. If you score by the rule “exceeding the limit means truncation, and truncated outputs are not scored,” nearly half the questions in this mode get flagged as invalid samples, which severely undermines the statistical meaning of the final score; but if you score every truncation as 0, the resulting median ends up significantly below the model’s true capability.

    Reasoning mode is affected by the same problem: a small number of questions exceed the 80K limit across repeated runs, are forced to count as 0, and directly drag down the median result.

    Engineering recommendation: Don’t treat the “output limit” as a fixed constant; treat it as an independent variable of the evaluation. Concretely:

    1. Tiered budgets: bin fast thinking at 4K/8K/16K and slow thinking at 16K/32K/64K/128K, each measured independently;
    2. Separate three sample categories: normally completed, truncated and incomplete, and format-noncompliant—report their proportions separately rather than blending them into a single accuracy number;
    3. Report both mean and median: the mean reflects overall cost, while the median reflects the typical experience;
    4. Keep the raw outputs: the reasoning text of truncated samples is key evidence for analyzing “why the model can’t finish.”

    3. The Real Weakness of Long-Chain Reasoning: Not Failing to Answer, but Answering Incompletely

    Comparing the old and new generations reveals a valuable finding: the previous generation did not score low on the same questions, but suffered from missing solutions—it stopped after producing one valid solution without adequately exploring the solution space. The new generation has largely solved this: it actively enumerates possibilities and tries to find all solutions.

    This shows that “a correct answer” and “solution completeness” are two different capabilities, yet most benchmarks only test the former. To make this capability measurable, benchmark questions need to be redesigned:

    • Use set matching rather than exact-match scoring: compute coverage and false-positive rates between the model’s solution set and the reference set;
    • Add partial credit: finding 2 of 3 solutions should earn partial points, not 0;
    • Require the model to explicitly state the solution-space size and self-check, probing its metacognition;
    • Construct questions that are deliberately unsolvable or have a unique solution to test whether the model over-enumerates.

    The cost is equally clear: this “running against the thinking-budget ceiling” behavior pushes token consumption on complex logic problems to more than 3× that of the previous generation. Capability has improved, but cost has deteriorated in step—this is where the new generation most needs to be constrained in engineering terms.

    4. Thinking Efficiency: Same Score, Different Cost Is the New Competitive Frontline

    A worthwhile cross-sectional observation: on several complex reasoning problems, when different models all reach a perfect score, the top closed-source models may consume only 1/2 or even 1/6 of the tokens that the new-generation open-source models do. Once capability has caught up, the gap shows up in thinking efficiency.

    Common causes of poor thinking efficiency:

    • No early-exit mechanism: the model struggles to judge “I’m already sure” and tends to re-verify repeatedly;
    • Inefficient verification: re-deriving everything instead of spot-checking, turning linear work into quadratic work;
    • Redundant chain of thought: repeating premises and re-reading the question, with low effective information density;
    • No confidence calibration: unable to output “how confident I am,” and therefore unable to adaptively allocate budget.

    Corresponding optimization directions:

    • Verifiable reward training: use outcome-level rewards for tasks with reference answers to suppress ineffective self-restatement;
    • Process supervision: score intermediate steps so the model learns to stop at the right place;
    • Reasoning-trajectory distillation: compress the student model’s average thinking length using short-trajectory teacher samples;
    • Parallel sampling + verifier selection: generate multiple candidates via pass@k and pick the best with a lightweight verifier or majority voting, replacing long serial thinking with parallel short thinking;
    • Chain-of-thought compression and summarization: produce structured summaries of prior reasoning to reduce context bloat in long tasks.

    To evaluate whether such optimizations work, don’t look at a single-point score; instead, plot an accuracy–budget curve:

    1. # Illustration: use thinking budget as the independent variable and observe accuracy convergence
    2. budgets = [2_000, 4_000, 8_000, 16_000, 32_000, 64_000]
    3. for b in budgets:
    4. recs = [run_case(model, p, mode="slow", max_output_tokens=b) for p in dataset]
    5. acc = mean(r.correct and not r.truncated for r in recs)
    6. tokens = mean(r.output_tokens for r in recs)
    7. print(f"budget={b:>6} acc={acc:.3f} avg_tokens={tokens:>8.0f}")

    The earlier the curve rises and enters a plateau, the higher the model’s thinking efficiency; if it takes nearly the ceiling to reach the peak, the model lacks early-exit capability.

    5. Coding Ability: One-Shot Performance and Real Engineering Capability Are Two Different Things

    The newer generation’s progress in coding has a clear phase characteristic: the previous generation improved markedly on one-shot algorithm problems, but declined in real Agent environments and on real projects, with the main weakness being a lack of effective debugging—when generation fails, it cannot locate the cause and simply rewrites.

    Judging from the capability shown in publicly available preview builds, mainstream front-end and back-end programming has entered a “usable” range, and on specific tasks it even surpasses the previous generation of mainstream AI coding tools. But the engineering meaning of “usable” needs to be strictly defined; acceptance is recommended along these dimensions:

    Acceptance Dimension Specific Checks
    First-pass rate Proportion of generated code that directly compiles / passes type checks
    Test pass rate Unit tests passed / total cases, not just “looks right”
    Fix convergence Success rate over rounds 2–5 given a failure log
    Regression safety Whether a fix introduces new failing cases
    Change scope Whether changes stay within the necessary files, avoiding large-scale refactoring
    Reproducibility Variance in results across multiple runs of the same prompt

    The key to making coding ability truly measurable is providing the Agent with an observable feedback loop: unified log collection, structured error classification, and traceable diffs. Running this evaluation pipeline in the cloud usually means using a container platform to host stateless evaluation tasks, object storage to keep raw outputs and trajectories, a message queue for task distribution, and a logging service with monitoring and alerting to track truncation rates and cost anomalies—general capabilities offered by platforms like Baidu AI Cloud are well suited to building a sustainable regression-evaluation system, rather than running ad-hoc scripts every time you need to select a model.

    A minimal viable evaluation skeleton:

    1. BUDGET = {"fast": 15_000, "slow": 80_000}
    2. def run_case(model, case, mode):
    3. limit = BUDGET[mode]
    4. resp = model.chat(case.prompt, mode=mode, max_output_tokens=limit)
    5. return Record(
    6. mode=mode,
    7. truncated=(resp.finish_reason == "length"), # Count truncated samples separately
    8. out_tokens=resp.usage.completion_tokens,
    9. correct=grade(case, resp.text),
    10. )
    11. def report(records):
    12. for mode in ("fast", "slow"):
    13. rs = [r for r in records if r.mode == mode]
    14. if not rs:
    15. continue
    16. trunc = mean(r.truncated for r in rs)
    17. # Report both "truncated counts as 0" and "excluding truncated" to avoid misleading single-metric results
    18. strict = mean(r.correct for r in rs)
    19. relaxed = mean(r.correct for r in rs if not r.truncated)
    20. avg_t = mean(r.out_tokens for r in rs)
    21. print(f"[{mode}] truncation_rate={trunc:.1%} strict_acc={strict:.3f} "
    22. f"valid_sample_acc={relaxed:.3f} avg_output={avg_t:.0f} "
    23. f"efficiency={relaxed / max(avg_t, 1) * 1000:.2f}/K")

    6. Long Text and File Understanding: Evaluate “Long-Range,” Not Just “Long”

    The newer generation supports long-context input and offers file understanding, content writing, and data-analysis capabilities, with fairly diverse integration options. But long-context ability shouldn’t be measured with a “needle-in-a-haystack” retrieval test; real business use cases require:

    • Cross-paragraph multi-hop reasoning: conclusions must synthesize the beginning, end, and several middle paragraphs of a document;
    • Long-range consistency: in writing tasks of 20,000+ characters, whether character settings, terminology, and formatting stay consistent throughout;
    • Structured extraction: extracting tables from long documents and doing numeric calculations, checking for “number hallucination”;
    • Distraction robustness: inserting similar but contradictory paragraphs into the document and observing whether the model is misled.

    Engineering-wise, note that long context pushes input cost into the dominant position, so context governance is often more cost-effective than switching models: reranking after retrieval, pruning irrelevant paragraphs, summarizing and compressing conversation history, and splitting long documents into cacheable stable prefixes to reuse the cache.

    7. Cost Structure and Integration Strategy

    Stronger reasoning capability usually brings a price increase, but as long as chain-of-thought length stays controllable, total cost remains predictable; the real risk comes from runaway long thinking. Recommended cost-control strategies:

    1. Mode routing: split traffic by task complexity—route simple tasks to fast thinking with a low limit, and escalate to slow thinking only for complex tasks;
    2. Budget guardrails: set an output limit and a cost cap per call; on exceeding them, downgrade or hand off to a human;
    3. Tiered fallback: automatically retry with fast thinking after a slow-thinking timeout, to preserve availability;
    4. Cache reuse: route stable system prompts and long-document prefixes through the cache to reduce repeated billing;
    5. Observability: break down token consumption by business line, scenario, and model mode on a dashboard, with anomaly alerts;
    6. Abstraction layer: keep a layer between business code and the model interface to avoid being locked into a single integration method, making later cross-comparison and switching easy.

    8. Deployment Checklist

    • Have you included output tokens and truncation rate as selection metrics, rather than accuracy alone;
    • Do the benchmark questions include multi-solution problems, and does scoring support partial credit;
    • Have you plotted an accuracy–budget curve to gauge the model’s thinking efficiency;
    • Is coding evaluation run on real projects and in real Agent environments, checking fix convergence and regression safety;
    • Have you set budget guardrails and fallback paths for different modes;
    • Have you built a reusable, automated evaluation pipeline and cost dashboard;
    • Have you abstracted the integration layer to preserve the ability to switch and compare models.

    Overall assessment: the new generation of open-source models has made substantive progress in long-chain reasoning and real-world programming, with weaknesses concentrated in thinking efficiency and uncontrolled output length. For developers, the right way to bring it into production isn’t a simple swap, but completing three things—evaluation methods, budget governance, and observability. The capability ceiling is determined by the model; the cost floor is determined by engineering.