Evaluating Long Chain-of-Thought Models: Balancing Thinking Budget and Efficiency

    Long chain-of-thought (CoT) models expose a tension that capability checklists hide: how much does a new generation actually improve on real tasks, and at what cost in tokens and latency? This article explains how to build a repeatable evaluation baseline for reasoning and non-reasoning modes. It covers the divergent consumption profiles of fast and slow thinking, why truncation rate and efficiency must sit alongside accuracy, how to measure long-chain logic and agentic coding, and how to govern cost through mode routing and thinking budgets. The goal is to turn every model upgrade into a quantifiable gain rather than passive migration cost.

    1. Beyond the Launch: You Need to Measure

    Over the past two years, the main capability lines of large models have converged on two fronts: long context on the input side, and long chain of thought on the output side. Each release tends to publish a capability list: longer input windows, stronger file and document understanding, more complete content generation and data analysis, richer integration options. These are indeed what ordinary and enterprise users perceive most directly, but they cannot answer a more critical question: on the same set of real tasks, how much better is this generation than the last one, and at what cost.

    Observing the release cadence of comparable models leads to a plain conclusion: leadership is not stable. The top spot on a niche leaderboard can change hands within months, and different models take turns leading on different capability dimensions. Building model selection on the impression of a single release is therefore unreliable. The right approach is to establish an evaluation baseline for your own business and continuously track “output per unit cost.”

    An evaluation should answer three questions: whether capability has genuinely improved, on which task types the improvement occurs, and how many extra tokens and how much latency the improvement costs. This article is organized around these three questions.

    2. Engineering Profile of Two Modes: Fast Thinking and Slow Thinking

    Mainstream models commonly offer two operating modes:

    • Non-reasoning mode (fast thinking): fast responses and stable consumption, suited to structured tasks like classification, extraction, rewriting, and formatting.
    • Reasoning mode (slow thinking): explicitly generates a chain of thought, suited to math, multi-step logic, code understanding, and planning tasks.

    A notable trend across this generation is worth flagging: non-reasoning mode behaves increasingly like “an adaptive reasoning model with a small thinking budget.” In other words, even without an explicit chain of thought, the model still performs implicit multi-step reasoning internally. This has two direct consequences:

    1. Simple questions still cost very little, typically completed in a few thousand tokens, so fast thinking’s cost advantage is not eroded;
    2. The output-length tail for complex tasks is extremely long, and in testing, consumption on complex tasks can jump from a few thousand tokens to the 20K–30K range.

    The direct impact on engineering is: you can no longer use averages for capacity planning. The P99 latency and consumption of fast-thinking mode may approach or even exceed the median of reasoning mode. When setting rate limits, timeouts, and cost estimates, you must look at the quantiles of the length distribution, not the mean.

    Mode Typical consumption range (illustrative) Latency profile Suitable tasks
    Non-reasoning 0.5K–30K, pronounced long tail Low median, high P99 Structured output, extraction, rewriting, lightweight Q&A
    Reasoning 3K–80K, rises steeply with difficulty Overall high and stable Multi-step logic, math, code analysis and refactoring

    3. Designing an Interpretable Evaluation

    The credibility of an evaluation result depends on the rule design, not on the model itself. Four areas are the most commonly overlooked.

    1) Output limits and truncation scoring rules. A common failure mode in practice is setting a 15K output limit for fast-thinking mode, only to have nearly half of the complex questions hit the ceiling and get truncated. If truncated samples are scored as no-credit per the rule, this mode’s score is systematically depressed; if you switch to lenient scoring (truncated samples get partial credit), the ranking can flip. Therefore you must report on two tracks: the strict definition gives an upper-bound reference, the lenient definition reflects real usability, and presenting both numbers side by side lets readers judge the boundaries of the conclusion.

    2) Repeated sampling and statistical definitions. A single run of a long chain-of-thought model has high variance, so taking the median across multiple runs is more meaningful than taking the best. A reverse example is equally common: on a handful of hard problems, reasoning mode exceeds the 80K limit across repeated runs and can only be scored 0 under the rule, which is one cause of reasoning mode’s low median score. A low median does not mean weak capability; it is very likely a mismatch between budget settings and problem difficulty.

    3) Truncation rate must be a first-class metric. When the truncation rate exceeds 30%, the absolute scores of a result have largely lost comparability and can only be used for relative comparison.

    4) Efficiency metrics must sit alongside accuracy. Looking only at correctness rewards an “think without limit” strategy. It is advisable to also compute score per thousand tokens, average consumption over the solved set, and the accuracy–consumption Pareto frontier.

    A reusable evaluation harness skeleton is shown below:

    1. MAX_OUTPUT_FAST = 15_000 # Fast-thinking output limit (illustrative)
    2. MAX_OUTPUT_SLOW = 80_000 # Slow-thinking output limit (illustrative)
    3. REPEATS = 5 # Number of repeated samples
    4. def run_case(model, prompt, max_tokens, grader, repeats=REPEATS):
    5. outs = [model.chat(prompt, max_tokens=max_tokens) for _ in range(repeats)]
    6. tokens = [len(o.tokens) for o in outs]
    7. truncated = [t >= max_tokens for t in tokens]
    8. return {
    9. "scores_strict": [0 if tr else grader(o) for o, tr in zip(outs, truncated)],
    10. "scores_loose": [grader(o) for o in outs],
    11. "tokens": tokens,
    12. "trunc_rate": mean(truncated),
    13. }
    14. def summarize(rows):
    15. """Strict track gives the upper bound, lenient track gives usability, efficiency on its own."""
    16. return {
    17. "strict_median": median([median(r["scores_strict"]) for r in rows]),
    18. "loose_median": median([median(r["scores_loose"]) for r in rows]),
    19. "p50_tokens": median([median(r["tokens"]) for r in rows]),
    20. "trunc_rate": mean([r["trunc_rate"] for r in rows]),
    21. # Score per thousand tokens, for cross-model cost-effectiveness comparison
    22. "efficiency": median([median(r["scores_loose"]) for r in rows])
    23. / max(median([median(r["tokens"]) for r in rows]), 1) * 1000,
    24. }

    4. Long-Chain Logic: From “Right Answer” to “All Answers Right”

    The focus of evaluating long-chain logic is not whether the final answer is correct, but whether the solution space has been explored sufficiently. There are two typical failure modes: missing solutions, where the model stops at the first feasible solution; and conflicting solutions, where the model outputs mutually contradictory answer sets without adjudicating between them. The previous generation did not score low on such problems, but its errors were concentrated on “insufficient exploration.” The new generation largely alleviates this: it can enumerate possibilities and try to find the complete set of solutions.

    The cost is equally clear. To find all the solutions, the model runs continuously against the thinking-budget ceiling: against a background of overall token consumption on par with the previous generation, the per-question consumption for complex logic problems can actually rise to more than three times that of the previous generation. Meanwhile, top international models achieving the same full score, on the same problems and standard, often consume only half or even one-sixth as much.

    This means the competitive focus has shifted from “can it get it right” to “thinking efficiency.” An actionable way to quantify this is to fix the problem set, sweep across thinking-budget tiers, and plot an accuracy–consumption curve:

    1. Accuracy
    2. | o---o Top international model (full score at low consumption)
    3. | o
    4. | o <- New domestic model (full score only at the ceiling)
    5. | o
    6. +-------------------------------> Average consumption (tokens/question)

    The further a curve sits toward the top-left, the higher its cost-effectiveness. If a model only reaches a stable full score after consuming more than 90% of the budget, it indicates substantial reasoning redundancy, and in production the cost should be contained through prompt constraints, budget caps, and result validation.

    5. Coding and Agent Capabilities: From One-Shot to Real Projects

    Evaluating coding ability must distinguish two scenarios:

    • One-shot code generation: given a function signature and comments, generate code that passes unit tests in a single attempt. Most new models show significant progress on this dimension.
    • Agentic real-project development: cross-file search, modification, test execution, and iterative repair based on errors within a real repository. The difficulty here is not writing code but establishing an effective debugging loop.

    The typical weakness of the previous generation lay precisely in the latter: single-turn generation looks reasonable, but without effective debugging means, the model cannot autonomously converge based on compilation errors, test failures, or log stacks. Some recently released code-specialized preview versions show that agentic coding ability can improve markedly, and routine front-end and back-end development tasks have entered a “usable” range.

    Recommended dimensions for evaluating agentic coding:

    1. First-pass success rate: whether the first-generated patch passes all tests;
    2. Convergence rounds: the number of iterations needed from failure to pass; more than 5 rounds basically means out-of-control cost;
    3. Tool-call rationality: whether it reads enough context before acting, whether it repeats ineffective retrievals;
    4. Regression introduction rate: whether fixing problem A breaks existing tests;
    5. Reproducibility: whether results are stable across multiple samples and can be regression-tested in CI.

    A minimal iteration loop is illustrated below:

    1. for attempt in range(MAX_ATTEMPTS):
    2. patch = agent.propose(task, context=repo.diff_and_relevant_files(task))
    3. result = run_tests(patch) # compile + unit tests + lint
    4. if result.passed:
    5. return patch, attempt + 1
    6. agent.observe(result.failure_log) # Key: feed executable error signals back
    7. raise AgentBudgetExceeded(task, MAX_ATTEMPTS)

    The core design principle here is: error signals must be structured and actionable (failing test names, assertion diffs, exception stacks), rather than feeding back an entire block of natural-language “this looks wrong” to the model.

    6. Cost Governance and Tiered Scheduling of Thinking Budgets

    Evaluation conclusions ultimately come down to cost. A few general practices:

    • Route modes by task complexity to avoid sending all requests through slow thinking:
    1. def pick_mode(task):
    2. if task.est_difficulty >= 3 or task.requires_multi_step_logic:
    3. return "reasoning"
    4. return "non_reasoning"
    • Set output limits and handle truncation explicitly: when the limit is hit, return a recognizable status so the upper layer can decide whether to retry, degrade to a larger budget, or split the problem into subproblems.
    • Splitting the problem takes priority over increasing the budget: breaking a hard problem that needs 80K of chain of thought into several verifiable subtasks is usually more economical than linearly adding tokens.
    • Streaming output plus timeout circuit breaking: in long chain-of-thought scenarios, time-to-first-token is controllable but total duration is long, so timeouts and retry strategies must be in place to avoid request pile-up.
    • Observability first: persist the input/output length, truncation flag, mode, elapsed time, and correctness of each call in a unified archive, so that retrieval, monitoring, and alerting dashboards can continuously reveal the actual distribution of thinking budgets. These general capabilities are available across Baidu AI Cloud’s foundational product categories such as object storage, log service, and monitoring and alerting, and can serve as the hosting foundation for a self-built evaluation and scheduling pipeline.

    7. Conclusion

    For this kind of new model, what truly deserves attention is not a single leaderboard position but three trend lines: reasoning capability has entered the first tier of domestic models and changes hands periodically; the cost of capability gains is shifting from accuracy to thinking efficiency; and coding ability is moving from single-turn generation toward agentic closed-loop debugging, entering a range that can actually be delivered.

    For enterprises and developers, the pragmatic approach is: fix your own problem set and truncation rules, report strict and lenient definitions on two tracks, put token efficiency alongside accuracy on a dashboard, and control cost through mode routing and budget constraints. The model landscape has always seen one era succeed another. Only a self-built, reproducible evaluation baseline can turn each version change into a quantifiable gain rather than a passively-followed migration cost.