Evaluating LLM Reasoning: Long Chain of Thought and Token Cost Analysis

1. Background: Shifting Focus from “Correct Answers” to “Task Completion”

Over the past two years, the evaluation framework for Large Language Models (LLMs) has undergone a significant shift. Early evaluations focused primarily on “one-shot” tasks like knowledge Q&A and reading comprehension. Today, the industry’s focus has pivoted to long-horizon tasks, multi-step reasoning, and agentic workflows. The rationale is straightforward: whether a model can deliver results in real-world business scenarios depends on its ability to decompose a complex goal into a series of executable reasoning and operational steps, while continuously self-correcting along the way.

Against this backdrop, simply looking at a “total score” is no longer enough. A rigorous cross-evaluation of reasoning capabilities must answer at least three questions simultaneously: How significant is the behavioral difference between reasoning and non-reasoning modes? To what extent does the Chain of Thought (CoT) expand given a specific “thinking budget”? And what is the token cost required to achieve a specific score? Recent iterations in the open-source community provide excellent samples for observing these three questions. Rather than focusing on specific products, this article abstracts these observations into general evaluation methodologies and engineering practices.

2. Reasoning vs. Non-Reasoning Modes: More Than Just a Toggle

2.1 Non-Reasoning Modes are Becoming “Adaptive”

A notable trend is that the non-reasoning mode of newer generation models increasingly behaves like an “adaptive reasoning model with a low thinking budget.” For simple queries, it can quickly generate an answer within a few thousand tokens. However, when faced with complex tasks, the output consumption can climb to the 20K or even 30K token range.

This creates a practical issue for evaluators: if the output limit for non-reasoning mode is capped at 15K tokens, a significant portion of tasks will be truncated due to exceeding the limit. According to standard evaluation rules, truncated outputs that fail to produce a complete answer must be scored as zero or disregarded. This means that without adjusting the upper limit, a complete test of the non-reasoning mode cannot even be conducted.

2.2 CoT Expansion in Reasoning Mode

The reasoning mode presents a problem on the opposite end of the spectrum. In multiple test runs of a small number of high-difficulty tasks, the CoT output consistently exceeded the preset 80K upper limit, again resulting in a zero score. This directly pulls down the median score of the reasoning mode—note that it affects the median, not the average. This indicates a long-tail phenomenon caused by a few extreme samples rather than a universal capability deficit.

The takeaway for evaluation interpreters is that when encountering a lower-than-expected score, one must first distinguish between two entirely different causes: “insufficient capability” and “budget truncation.” The former is an issue with the model itself, while the latter is a mismatch between the evaluation configuration and the generational characteristics of the model. As CoT length continues to expand with model iterations, if evaluation configurations are not adjusted accordingly, the conclusions will suffer from systematic bias.

3. Making Scores Credible: Three Keys to Evaluation Design

1. Explicitly Declare Limits and Scoring Rules. The output limit is essentially a quantitative constraint on the “thinking budget.” The rule design must clarify: how are truncated outputs scored (zero, excluded, or partial credit)? Are retries allowed? How are retry counts recorded? Scores under different criteria are not directly comparable, and this must be explicitly stated in cross-evaluations.

2. Multiple Test Runs and Statistical Metrics. A single sampling is highly susceptible to randomness, especially for long-chain reasoning tasks. A sound approach is to conduct multiple test runs per task and report the median and distribution rather than just the mean. A low median with a normal average often indicates the presence of a few truncated samples. Conversely, a low average with a normal median suggests a widespread capability deficit. The statistical metric itself serves as diagnostic information.

3. Control Groups and Mode Labeling. A rigorous cross-evaluation should separately label and compare results from the reasoning mode (slow thinking) and the non-reasoning mode (fast thinking). Mixing them is equivalent to ranking “open-book exam” scores alongside “closed-book exam” scores. Additionally, the control group should consist of a comparable subset of models, explicitly stating that the table is not a comprehensive ranking to prevent misinterpretation by readers.

4. Core Observations on Long-Chain Reasoning: Sufficiency of Solution Space Exploration

4.1 From “Finding a Solution” to “Finding All Solutions”

The gap between previous-generation and newer-generation models is highly representative on logic puzzles with multiple solutions. Previous-generation models scored reasonably well but lacked sufficient exploration of the solution space, often halting the search after finding a single valid solution. Newer-generation models, however, can systematically enumerate various possibilities, striving to find all solutions. This difference is virtually invisible on single-solution tasks and only surfaces on multi-solution problems with coupled constraints. This explains why evaluation sets must be specifically designed with multi-solution tasks: they measure the depth of the model’s understanding of the problem structure, rather than luck-based searching.

4.2 The Cost: Operating at the Budget Ceiling

Comprehensive exploration is not free. The token consumption of newer-generation models on complex logic puzzles can reach more than three times that of previous generations. Although overall consumption remains on par with previous generations, the distribution heavily skews toward high-difficulty tasks—the budget saved on easy questions is consumed by the hard ones. This “budget reallocation” is a rational evolutionary step, but it raises the worst-case invocation costs, a factor that cost-sensitive applications must consider.

5. Thinking Efficiency: The Next Competitive Frontier

Broadening the view to leading international models reveals that, given the same tasks and perfect scores, the token consumption of top-tier models is only 1/2 to 1/6 of that of the aforementioned newer-generation models. In other words, the scores are similar, but the “unit price” gap is staggering.

Here, we can define an intuitive metric: Thinking Efficiency—the reasoning quality derived per unit of token consumption. Potential areas for improvement include:

• Adaptive Budget Allocation: Dynamically adjusting the CoT length based on task difficulty, allowing fast convergence on simple problems.
• Chain of Thought Compression: Trimming redundant self-confirmations and repetitive backtracking while preserving the core reasoning skeleton.
• Search Pruning and Early Stopping: Introducing confidence judgments during solution space exploration to avoid meaningless exhaustive searches.

For model providers, thinking efficiency is the next competitive dimension following raw scores. For application developers, it directly dictates the slope of the inference cost curve.

6. Coding and Agent Scenarios: The Gap Between One-Shot Scores and Real-World Projects

Recent model iterations universally exhibit a pattern: One-shot coding performance (generating runnable code from a single prompt) improves rapidly, but deploying models into real-world agents and projects often yields disappointing results.

The core of this gap lies in the structural differences between real-world projects and exam questions. Real projects carry heavy historical technical debt, implicit dependencies, and long-tail errors. Models must not only “write correctly” but also “debug correctly.” This requires robust debugging capabilities—reading error stacks, designing minimal reproducible examples, and iteratively correcting based on execution feedback. Evaluations that only test One-shot capabilities will systematically overestimate a model’s usability in engineering scenarios.

A more rational design introduces a traceable execution feedback loop: generate code → run → feed errors back to the model → observe its repair capabilities. The performance within this loop is much closer to the real development experience than any One-shot score.

7. Practical Recommendations for Developers

1. Mode and Budget Matching Strategy. Use the non-reasoning/fast-thinking mode with a lower output limit for simple tasks (format conversion, summarization, single-turn Q&A). Use the reasoning mode with an ample budget for complex, multi-step tasks (multi-constraint planning, code debugging, data modeling). For mixed-traffic scenarios, perform difficulty classification or routing first, then allocate the appropriate mode and budget. An illustrative configuration:

difficulty_router(task) → fast / reasoning
fast:      max_output_tokens = 8K, downgrade to step-by-step prompting if limit exceeded
reasoning: max_output_tokens = 64K, fallback to human intervention if budget exhausted without convergence

2. Cost Observation and Degradation. Monitor the CoT length distribution for each invocation on the application side, and set tiered budgets with degradation strategies for the reasoning mode. CoT length is a more sensitive leading indicator of cost than standard input/output tokens.

3. Build Custom Evaluation Sets. Public cross-evaluations provide horizontal references, but business decisions ultimately rely on private evaluation sets. It is recommended to extract representative tasks from real business logs, design scoring rules and multi-run tests based on the methodology above, and perform regular regression testing as model versions update. During model selection and comparison, developers can also leverage platforms like Baidu AI Cloud, which offer unified multi-model access and monitoring capabilities, to quickly conduct comparative validation and cost observation, reducing the infrastructure overhead required to build these tools from scratch.

8. Conclusion

Two clear trends emerge from this iteration: First, the ceiling for reasoning capabilities is rising, with the ability to explore the solution space during long-chain reasoning becoming the new watershed. Second, thinking efficiency is beginning to replace raw scores as the key metric for measuring a model’s engineering value. For evaluators, understanding the “budget and truncation behind the score” is more important than memorizing the score itself. For developers, understanding mode differences, budget management, and cost observation is the only way to truly translate model capabilities into stable business delivery.