Rethinking LLM Evaluation: Reasoning Efficiency and Output Diversity Beyond Benchmark Scores

The Hidden Variable Behind Benchmark Scores: Inference Cost

When a new model is released, the conversation usually revolves around “how much stronger is it than the previous generation” and “how many points did the benchmarks go up.” Such discussions are intuitive, but they tend to ignore a dimension that is just as important — and often more critical in engineering practice: the inference resources consumed to achieve the same results.

A noteworthy phenomenon has recently emerged in the industry: a new-generation model has posted scores close to those of much larger models on several general intelligence benchmarks, while consuming significantly fewer reasoning tokens. The significance of this goes far beyond “saving a bit of money” — it indicates that the model’s inference efficiency and the compactness of its internal decision paths have changed. On identical hardware, fewer reasoning tokens translate directly into lower per-request latency, higher concurrent throughput, and more predictable inference costs.

For developers, this suggests building a new evaluation coordinate system: the horizontal axis is the capability score, and the vertical axis is inference cost (token consumption, latency, and cost per request). The ideal model moves toward the upper right, rather than merely sliding right along the horizontal axis.

Why Reasoning Tokens Serve as an Efficiency Metric

The number of reasoning tokens consumed essentially reflects how verbose a model’s “thinking” is before it produces the final answer. In tasks that require multi-step reasoning, models often generate intermediate reasoning steps, and these steps are counted as token consumption. When token usage drops while scores stay flat, several explanations are possible:

1. More compact reasoning paths: The model reaches correct answers with shorter intermediate steps, reducing ineffective trial-and-error and redundant argumentation.
2. Improved internal representation efficiency: Optimization of the reasoning process during training allows the model to perform equivalent computation with fewer explicit steps.
3. More precise task adaptation: The model judges question types more accurately, avoiding “overthinking” simple problems.

It is worth emphasizing that fewer reasoning tokens is not inherently better. For complex mathematical proofs or long-chain logic tasks, an appropriate number of intermediate steps is a necessary cost to guarantee correctness. What really deserves attention is the ratio between score and token consumption on a fixed benchmark — that is, the level of intelligence obtained per unit of inference cost. This ratio can serve as a reference metric for the direction of model iteration: it optimizes capability and efficiency at the same time.

A simplified estimation sketch (pseudocode, for illustration only):

def efficiency_score(accuracy, avg_reasoning_tokens):
    # accuracy: accuracy rate on the evaluation set
    # avg_reasoning_tokens: average reasoning tokens consumed per question
    if avg_reasoning_tokens <= 0:
        raise ValueError("token consumption must be positive")
    return accuracy / avg_reasoning_tokens

In real-world evaluation, variables such as prompt length, sampling parameters, and the difficulty distribution of the evaluation set must also be controlled before this ratio becomes meaningfully comparable across models.

Model Behavior Convergence: An Underestimated Blind Spot

Another phenomenon developers should watch comes from the analysis of large-scale real-world battle data. One study of more than 30,000 real head-to-head model matchups found that the average overlap in viewpoints between different models is only about 43.1% — in other words, more than half of the outputs differ in stance or expression. At the same time, however, new-generation models are showing increasingly obvious convergence: their response styles, reasoning patterns, and even error modes are beginning to align.

This carries two implications for evaluation practice:

• Similar scores do not mean complementary capabilities. Two models with close scores on the same benchmark may simply “fail in the same way” under the same data distribution and the same alignment strategy. In engineering approaches such as ensembling, routing, and multi-model voting, this convergence weakens the gains that combination would otherwise bring.
• Convergence amplifies systemic risk. If many models share similar training data sources and alignment preferences, they may fail simultaneously on out-of-distribution inputs, undermining the robustness of the entire system.

Therefore, when selecting models, look beyond scores and pay attention to output diversity: given the same batch of prompts, how dispersed are the answer distributions across candidate models? A simple overlap statistic can quantify this:

def overlap_ratio(answers_a, answers_b):
    # Roughly measure the similarity between two models' outputs via set overlap
    set_a, set_b = set(answers_a), set(answers_b)
    if not set_a or not set_b:
        return 0.0
    return len(set_a & set_b) / len(set_a | set_b)

When the overlap ratio is notably high, the two models behave very similarly, and the marginal benefit of using them in combination is limited.

Practical Recommendations for Model Selection and Evaluation

Based on the observations above, a more engineering-oriented evaluation framework can be distilled:

1. Two-dimensional scoring: Record both the capability score and the average reasoning token consumption, and compute an intelligence-per-cost ratio instead of comparing absolute scores alone.
2. Diversity sampling: On your own business prompt set, measure the output overlap between candidate models to avoid choosing model combinations with highly homogeneous behavior.
3. Task-type-specific evaluation: For simple tasks, focus on latency and cost; for complex reasoning tasks, focus on the balance between accuracy and the number of reasoning steps. Avoid covering all scenarios with a single metric.
4. Track iteration trends, not single data points: Efficiency improvements in new-generation models tend to be incremental. Continuously tracking the joint change of scores and costs is more valuable for decision-making than chasing the benchmark numbers of a single release.
5. Keep manual spot checks: Automated metrics cannot fully capture the blind spots created by convergence. Regular manual review of out-of-distribution samples helps detect the risk of collective model failure.

Conclusion

The progress of new-generation models is shifting from “simply stacking capability” to “balancing capability and efficiency.” Falling reasoning token consumption and converging model behavior are two sides of the same coin: the former shows that the cost per unit of intelligence is being optimized, while the latter warns that diversity is being lost. For developers, building a multi-dimensional evaluation system that includes efficiency, cost, and diversity supports long-term technical decisions far better than staring at leaderboard scores alone. The same evaluation applies when building inference services on general-purpose cloud platforms such as Baidu AI Cloud — only by including token cost, latency, and output diversity in monitoring metrics can model capability be converted into sustainable business value.