From Single Leaderboards to Three-Dimensional Evaluation

In real-world LLM adoption, model selection is one of the first decisions teams face—and one of the easiest to get wrong. The common early-stage approach: open a public leaderboard, sort by total score, pick the top-ranked model, and integrate it. The problem is that leaderboard scores only answer “can it get things right?” They say nothing about “at what cost?” or “how distinctive is its thinking?”—two questions that matter just as much in real business settings.

Recently, two noteworthy evaluation signals have emerged in the industry. The first concerns efficiency: in third-party intelligence benchmarks, some next-generation models match the scores of the previous generation’s flagships while consuming fewer inference tokens, which means the “capability density” per unit of compute is improving. The second concerns diversity: one evaluation organization ran a statistical analysis on more than 30,000 pairs of real-world head-to-head model outputs and found that the average viewpoint overlap across different LLMs is about 43.1%. Next-generation models are converging overall, adjacent versions within the same model family are especially similar, and models trained by teams in different countries show no clear “boundary of thinking” either.

Putting these two threads together yields a more complete selection framework: capability, efficiency, and diversity—none of the three can be skipped.

Inference Efficiency: What Fewer Tokens for the Same Score Really Means

What Makes Up Inference Tokens

Before discussing inference tokens, it helps to define their boundaries. A complete model call typically consumes tokens in three places: the input prompt, the intermediate reasoning process (chains of thought, intermediate tool-call results, repeated attempts during self-correction), and the final output. Since long chains of thought became the mainstream paradigm, the intermediate process usually accounts for the majority of total consumption—sometimes several times the length of the final answer alone.

The Engineering Value of Fewer Tokens

• Cost: Inference services generally bill by token, and intermediate tokens cost just as much as output tokens. If two equally capable models differ so that one spends twice the inference tokens on average, its effective unit cost is twice as high.
• Latency: Generation time scales roughly linearly with token count. Longer reasoning chains mean longer user waits, harder-to-control tail latency, and an especially noticeable impact on streaming interactions.
• Throughput and concurrency: Intermediate tokens occupy cache and scheduling resources in the inference framework; the longer the chain, the fewer concurrent requests a single GPU can support.
• Stability: Longer generation chains mean more chances to drift off track—redundant reasoning can sometimes introduce new error branches rather than fix existing ones.

How to Measure: Score-per-Token Curves

Teams are advised to build a lightweight internal evaluation: fix the question set and sampling parameters, record both scores and average inference token counts, plot a “score per token” scatter chart, and identify the Pareto frontier. Sketch code:

# Evaluation framework sketch: comparing "score / inference-token" efficiency on the same dataset
import statistics
def evaluate(models, dataset, max_tokens=8192):
    report = {}
    for model in models:
        scores, tokens = [], []
        for case in dataset:
            resp = model.generate(
                prompt=case.prompt,
                temperature=0,            # Fixed sampling for comparability
                max_tokens=max_tokens,
            )
            scores.append(case.score(resp.text))
            tokens.append(resp.usage.total_tokens)  # Includes intermediate reasoning tokens
        report[model.name] = {
            "score": statistics.mean(scores),
            "avg_tokens": statistics.mean(tokens),
            "efficiency": statistics.mean(scores) / statistics.mean(tokens),
        }
    return report

Note that the optimal efficiency point differs by task type: math and coding tasks justify trading long reasoning for higher correctness, while for classification and extraction tasks, long chains are almost pure waste. Efficiency evaluation must be broken down by task type.

Opinion Convergence: What the 43.1% Overlap Rate Signals

What the Phenomenon Itself Says

Viewpoint overlap measures how much of the set of opinions two models give on the same open-ended question is semantically equivalent. An average of 43.1% means that for two randomly selected models, more than half of their viewpoints duplicate each other. The trend is even more notable: newer models are more similar than older ones, and adjacent versions within the same family are the most alike—which means the incremental new thinking you get from “upgrading to the latest version” may be far smaller than what you get from “switching to a different technical lineage.”

Four Causes of Convergence

• Heavily overlapping training corpora: Mainstream models all rely on public internet text; when information sources converge, the “worldview” the models learn converges too.
• Converging alignment paradigms: Most teams follow similar human-preference alignment pipelines, and the distributions of preference datasets themselves differ little.
• Leaderboard-driven optimization: When every team optimizes against the same public benchmarks, models converge on the “problem-solving patterns” those benchmarks represent.
• Distillation and synthetic data amplification: Outputs from strong models are widely reused as training material, spreading a handful of “dominant lines of thinking” across the entire industry.

What Convergence Means for Engineering Teams

Convergence has two practical consequences. First, simply swapping one model for another in the same tier yields diminishing marginal returns. Second, the “diversity dividend” of multi-model ensembles is shrinking—if two models overlap on 80% of their viewpoints, there is little value in combining them. This leads directly into the strategy questions of the next section.

“More Expensive Doesn’t Mean More New Ideas”: Combination Strategies for Brainstorming

Capability and Diversity Are Independent Dimensions

A counterintuitive but data-supported conclusion: the strongest and most expensive models tend to be the most similar to one another—because the top models all sit near the technical frontier, with the most similar training paradigms and data. Yet the goal of brainstorming-style tasks is not “single-point optimality” but “viewpoint coverage.” In this setting, stacking the two strongest but highly similar models may work worse than pairing “one model that meets the capability bar + one with a clearly different line of thinking.”

Three Actionable Principles

• Set a capability threshold first, then pick for diversity above it: Use basic evaluations to filter out a pool of “good enough” models, then choose low-overlap combinations within that pool—instead of simply taking the top two by score.
• Diversity comes from technical lineage, not price tier: Models with different data emphases, scale tiers, or architectural lineages usually differ more in viewpoint; price gaps by themselves are not a reliable proxy for diversity.
• Verify differences with data, not intuition: Run small-scale overlap sampling on candidate combinations and quantify the results before committing.

Quantifying Idea Coverage with Semantic Deduplication

# Opinion diversity evaluation sketch: generate -> split viewpoints -> embed -> cluster and deduplicate
def diversity_score(answers, encoder, threshold=0.85):
    points = []
    for text in answers:
        points.extend(split_viewpoints(text))   # Split by sentence or key point
    vecs = encoder.encode(points)
    kept = []
    for v in vecs:
        if all(cos(v, k) < threshold for k in kept):
            kept.append(v)                      # Keep only semantically "new" viewpoints
    return len(kept) / max(len(vecs), 1)        # The higher the share of new viewpoints, the more complementary the combination

Implementation Essentials: An Engineering Skeleton for Multi-Model Orchestration

• Access layer: Integrate multiple candidate models uniformly through cloud-hosted LLM services to abstract away interface differences. Whether you use a general-purpose LLM service from a cloud platform such as Baidu AI Cloud or operate your own inference cluster, the key is turning “switching models” into a configuration change rather than a code change.
• Routing layer: Dispatch by task profile—structured tasks go to high-efficiency models; open-ended exploratory tasks go to high-diversity combinations.
• Aggregation layer: Split multi-source results into individual viewpoints, deduplicate via embeddings, merge by clustering, then rank and output by novelty and quality.
• Observability layer: Continuously log each model’s token consumption, latency distribution, and “viewpoint contribution” (the share of deduplicated viewpoints it uniquely provides). These metrics can be collected and retained using standard cloud capabilities such as logging, monitoring and alerting, and object storage.
• Cost layer: Set token budgets and fallback/degradation policies to keep long chains of thought from running out of control on low-value tasks.

Conclusion

LLM selection is evolving from “who scores highest” to “who is most cost-effective and who is most distinctive.” Inference token efficiency determines the real cost and experience at a given capability level, while opinion diversity determines the ceiling on the value of multi-model collaboration. Put capability, efficiency, and diversity into a single evaluation framework, and let data from your own evaluations speak—only then can you make technical decisions that stand up to business scrutiny across fast-moving model release cycles.