Why LLM Benchmark Scores Are Converging: Scale, Post-Training, and the Evolution of Evaluation

1. The Phenomenon: Leading Models Are Entering a “Narrow Band”

Over the past year, the shape of public evaluation leaderboards has changed noticeably. In earlier periods, top models were often separated by sharp, step-function gaps—the distance between first and fifth place could exceed ten percentage points. Today, however, the composite intelligence indices of leading models cluster within a very narrow range, with gaps of only two or three points—sometimes smaller than the statistical noise of the evaluation itself. At the same time, some open-source models using sparse activation architectures have matched models with far larger parameter counts on composite benchmarks, while using significantly less inference compute. On security-focused benchmarks that target real-world codebase vulnerability discovery, open-source models have topped the leaderboard—and because their inference costs are low, the same budget can execute several times more tasks than high-end closed-source models in large-scale concurrent scanning scenarios.

It is important to emphasize that score convergence does not mean “there are no differences between models.” Rather, it signals that the gaps the current evaluation system can resolve are narrowing, and that the competitive dimension is shifting from “single-point intelligence” toward “system efficiency, autonomous execution, and unit cost.” Understanding this shift is more valuable than memorizing any particular leaderboard ranking.

2. Sparse Expert Architecture: Decoupling Total Parameters from Activated Parameters

To understand why “small-activation models catching up to massive models” is technically entirely reasonable, it helps to first clarify the fundamentals of mixture-of-experts (MoE) architectures. In a standard dense Transformer, every token passes through the complete feed-forward network. In a sparse expert architecture, the feed-forward layers are replaced by multiple parallel experts, and a router selects only the Top-K experts for each token:

import torch
import torch.nn as nn
class TopKRouter(nn.Module):
    """Minimal illustration of sparse expert routing: each token activates only Top-K experts"""
    def __init__(self, hidden_dim, num_experts, top_k=2):
        super().__init__()
        self.gate = nn.Linear(hidden_dim, num_experts, bias=False)
        self.top_k = top_k
    def forward(self, x):
        logits = self.gate(x)                       # [batch, seq, num_experts]
        probs = torch.softmax(logits, dim=-1)
        topk_probs, topk_idx = probs.topk(self.top_k, dim=-1)
        topk_probs = topk_probs / topk_probs.sum(-1, keepdim=True)
        return topk_probs, topk_idx                 # Unselected experts do not participate in computation

This design introduces a key decoupling: total parameter count determines the model’s “knowledge capacity ceiling,” while activated parameter count determines “per-inference compute and cost.” The two can scale independently. A model with hundreds of billions of total parameters but only tens of billions activated per pass may have a knowledge scope not inferior to a dense model several times its size, yet at a much lower inference cost. So when evaluations show that “a sparse model has caught up to a model several times its scale,” what is really being compared is not a single dimension: the former achieves comparable performance with less compute, which is itself a victory for architectural efficiency.

Of course, sparse models are not a “free lunch.” The design of load-balancing losses, the communication overhead of expert parallelism, and the prevention of router collapse all make training engineering significantly harder. Being able to solve these engineering problems well is itself a reflection of a team’s competitiveness.

3. Post-Training: Further “Lifting” Composite Capabilities on an Existing Base

One noteworthy public observation is that a certain model raised its composite index by roughly 7 points through systematic post-training alone, with its pre-training base essentially unchanged. Behind this is a shift in the center of gravity of current LLM competition.

Pre-training determines the “ceiling” of capabilities—knowledge coverage, language understanding, reasoning priors. Post-training determines the degree to which those capabilities are “invoked.” A typical training pipeline looks like this: supervised fine-tuning establishes instruction following and format compliance; preference optimization aligns with human feedback; reinforcement learning with verifiable rewards amplifies reasoning on tasks with clear right-or-wrong criteria such as math and code; and agentic training teaches the model to use tools, perform long-horizon planning, and reflect on and correct errors.

Post-training has become the main battlefield for two reasons: first, the marginal returns from pre-training are diminishing and high-quality data bottlenecks are emerging; second, post-training has short iteration cycles and fast payoffs—the same base paired with different data mixtures and reward designs can produce considerable gaps on leaderboards. But its boundaries should also be recognized: post-training is better at “aligning and eliciting” and cannot create from nothing capabilities that were not covered during pre-training. This explains why post-training results from different teams tend to converge near the capability ceiling of their respective bases.

4. Benchmark Saturation: Why Leaderboard Gaps Keep Shrinking

When leading models’ scores crowd into a narrow band, one question must be confronted directly: is it that models have no differences, or that benchmarks cannot detect differences? Three mechanisms deserve analysis:

1. Data contamination. Questions from mainstream benchmarks enter training corpora through public discussion. Models may have “memorized the answers,” inflating scores broadly and reducing discriminative power.
2. Measurement ceiling. When most models’ capabilities exceed the difficulty ceiling a benchmark was designed for, scores naturally cluster at the high end—just as when everyone scores above 95 on an exam, it indicates the test was easy, not that all test-takers are identical.
3. Smoothing effect of composite indices. A composite leaderboard is typically a weighted average across many sub-items. Significant differences on individual items cancel each other out after weighting, producing the illusion of “clustering together.”

Therefore, rank changes within the narrow band should not be over-interpreted. A more informative approach is to observe model performance on newer, harder evaluations that are closer to real-world workflows—which is the subject of the next section.

5. Autonomous Task Evaluation: A Better Touchstone for Revealing Real Gaps

A recent class of “autonomous research” evaluations provides a good observation window. The typical setup: let a model fully autonomously complete a research task in an offline environment—for example, under a fixed compute budget, improve the training pipeline of a small language model and reduce validation loss to a specified level. The model must autonomously complete the full loop of reviewing material, proposing hypotheses, modifying code, launching experiments, analyzing results, and iterating.

The most interesting finding from these evaluations is that nearly all frontier models can find the “right direction”—they propose similar or even identical optimization ideas. In other words, the “local intelligence” gap between models is not large. The real differences appear at the execution level:

• Experiment design density: Can the model decompose one idea into multiple controlled experiments and advance them in parallel?
• Long-horizon persistence: Can it maintain its objective without drifting across tasks spanning dozens or even hundreds of steps?
• Failure recovery strategy: After an experiment fails, does it mechanically retry, or adjust its hypothesis based on the failure information?
• Resource budget management: Can it balance exploration and exploitation within limited compute?

A simplified evaluation framework can be expressed as:

def autonomous_eval(model, task, max_steps=50):
    """Illustration of autonomous task evaluation: assessing path quality, efficiency, and failure recovery"""
    state, trajectory = task.reset(), []
    for _ in range(max_steps):
        action = model.propose(state)      # Propose a hypothesis or experiment plan
        result = task.execute(action)      # Execute and return observations and metrics
        state = task.observe(result)
        trajectory.append((action, result))
        if task.target_reached():
            break
    return task.score(trajectory)          # Holistic assessment of idea quality and execution efficiency

These evaluations are closer to real R&D workflows, so they better amplify differences between models. They also provide a footnote to the “score convergence” phenomenon: on single-step question-answering benchmarks everyone converges, but on tasks requiring sustained autonomous execution, the gaps remain clearly visible.

6. The Cost Dimension: From “Score-Only” to “Intelligence per Unit Cost”

For engineering deployment, leaderboard scores are only half the story; the other half is cost. Sparse activation models have a structural advantage here: fewer activated parameters means less compute per token, which in turn lowers per-task inference cost. An example abstracted from recent public information: a model achieved a score close to the frontier at a per-task cost below $1, making it highly competitive on the “score/cost” curve.

A more pragmatic selection metric, therefore, is “task success rate per unit cost”:

def cost_efficiency(success_rate: float, cost_per_task: float) -> float:
    """Intelligence per unit cost: success rate / per-task cost, for cross-model comparison"""
    if cost_per_task <= 0:
        raise ValueError("cost_per_task must be positive")
    return success_rate / cost_per_task

Under budget constraints, a model with a slightly lower score but dramatically lower cost may have significantly higher engineering value than a flagship model with a slightly higher score. Selection decisions should return to the business context: high-value, low-frequency, complex tasks can tolerate high cost; large-scale, high-frequency, patterned tasks should prioritize cost-effectiveness.

7. The Multimodal Trade-Off: The Opportunity and Cost of Single-Modality Focus

Another phenomenon worth discussing: some models focused on text and code do not yet offer native multimodal capabilities. Behind this is a trade-off between two technical routes:

• Native multimodal route: Unify image, text, audio, and video representations during pre-training. The advantage is cross-modal transfer and unified understanding; the cost is a sharp increase in data engineering complexity, mutual interference among multiple training objectives, and potentially diluted resources for text capability optimization.
• Single-modality focus route: Concentrate compute, data, and post-training budget on text and code. It is easier to achieve excellence on corresponding benchmarks, and the inference pipeline is simpler with more controllable cost.

Whether “native multimodality drags down text capability optimization” has no definitive answer yet—it depends on both architecture design and team resource constraints. But recent results suggest that, under limited resources, a focused strategy does produce results faster. The practical takeaway for developers: applications primarily built on text and code need not pay a premium for multimodal capabilities they do not yet use; conversely, if the business strongly depends on image or speech understanding, multimodal support should be a non-negotiable requirement.

8. The Developer’s Perspective: How to Evaluate and Select Rationally

Based on the analysis above, here are several actionable practical recommendations:

1. Build your own evaluation set. Sample from real business scenarios to construct an evaluation set covering the typical task distribution. Treat public leaderboards only as reference coordinates, not as a decision basis.
2. Evaluate across multiple dimensions. Cover at least five dimensions: task success rate, average latency, per-task cost, long-horizon task stability, and security/compliance. Avoid single-score orientation.
3. Tiered model selection. Use flagship-tier models for high-value complex tasks and cost-effective tier models for large-scale batch processing. Route different traffic to different tiers through request routing strategies.
4. Automated regression. Model iteration is extremely fast. Build automated evaluation pipelines using general-purpose cloud platform capabilities such as model service integration, logging services, and monitoring/alerting (for example, platforms like Baidu AI Cloud provide such infrastructure), and run regressions monthly or quarterly to capture capability changes in time.
5. Focus on execution capability, not just Q&A scores. For agent-type applications, emphasize the model’s planning, tool use, and failure recovery in multi-step tasks.

9. Conclusion

Score convergence is not the end of competition—it is a signal that the competitive dimension is shifting. When single-point intelligence gaps are erased, the real watershed turns to system efficiency, autonomous execution capability, and intelligence per unit cost. Evaluation systems need to evolve alongside capability boundaries, and the most important action for developers is to build a measurement system that belongs to their own business. After all, leaderboards measure a model’s average performance, while your business tests how a model performs on a specific distribution. Whoever completes the transition from “chasing leaderboards” to “building their own ruler” earlier will be better positioned to capture the next wave of model dividends.