When Aggregate LLM Scores Converge, Real Capability Still Diverges

    When top models cluster within a few points on aggregate intelligence scores, those scores stop being a reliable way to rank them. This article explains why benchmark saturation, hidden long-tail failures, and evaluation variance blur rankings, and how post-training, sparse activation, process metrics, unit-task cost, and modality fit change model selection.

    Recent discussion has focused on mid-scale activated models catching larger models on aggregate intelligence scores. On the surface, this loosens the parameter-scale narrative. More importantly, it raises a practical question: once leading models sit in a narrow score band, can a single leaderboard still tell which model is stronger? If several models differ by only a few index points—and that gap may be smaller than evaluation noise—what does “catching up” actually mean? This article does not rank products. It turns the pattern into a reusable method for evaluation and selection.

    1. Score convergence means lower resolution, not equal capability

    Aggregate intelligence scores usually combine many task types: knowledge Q&A, code generation, math reasoning, instruction following, and tool use. An aggregate score is useful for identifying a frontier cluster, but weak for fine ranking. Three reasons explain why.

    First, benchmarks saturate. When leading models approach the ceiling on most public tasks, individual items lose discriminative power and composite scores bunch together. A gap of a few index points may come from task weighting, sampling temperature, answer-format fit, or overlap between the evaluation set and training data—not only from underlying reasoning ability.

    Second, aggregation hides the long tail. A model may score well on general Q&A while remaining unstable on multi-file dependency analysis in a real codebase, failure reproduction, or permission-boundary judgment. The composite score averages those long-tail failures away. Readers see “similar totals,” not different failure modes.

    Third, evaluation variance is not negligible. Generative outputs are stochastic, so the same model can fluctuate across runs. If the gap among leaders is only a few index points, rankings mean little unless confidence intervals, run counts, and evaluation-script versions are reported.

    A sounder reading is that aggregate scores show whether a model has entered the frontier band, not that A is certainly stronger than B. Real selection should drill into task distribution, failure cases, and process metrics.

    2. Pre-training and post-training: scale is no longer the only lever

    Another point in this discussion is that base pre-training ability may not have made a generational jump, while post-training can still lift aggregate scores. Post-training includes instruction tuning, preference optimization, rejection sampling, reasoning-trace curation, tool-use training, and answer-format alignment. Its effect on composite scores is direct: the model better matches how evaluation tasks are phrased, drifts less, and answers in clearer steps.

    That does not mean post-training can indefinitely replace pre-training. Post-training is better at stably releasing capability that already exists than at creating new knowledge from nothing. If the base model is weak at long-context retrieval, complex code understanding, or cross-file reasoning, post-training can improve how that ability is invoked, but it cannot remove the capability boundary.

    Sparse activation also separates total parameter count from the amount activated at inference. A model can keep a large total capacity while activating only part of its parameters, lowering cost per inference. That is why parameter scale is no longer the only comparison axis: total capacity sets the upper bound on knowledge, activation paths and routing quality set inference efficiency, and post-training sets how fully capability is released. These three should be judged separately.

    3. From outcome scores to process ability: lessons from autonomous research evaluations

    Once public aggregate scores converge, more discriminative tests shift toward process ability. One example is an offline task in which a model must autonomously improve a small language model, drive validation loss below a set threshold within a fixed budget, and be compared on training steps, effective experiments, and resource use. The test is not only the final answer. It asks whether the model can form a hypothesis, design an experiment, read the result, correct course, and keep iterating.

    A typical evaluation loop can be abstracted as:

    1. evaluate(task, budget):
    2. state = init_state()
    3. while budget.remaining > 0 and not task.solved(state):
    4. plan = model.propose_plan(state)
    5. result = isolated_env.run(plan)
    6. state = model.update_state(state, result)
    7. record(plan, result, cost, failure_reason)
    8. return task.metrics(state), process_metrics

    The pseudocode is not the point. The process metrics are: share of effective experiments, repeat-experiment rate, time to first breakthrough, failure-recovery count, cost per unit of improvement, and whether the run converges before the budget is exhausted. Many models can propose similar correct directions, such as changing learning rate, data mix, or training epochs. The gap is whether they can turn a direction into executable experiments, extract signal from failed results, and keep search efficient after repeated failures.

    This suggests that gaps in “local intelligence” may be smaller than aggregate scores imply. The real split shows up in deep composition, sustained experimentation, and engineering execution. For developers, selection should require multiple runs on real repositories, real logs, and real failure cases—not a single Q&A turn.

    4. Cost and security review: unit-task cost changes how many rounds you can run

    On codebase vulnerability-discovery tasks, a lower-cost model matters not only because it is cheaper, but because it changes the design space of the review pipeline. Security review is rarely one question. It is a multi-stage flow: static scanning to locate suspicious points, generating vulnerability hypotheses, constructing a reproduction path, proposing a fix, regression checking, and filtering false positives. Each stage may need several attempts.

    If unit-task cost is mid-range, the same budget can support more review rounds, a wider candidate set, and parallel checks of more hypotheses. That is especially attractive for open models, because a team can run them in an isolated container, store evidence in object storage, send run records to a logging service, and watch anomalies with monitoring alerts. The relevant cloud capabilities here are isolation, retention, and observability—not the model’s own ability.

    Lower unit cost is not the same as lower total cost. Total security-review cost also includes false-positive rechecks, sandbox resources, permission isolation, human confirmation, and fix regression. A model with low unit-task cost but a high false-positive rate can raise total cost. Evaluation should therefore record candidate vulnerabilities per thousand lines of code, reproducible share, time spent handling false positives, average adoption rate of fix suggestions, and end-to-end cost per review.

    5. Is missing multimodality a hard limitation? It depends on the task boundary

    Lack of multimodality is often treated as an obvious weakness, but whether it is a hard limitation depends on the task. For code security, log analysis, configuration review, and autonomous experimentation, inputs are mainly text, code, and structured results, so native multimodality is not required. For UI screenshot review, design-to-code, monitoring-video understanding, and scanned-document processing, missing multimodality is a capability boundary.

    It is also wrong to infer that native multimodality necessarily hurts post-training. The two are not causally linked. Multimodal training adds complexity in data alignment, encoder design, and compute scheduling, and it can compete with text-reasoning optimization for resources. It can also be attached in a modular way without sacrificing text post-training. What actually determines post-training results is data quality, verifier design, training objectives, and the evaluation loop—not whether multimodal input exists.

    Treat multimodality as a scenario-fit item, not a universal quality ranking. If the workload is mostly code and text, prioritize long context, tool use, process stability, and unit-task cost. If it involves images, scanned documents, or interface understanding, multimodality is a hard gate.

    6. How to read “catching up”: a practical evaluation checklist

    When aggregate scores converge, replace a single leaderboard judgment with the following:

    1. Look at task distribution, not the total score. Split evaluation into code understanding, cross-file reasoning, tool use, security review, long-context retrieval, and similar subtasks, and inspect failure modes.
    2. Compare under a fixed budget. Use the same token budget, run count, and tool permissions, and record process metrics plus confidence intervals.
    3. Build an internal real set. Run a small blind test on private codebases, historical incident tickets, and real review cases so saturated public benchmarks do not mislead the decision.
    4. Calculate total cost of ownership. Include inference cost, verification cost, false-positive handling, isolation resources, and human review.
    5. State capability boundaries explicitly. Multimodality, very long context, private deployment, and compliance isolation should be gates, not patches applied afterward.

    Overall, “catching up” is better read as this: leading models have entered the same frontier band on public aggregate metrics, and total scores can no longer open a clear gap. Real differentiation is moving toward process ability, engineering execution, cost structure, and scenario boundaries. For developers and enterprise users, the useful move is not arguing over a few points. It is building an evaluation loop close to real work, and requiring the model to prove itself in an environment that is reproducible, measurable, and auditable.