After Benchmark Scores Converge: How to Evaluate the Real Capabilities of Large Language Models

    When top LLMs post near-identical leaderboard scores, the aggregate number can no longer reliably answer which model to choose. This article explains why scores converge and become misleading—benchmark saturation, data contamination, opaque task weighting, the gap between static tasks and real work, and inconsistent grading. It then proposes a framework of six measurable capability dimensions (knowledge and reasoning, instruction following and format constraints, long-context robustness, code and software engineering, tool calling and structured output, and multimodal understanding), a method for building reproducible self-built evaluation sets backed by statistical significance testing, and techniques for turning perceived quality decay into observable metrics. Finally, it shows how to embed evaluation into the development workflow through a three-tier system, upgrade regression gates, and continuous online observation.

    1. Introduction: Answering “Who Is Really Stronger” When Scores Tie

    Open the public leaderboards and the top tier often differs by only one or two points in overall score: some tie, others lead by a single point. In daily use, these models are often described as “each having its own strengths,” some praised, others suspected of having their real performance overstated. Skepticism is not the same as rejection—it points to an engineering fact: when scores converge at the same level, the total score can no longer reliably answer “which one should I choose.” Anyone who needs to make selection decisions, along with architects and algorithm engineers who rely on rankings as backing, needs a more precise evaluation method.

    This article does not judge the merits of specific products. It answers three questions:

    • Why do total scores converge, and why do they become misleading?
    • Beyond the total score, what measurable dimensions should “capability” be broken into?
    • How do you use reproducible, regression-friendly engineering methods to turn “perceived differences” into a basis for decisions?

    2. Deconstructing the Phenomenon: How Top Scores Converge

    2.1 Benchmark Saturation and Declining Discrimination

    Once a capability is widely mastered by most top models, the benchmark’s pass rate approaches the ceiling across the board, item difficulty no longer has a gradient, and a handful of wrong answers decide the ranking. Meanwhile, the variance of a single-sample metric can reach several percentage points: change the prompt template or adjust sampling parameters, and the ranking may flip. In the saturated range, a benchmark has weak power to identify the “strongest model” and is better suited to finding models that are “clearly unqualified.”

    2.2 Data Contamination and Overfitting

    As training corpora grow in scale and diversity of sources, public evaluation questions and their variants can easily enter the training data during pre-training or post-training. The model may have memorized the questions and their solutions rather than learned to reason. Discrimination is usually judged on two points:

    1. Held-out private set performance: self-built, unpublished tasks with no overlap with the training corpus.
    2. Out-of-distribution generalization: after rephrasing a question, changing a set of values, or adjusting the task format, whether the score remains stable.

    High scores on private questions are more convincing; high scores on public questions call for some skepticism.

    2.3 Task Weighting Masking Weaknesses

    The total score is a weighted aggregation of multiple subtasks, and the weights are often opaque. Take two models as an example: Model A is slightly ahead on knowledge Q&A, math, and code, while Model B is clearly stronger on long-document understanding and multimodal document parsing; combined, they may differ by only one point. That one point may come from Model A’s overall balance, or it may be that Model B’s critical weakness is “offset” by high-scoring tasks.

    2.4 Mismatch Between Static Tasks and Real Work

    Benchmark questions are mostly tasks with clear boundaries, limited context, and one-shot adjudication, whereas real work often involves:

    • Inputs containing tens of thousands of words of material;
    • Multiple rounds of clarification;
    • Calling external tools;
    • Long-cycle, multi-step modifications;
    • Cost and latency constraints.

    A model that trails by one or two points on static tasks may be clearly stronger or weaker on long-task stability, instruction following, tool calling, and other dimensions—and the total score is “blind” to this.

    2.5 Inconsistent Scoring Standards

    Code tasks are often scored by whether they pass test cases, text tasks often rely on model review or human scoring, and each leaderboard has different prompt templates, sample sizes, and sampling temperatures. Cross-leaderboard comparison is inherently not rigorous.

    Summary: A tied score does not mean equivalent capability. To judge “who is more suitable,” you need to break the total score into scenario-specific, repeatably measurable sub-metrics.

    3. A Framework: Six Measurable Dimensions of LLM Capability

    It is advisable to break “capability” into the following six categories, each grounded in executable, automatically scorable tests.

    3.1 Knowledge and Reasoning

    Includes basic factual Q&A, multi-step math, and logical reasoning. Scoring can use questions with unique, verifiable answers, combined with multiple sampling runs to assess stability and reduce the impact of single-run fluctuations.

    3.2 Instruction Following and Format Constraints

    Includes length constraints, structural constraints, conditional constraints (e.g., “execute Y only when X is satisfied”), and negative constraints (e.g., “must not output a certain type of information”). This capability can be judged by strict rules and is key to distinguishing “able to answer” from “usable.”

    3.3 Long Context and Position Robustness

    Includes cross-document information extraction, multi-document conflict resolution, and long-range coreference tracking. A common pitfall is that “needle in a haystack” tests are too easy: the key sentence stands alone in a paragraph, wording is highly consistent, and positions are concentrated, so the model easily hits it, whereas key information in real long documents is scattered, variably worded, and contradictory. Two more effective approaches:

    • Scatter key information across the beginning, middle, and end;
    • Add difficulty with synonymous rephrasing and noise distractors.

    3.4 Code and Software Engineering

    Layered evaluation is recommended:

    • Snippet level: completion, bug fixing, unit test generation.
    • Function level: implement a function from an interface specification, judged by execution results.
    • Repository level: locate defects in a multi-file project, make cross-file changes, and preserve existing conventions, measured by regression test pass rate.

    Repository-level tasks are closest to real development and also the most discriminating.

    3.5 Tool Calling and Structured Output

    Key evaluation points include: whether parameter types and values are valid, whether required fields are complete, whether multi-tool orchestration is correct, whether it can retry or correct after failure, and whether the JSON schema is strictly followed.

    1. import json
    2. REQUIRED_FIELDS = {"city": str, "date": str}
    3. def check_tool_call(raw: str):
    4. """Example: validate that the model-generated tool call arguments are valid"""
    5. try:
    6. args = json.loads(raw)
    7. except json.JSONDecodeError:
    8. return False, "invalid JSON"
    9. if not isinstance(args, dict):
    10. return False, "top level is not an object"
    11. missing = {k for k in REQUIRED_FIELDS if k not in args}
    12. if missing:
    13. return False, f"missing fields: {missing}"
    14. for k, t in REQUIRED_FIELDS.items():
    15. if not isinstance(args[k], t):
    16. return False, f"wrong field type: {k}"
    17. return True, "ok"

    3.6 Multimodal Understanding

    The focus is document understanding, including printed and handwritten text recognition, table restoration, layout reading order, cross-image comparison, and chart value reading. It should also be cross-validated with pure-text capability:

    • Questions that can be guessed correctly using only language priors say nothing about visual understanding;
    • Only questions that must “see” the information in the image to answer correctly can truly distinguish models.

    In addition, differences in training objectives leave structural differences across capabilities. A common approach is to jointly train a vision encoder and a language model: use contrastive learning to align coarse-grained image-text semantics, then use a “predict the next word” language modeling objective to learn fine-grained information. According to related technical disclosures, the latter is more conducive to modeling fine-grained information such as text details and layout structure. Therefore, multimodal evaluation must design separate tests for text, layout, and structure, and cannot look only at overall accuracy.

    4. Method: Building a Trustworthy Self-Built Evaluation Set

    4.1 Task Source: Sample from the Real Distribution

    Public questions are for a “health check”; the self-built set is for “decisions.” Sources of self-built tasks include:

    • Desensitized real request logs;
    • Online failure cases;
    • Typical tickets from domain experts;
    • Variant transformations of public questions, used to test generalization.

    4.2 Scoring Method: Choose by Task

    Task type Recommended scoring method
    Unique answer Exact match + numerical tolerance
    Code/function Execute unit tests in an isolated sandbox
    Tool call/JSON Schema validation + business rule validation
    Open-ended text Rubric + majority vote over multiple reviews + human spot checks
    Subjective preference Double-blind pairwise comparison with anonymized order and randomized left/right position

    The review model can also err, and may even “prefer longer answers” due to style bias. Therefore, open-ended tasks need three constraints:

    • A clear rubric;
    • Majority vote over multiple reviews;
    • Human spot-check review.

    4.3 Statistical Significance: Don’t Be Convinced by a One-Point Difference

    Model A answers 420 of 500 questions correctly (84.0%), and Model B answers 410 correctly (82.0%). A appears to lead by 2 percentage points, but if we reduce the sample size to 100: A answers 84 correctly, B answers 82 correctly, and the gap between them can no longer be distinguished from random fluctuation.

    A more robust approach is to do a paired comparison on the same set of questions, count only the questions where “the two models’ answers differ,” and then report a confidence interval:

    1. import random
    2. def paired_win_rate(records, n_boot=2000, alpha=0.05, seed=0):
    3. """
    4. Example: paired comparison and bootstrap confidence interval on the same question set
    5. records: [{'a_correct': bool, 'b_correct': bool}, ...]
    6. returns: A's win rate over B (ties count as 0.5) and the 95% confidence interval
    7. """
    8. rng = random.Random(seed)
    9. def score(rs):
    10. a = b = t = 0
    11. for r in rs:
    12. if r["a_correct"] and not r["b_correct"]:
    13. a += 1
    14. elif r["b_correct"] and not r["a_correct"]:
    15. b += 1
    16. else:
    17. t += 1
    18. return (a + 0.5 * t) / len(rs)
    19. base = score(records)
    20. samples = sorted(
    21. score([rng.choice(records) for _ in records])
    22. for _ in range(n_boot)
    23. )
    24. lo = samples[int(n_boot * alpha / 2)]
    25. hi = samples[int(n_boot * (1 - alpha / 2)) - 1]
    26. return base, lo, hi

    How to interpret:

    • Confidence interval contains 0.5: the two cannot be considered different;
    • Confidence interval does not contain 0.5: the difference is statistically robust;
    • Confidence interval is narrow but close to 0.5: the difference is real but of limited business significance.

    What ultimately drives selection is often business impact: a tiny gap in the same capability may be amplified in scenarios where the cost of errors is high.

    5. Advanced: Turning “Perceived Differences” into Observable Metrics

    Users often report a phenomenon: the same model performs well early in a conversation, but as the context grows longer, output begins to drift off-topic, repeat, and omit constraints—what is commonly called “quality decay.” Such observations are valuable, but they need to be turned into measurable, reproducible metrics to enter the decision process.

    5.1 Measuring Quality Decay

    Design a “long-session task chain”: fix a task sequence, add task-relevant material to the context each turn, and check whether the model:

    • Keeps constraints consistent;
    • Correctly retrieves earlier information;
    • Avoids self-contradiction.

    Using task turn as the horizontal axis, plot a “single-turn accuracy vs. turn” curve to quantify the rate of decay.

    1. def quality_decay(task_chain, model_call):
    2. """
    3. Example: measure instruction retention as the conversation grows
    4. task_chain: tasks ordered by turn, each with a prompt and a check function
    5. returns: [(turn, pass rate for that turn), ...]
    6. """
    7. history, result = [], []
    8. for turn, task in enumerate(task_chain, start=1):
    9. history.append({"role": "user", "content": task["prompt"]})
    10. out = model_call(history)
    11. history.append({"role": "assistant", "content": out})
    12. passed = sum(task["check"](out) for _ in range(1)) # in practice, average over multiple samples
    13. result.append((turn, passed))
    14. return result

    Beyond “how fast it decays,” you should also care about “whether it can recover after decay.” For example, ask the model to restate all constraints, or to explicitly clarify when a conflict arises, and see whether it can get back on track. This directly determines the fault-tolerance cost in long tasks.

    5.2 Cost, Latency, and Stability

    With the same capability, latency and cost can differ by several times. Load-test on your own traffic and record:

    • Time to first token;
    • Output speed;
    • P95/P99 latency;
    • Average token consumption per task;
    • Error rate and retry rate.

    If the inference service supports context caching, the reuse of repeated prefixes should also be factored into the cost calculation.

    5.3 Comprehensive Evaluation

    Make decisions with a “capability–cost” matrix rather than a single total score:

    • When capabilities are close, choose the one with lower latency, lower cost, and greater stability;
    • When a task has extremely high demands on detail, keep a stronger model on the critical path and route the remaining requests to a more economical model.

    On the “dumbing down” rumors: Such claims mostly come from scattered experiences. Rather than arguing whether they are true, it is better to reproduce them in your own scenarios using the method in 5.1: can it be reliably reproduced, under what conditions does it appear, and how large is the impact. Measured conclusions are more reliable than rumors.

    6. Implementation: Embedding Evaluation into the R&D Process

    6.1 A Three-Tier Evaluation System

    Tier Content Trigger Scale
    Smoke tier Core instruction following, format, tool calling Every commit Minutes
    Regression tier Full self-built set + private held-out set Daily / every version upgrade Hours
    Special tier Long context, multimodal, repository-level code Milestones / selection evaluation On demand

    6.2 Upgrade Regression Gate

    After model service upgrades, prompt changes, or inference parameter adjustments, the regression tier should be rerun, and old and new versions compared using a “frozen” golden set:

    1. # Example: upgrade regression gate
    2. GATE_MIN_SCORE = 0.80 # minimum overall pass rate for the regression set
    3. GATE_MAX_DROP = 0.02 # maximum allowed regression relative to the current online version
    4. def gate(old_rate, new_rate, p95_latency_ms, p95_budget_ms):
    5. if new_rate < GATE_MIN_SCORE:
    6. return "Block: overall pass rate too low"
    7. if new_rate < old_rate - GATE_MAX_DROP:
    8. return "Block: key capability regression"
    9. if p95_latency_ms > p95_budget_ms:
    10. return "Warn: latency exceeds budget"
    11. return "Pass"

    6.3 Continuous Online Observation

    Offline evaluation is only sampling; online behavior is full-scale evidence. Available observation signals include:

    • User regenerate, copy, and thumbs-down behaviors;
    • Human spot-check failure rate;
    • Failure request distribution;
    • Success rate of extra-long context requests.

    With logging services and monitoring alerts, a review can be triggered as soon as a key metric crosses a threshold. Failure cases then flow back into the self-built evaluation set, forming a closed loop of “evaluate—improve—re-evaluate.”

    7. Common Pitfalls

    1. Looking only at the total score: The total score masks weaknesses; different scenarios should assign different weights.
    2. Drawing conclusions from a single evaluation: Ignoring variance; you must use paired comparison and confidence intervals.
    3. Using training-set questions as the exam: Data contamination inflates scores, so a private held-out set must be preserved.
    4. Evaluation disconnected from production: Leaderboard questions are not equal to real tasks; the evaluation set must be continuously updated with real traffic.
    5. Assuming the review model is neutral: The review model can also have style biases, requiring a rubric, multiple reviews, and human spot checks.
    6. Ignoring cost and latency: Capability is only one dimension of the decision; usability is equally critical.

    8. Conclusion

    When the public scores of top models crowd into the same range, “who is stronger” is no longer a question the leaderboard can answer. A reliable judgment comes from three shifts:

    • From “a single total score” to “multi-dimensional metrics broken down by scenario”;
    • From “a single evaluation” to “paired comparison plus statistical significance”;
    • From “subjective perception” to “a reproducible, regression-friendly, continuously observable engineering system.”

    For those making selections, the most trustworthy evidence always comes from stable performance on your own evaluation set and long-term data from real business.