After Benchmark Scores Converge: How to Evaluate the Real Capabilities of Large Language Models
When top LLMs post near-identical leaderboard scores, the aggregate number can no longer reliably answer which model to choose. This article explains why scores converge and become misleading—benchmark saturation, data contamination, opaque task weighting, the gap between static tasks and real work, and inconsistent grading. It then proposes a framework of six measurable capability dimensions (knowledge and reasoning, instruction following and format constraints, long-context robustness, code and software engineering, tool calling and structured output, and multimodal understanding), a method for building reproducible self-built evaluation sets backed by statistical significance testing, and techniques for turning perceived quality decay into observable metrics. Finally, it shows how to embed evaluation into the development workflow through a three-tier system, upgrade regression gates, and continuous online observation.
1. Introduction: Answering “Who Is Really Stronger” When Scores Tie
Open the public leaderboards and the top tier often differs by only one or two points in overall score: some tie, others lead by a single point. In daily use, these models are often described as “each having its own strengths,” some praised, others suspected of having their real performance overstated. Skepticism is not the same as rejection—it points to an engineering fact: when scores converge at the same level, the total score can no longer reliably answer “which one should I choose.” Anyone who needs to make selection decisions, along with architects and algorithm engineers who rely on rankings as backing, needs a more precise evaluation method.
This article does not judge the merits of specific products. It answers three questions:
- Why do total scores converge, and why do they become misleading?
- Beyond the total score, what measurable dimensions should “capability” be broken into?
- How do you use reproducible, regression-friendly engineering methods to turn “perceived differences” into a basis for decisions?
2. Deconstructing the Phenomenon: How Top Scores Converge
2.1 Benchmark Saturation and Declining Discrimination
Once a capability is widely mastered by most top models, the benchmark’s pass rate approaches the ceiling across the board, item difficulty no longer has a gradient, and a handful of wrong answers decide the ranking. Meanwhile, the variance of a single-sample metric can reach several percentage points: change the prompt template or adjust sampling parameters, and the ranking may flip. In the saturated range, a benchmark has weak power to identify the “strongest model” and is better suited to finding models that are “clearly unqualified.”
2.2 Data Contamination and Overfitting
As training corpora grow in scale and diversity of sources, public evaluation questions and their variants can easily enter the training data during pre-training or post-training. The model may have memorized the questions and their solutions rather than learned to reason. Discrimination is usually judged on two points:
- Held-out private set performance: self-built, unpublished tasks with no overlap with the training corpus.
- Out-of-distribution generalization: after rephrasing a question, changing a set of values, or adjusting the task format, whether the score remains stable.
High scores on private questions are more convincing; high scores on public questions call for some skepticism.
2.3 Task Weighting Masking Weaknesses
The total score is a weighted aggregation of multiple subtasks, and the weights are often opaque. Take two models as an example: Model A is slightly ahead on knowledge Q&A, math, and code, while Model B is clearly stronger on long-document understanding and multimodal document parsing; combined, they may differ by only one point. That one point may come from Model A’s overall balance, or it may be that Model B’s critical weakness is “offset” by high-scoring tasks.
2.4 Mismatch Between Static Tasks and Real Work
Benchmark questions are mostly tasks with clear boundaries, limited context, and one-shot adjudication, whereas real work often involves:
- Inputs containing tens of thousands of words of material;
- Multiple rounds of clarification;
- Calling external tools;
- Long-cycle, multi-step modifications;
- Cost and latency constraints.
A model that trails by one or two points on static tasks may be clearly stronger or weaker on long-task stability, instruction following, tool calling, and other dimensions—and the total score is “blind” to this.
2.5 Inconsistent Scoring Standards
Code tasks are often scored by whether they pass test cases, text tasks often rely on model review or human scoring, and each leaderboard has different prompt templates, sample sizes, and sampling temperatures. Cross-leaderboard comparison is inherently not rigorous.
Summary: A tied score does not mean equivalent capability. To judge “who is more suitable,” you need to break the total score into scenario-specific, repeatably measurable sub-metrics.
3. A Framework: Six Measurable Dimensions of LLM Capability
It is advisable to break “capability” into the following six categories, each grounded in executable, automatically scorable tests.
3.1 Knowledge and Reasoning
Includes basic factual Q&A, multi-step math, and logical reasoning. Scoring can use questions with unique, verifiable answers, combined with multiple sampling runs to assess stability and reduce the impact of single-run fluctuations.
3.2 Instruction Following and Format Constraints
Includes length constraints, structural constraints, conditional constraints (e.g., “execute Y only when X is satisfied”), and negative constraints (e.g., “must not output a certain type of information”). This capability can be judged by strict rules and is key to distinguishing “able to answer” from “usable.”
3.3 Long Context and Position Robustness
Includes cross-document information extraction, multi-document conflict resolution, and long-range coreference tracking. A common pitfall is that “needle in a haystack” tests are too easy: the key sentence stands alone in a paragraph, wording is highly consistent, and positions are concentrated, so the model easily hits it, whereas key information in real long documents is scattered, variably worded, and contradictory. Two more effective approaches:
- Scatter key information across the beginning, middle, and end;
- Add difficulty with synonymous rephrasing and noise distractors.
3.4 Code and Software Engineering
Layered evaluation is recommended:
- Snippet level: completion, bug fixing, unit test generation.
- Function level: implement a function from an interface specification, judged by execution results.
- Repository level: locate defects in a multi-file project, make cross-file changes, and preserve existing conventions, measured by regression test pass rate.
Repository-level tasks are closest to real development and also the most discriminating.
3.5 Tool Calling and Structured Output
Key evaluation points include: whether parameter types and values are valid, whether required fields are complete, whether multi-tool orchestration is correct, whether it can retry or correct after failure, and whether the JSON schema is strictly followed.
import jsonREQUIRED_FIELDS = {"city": str, "date": str}def check_tool_call(raw: str):"""Example: validate that the model-generated tool call arguments are valid"""try:args = json.loads(raw)except json.JSONDecodeError:return False, "invalid JSON"if not isinstance(args, dict):return False, "top level is not an object"missing = {k for k in REQUIRED_FIELDS if k not in args}if missing:return False, f"missing fields: {missing}"for k, t in REQUIRED_FIELDS.items():if not isinstance(args[k], t):return False, f"wrong field type: {k}"return True, "ok"
3.6 Multimodal Understanding
The focus is document understanding, including printed and handwritten text recognition, table restoration, layout reading order, cross-image comparison, and chart value reading. It should also be cross-validated with pure-text capability:
- Questions that can be guessed correctly using only language priors say nothing about visual understanding;
- Only questions that must “see” the information in the image to answer correctly can truly distinguish models.
In addition, differences in training objectives leave structural differences across capabilities. A common approach is to jointly train a vision encoder and a language model: use contrastive learning to align coarse-grained image-text semantics, then use a “predict the next word” language modeling objective to learn fine-grained information. According to related technical disclosures, the latter is more conducive to modeling fine-grained information such as text details and layout structure. Therefore, multimodal evaluation must design separate tests for text, layout, and structure, and cannot look only at overall accuracy.
4. Method: Building a Trustworthy Self-Built Evaluation Set
4.1 Task Source: Sample from the Real Distribution
Public questions are for a “health check”; the self-built set is for “decisions.” Sources of self-built tasks include:
- Desensitized real request logs;
- Online failure cases;
- Typical tickets from domain experts;
- Variant transformations of public questions, used to test generalization.
4.2 Scoring Method: Choose by Task
| Task type | Recommended scoring method |
|---|---|
| Unique answer | Exact match + numerical tolerance |
| Code/function | Execute unit tests in an isolated sandbox |
| Tool call/JSON | Schema validation + business rule validation |
| Open-ended text | Rubric + majority vote over multiple reviews + human spot checks |
| Subjective preference | Double-blind pairwise comparison with anonymized order and randomized left/right position |
The review model can also err, and may even “prefer longer answers” due to style bias. Therefore, open-ended tasks need three constraints:
- A clear rubric;
- Majority vote over multiple reviews;
- Human spot-check review.
4.3 Statistical Significance: Don’t Be Convinced by a One-Point Difference
Model A answers 420 of 500 questions correctly (84.0%), and Model B answers 410 correctly (82.0%). A appears to lead by 2 percentage points, but if we reduce the sample size to 100: A answers 84 correctly, B answers 82 correctly, and the gap between them can no longer be distinguished from random fluctuation.
A more robust approach is to do a paired comparison on the same set of questions, count only the questions where “the two models’ answers differ,” and then report a confidence interval:
import randomdef paired_win_rate(records, n_boot=2000, alpha=0.05, seed=0):"""Example: paired comparison and bootstrap confidence interval on the same question setrecords: [{'a_correct': bool, 'b_correct': bool}, ...]returns: A's win rate over B (ties count as 0.5) and the 95% confidence interval"""rng = random.Random(seed)def score(rs):a = b = t = 0for r in rs:if r["a_correct"] and not r["b_correct"]:a += 1elif r["b_correct"] and not r["a_correct"]:b += 1else:t += 1return (a + 0.5 * t) / len(rs)base = score(records)samples = sorted(score([rng.choice(records) for _ in records])for _ in range(n_boot))lo = samples[int(n_boot * alpha / 2)]hi = samples[int(n_boot * (1 - alpha / 2)) - 1]return base, lo, hi
How to interpret:
- Confidence interval contains 0.5: the two cannot be considered different;
- Confidence interval does not contain 0.5: the difference is statistically robust;
- Confidence interval is narrow but close to 0.5: the difference is real but of limited business significance.
What ultimately drives selection is often business impact: a tiny gap in the same capability may be amplified in scenarios where the cost of errors is high.
5. Advanced: Turning “Perceived Differences” into Observable Metrics
Users often report a phenomenon: the same model performs well early in a conversation, but as the context grows longer, output begins to drift off-topic, repeat, and omit constraints—what is commonly called “quality decay.” Such observations are valuable, but they need to be turned into measurable, reproducible metrics to enter the decision process.
5.1 Measuring Quality Decay
Design a “long-session task chain”: fix a task sequence, add task-relevant material to the context each turn, and check whether the model:
- Keeps constraints consistent;
- Correctly retrieves earlier information;
- Avoids self-contradiction.
Using task turn as the horizontal axis, plot a “single-turn accuracy vs. turn” curve to quantify the rate of decay.
def quality_decay(task_chain, model_call):"""Example: measure instruction retention as the conversation growstask_chain: tasks ordered by turn, each with a prompt and a check functionreturns: [(turn, pass rate for that turn), ...]"""history, result = [], []for turn, task in enumerate(task_chain, start=1):history.append({"role": "user", "content": task["prompt"]})out = model_call(history)history.append({"role": "assistant", "content": out})passed = sum(task["check"](out) for _ in range(1)) # in practice, average over multiple samplesresult.append((turn, passed))return result
Beyond “how fast it decays,” you should also care about “whether it can recover after decay.” For example, ask the model to restate all constraints, or to explicitly clarify when a conflict arises, and see whether it can get back on track. This directly determines the fault-tolerance cost in long tasks.
5.2 Cost, Latency, and Stability
With the same capability, latency and cost can differ by several times. Load-test on your own traffic and record:
- Time to first token;
- Output speed;
- P95/P99 latency;
- Average token consumption per task;
- Error rate and retry rate.
If the inference service supports context caching, the reuse of repeated prefixes should also be factored into the cost calculation.
5.3 Comprehensive Evaluation
Make decisions with a “capability–cost” matrix rather than a single total score:
- When capabilities are close, choose the one with lower latency, lower cost, and greater stability;
- When a task has extremely high demands on detail, keep a stronger model on the critical path and route the remaining requests to a more economical model.
On the “dumbing down” rumors: Such claims mostly come from scattered experiences. Rather than arguing whether they are true, it is better to reproduce them in your own scenarios using the method in 5.1: can it be reliably reproduced, under what conditions does it appear, and how large is the impact. Measured conclusions are more reliable than rumors.
6. Implementation: Embedding Evaluation into the R&D Process
6.1 A Three-Tier Evaluation System
| Tier | Content | Trigger | Scale |
|---|---|---|---|
| Smoke tier | Core instruction following, format, tool calling | Every commit | Minutes |
| Regression tier | Full self-built set + private held-out set | Daily / every version upgrade | Hours |
| Special tier | Long context, multimodal, repository-level code | Milestones / selection evaluation | On demand |
6.2 Upgrade Regression Gate
After model service upgrades, prompt changes, or inference parameter adjustments, the regression tier should be rerun, and old and new versions compared using a “frozen” golden set:
# Example: upgrade regression gateGATE_MIN_SCORE = 0.80 # minimum overall pass rate for the regression setGATE_MAX_DROP = 0.02 # maximum allowed regression relative to the current online versiondef gate(old_rate, new_rate, p95_latency_ms, p95_budget_ms):if new_rate < GATE_MIN_SCORE:return "Block: overall pass rate too low"if new_rate < old_rate - GATE_MAX_DROP:return "Block: key capability regression"if p95_latency_ms > p95_budget_ms:return "Warn: latency exceeds budget"return "Pass"
6.3 Continuous Online Observation
Offline evaluation is only sampling; online behavior is full-scale evidence. Available observation signals include:
- User regenerate, copy, and thumbs-down behaviors;
- Human spot-check failure rate;
- Failure request distribution;
- Success rate of extra-long context requests.
With logging services and monitoring alerts, a review can be triggered as soon as a key metric crosses a threshold. Failure cases then flow back into the self-built evaluation set, forming a closed loop of “evaluate—improve—re-evaluate.”
7. Common Pitfalls
- Looking only at the total score: The total score masks weaknesses; different scenarios should assign different weights.
- Drawing conclusions from a single evaluation: Ignoring variance; you must use paired comparison and confidence intervals.
- Using training-set questions as the exam: Data contamination inflates scores, so a private held-out set must be preserved.
- Evaluation disconnected from production: Leaderboard questions are not equal to real tasks; the evaluation set must be continuously updated with real traffic.
- Assuming the review model is neutral: The review model can also have style biases, requiring a rubric, multiple reviews, and human spot checks.
- Ignoring cost and latency: Capability is only one dimension of the decision; usability is equally critical.
8. Conclusion
When the public scores of top models crowd into the same range, “who is stronger” is no longer a question the leaderboard can answer. A reliable judgment comes from three shifts:
- From “a single total score” to “multi-dimensional metrics broken down by scenario”;
- From “a single evaluation” to “paired comparison plus statistical significance”;
- From “subjective perception” to “a reproducible, regression-friendly, continuously observable engineering system.”
For those making selections, the most trustworthy evidence always comes from stable performance on your own evaluation set and long-term data from real business.
