LLM Evaluation: How to Judge Real Capability When Benchmark Scores Cluster
On public LLM leaderboards, top scores have converged and small gaps no longer reliably indicate which model is better in real workloads. This article explains why scores distort in the high-score range—score saturation, benchmark contamination, narrow task formats, and missing engineering and cost factors—and traces the "quality drop" users often feel to stability, capability specialization, long-task degradation, and unsafe boundary behavior. It then lays out a reproducible evaluation system built from your own tasks: sampling real traffic, defining acceptance criteria, automated screening with sampled human review, repeated sampling to quantify stability, and metric tracking with regression comparison, accompanied by a sample script. It also shows how multimodal training objectives shape capability, and ends with recommendations for turning evaluation into an ongoing mechanism. Readers come away with a practical framework for choosing models by scenario rather than by leaderboard rank.
1. Introduction: Scores Keep Rising, but the Experience Varies
Every time a new generation of models ships, you hear claims like “best overall score” or “tops the leaderboard.” On public aggregate leaderboards, the top models have converged: the leading group sits around 63 points, with a second tier between 60 and 62. One model scoring 62 overall was reported by users to “lose IQ”—giving noticeably different results across sessions and times of day. This “similar scores, different experience” pattern is common, and it points to an important problem:
In the high-score range, tiny gaps between scores are no longer reliable. Reading the leaderboard alone is not enough to judge a model’s value in real workloads.
This article covers two things: why scores distort in the dense high-score zone, and how to build an evaluation system that fits your own business, is reproducible, and can be run as an ongoing regression. At the end, a training design detail from multimodal models shows that the choice of evaluation task often matters more than the score itself.
2. Three Types of Distortion in the Dense-Score Zone
2.1 Score saturation: a 0.1-point gap is almost meaningless
Aggregate leaderboards are usually a weighted combination of many dimensions. A total drop from 63 to 60 looks like just 3 points, but at the per-item level the difference can be large. Two reasons:
- The evaluation itself fluctuates: sampling temperature, prompt wording, the scoring model, and answer-extraction rules all affect the score, and random noise alone is often a few tenths of a point.
- The scoring method influences the conclusion: generative models are often graded 1–5 by another model, and small changes in the rubric or prompt can move the score up or down.
A more reasonable reading: leading models sit in the same capability band, and the real differences show up on specific tasks rather than in the total ranking. Besides, a 0.1-point gap is barely perceptible to users in a scenario where you handle thousands of requests a day.
2.2 Benchmark contamination: the test may already be “leaked”
Most leaderboard items are public. If the test questions or highly similar ones appear in the training corpus, the model may have memorized the answers rather than learned the skill, and its score is systematically inflated—while real performance can lag well behind.
On top of that, the mainstream evaluation sets have been reused for years. Models, training teams, and even evaluation teams may adapt to them, so the leaderboard becomes less representative of genuinely new tasks. Two ways to check:
- Score the public benchmark and your own private question set separately, and see whether they agree.
- Watch how models perform on “new questions” released after their evaluation.
2.3 Single evaluation dimension: multiple-choice ≠ real tasks
Multiple-choice questions have one correct answer, are easy to grade automatically, and are the easiest to score high on. Real workloads—code generation, multi-turn revisions, document processing—must be judged on correctness, completeness, and usability all at once.
A typical example: a model writes code that passes on the first try but skips edge-case handling. Feed that code to an AI coding assistant, let it generate and commit the result, and the problems only surface in production. There is a wide gap between “correct on the test” and “usable in business.”
2.4 Missing engineering and cost dimensions
Leaderboards barely reflect these real constraints:
- Time to first token and overall generation speed;
- Context window size and long-text handling;
- Rate limits, concurrency capacity, and availability at peak times;
- Average tokens consumed per task: verbose output translates directly into cost.
The takeaway: once scores fall within the same band, “which model is strongest” is no longer the key question—“which model best fits my workload” is.
3. Back to “the Experience”: Where Does the Gap Actually Come From?
Most differences in user experience trace back to one of four issues.
1. Stability. Common causes of the “IQ drop” mentioned earlier include:
- High sampling parameters (such as temperature), so each answer varies a lot;
- Multi-turn conversations that keep accumulating history, gradually “diluting” earlier constraints;
- Server load or version updates that make behavior inconsistent over time.
Strictly speaking, “IQ drop” is more of a subjective impression and needs quantifiable metrics to confirm.
2. Uneven capability. Some models are stronger at complex reasoning and coding; others are more reliable at long-document summarization and instruction following. Overall scores look similar, but specialized capabilities can differ markedly. For example, when a model repeatedly fails at drawing flowcharts, the root cause often turns out to be relatively weak long-context ability. This kind of unevenness matters most when choosing an AI coding tool or a document-processing solution.
3. Long-task degradation. Multiple-choice questions involve only one or two turns, while real tasks run dozens of turns: change a requirement, hit an error, fix it, verify again. As turns increase and context grows, the model may gradually forget the original constraints, reintroduce bugs that were already fixed, or lose track of dependencies in a multi-file project. This is exactly the weakness that single-run benchmarks fail to reveal.
4. Safety and boundary behavior. Actions like overstepping permissions, deleting data, or ignoring security requirements in the system prompt are almost invisible in benchmarks, yet they are what enterprises care about most when deploying.
4. Building Your Own Evaluation System: From Leaderboards to Business
4.1 Overall approach: a five-step loop
Sample business tasks -> Build the question set and acceptance criteria -> Automated screening -> Human/model spot-check review -> Metric analysis and regression comparison
Step 1: Work backward from real tasks to build the question set
- Source: Prefer real inputs such as production logs, tickets, and customer-service records rather than public question banks.
- Stratify: Group by scenario (Q&A, writing, code, document extraction) and by difficulty (easy, medium, hard). Difficulty can be defined by business metrics, such as average human handling time or failure rate.
- Anonymize: Remove personal information such as names, phone numbers, and ID numbers; sensitive enterprise data should be processed in your own environment or anonymized first.
- Scale: 30–100 questions per scenario is usually enough to reliably distinguish models.
Step 2: Define clear acceptance criteria for each question
Take a “contract clause summarization” task as an example:
| Check | Acceptance criterion |
|---|---|
| Key clauses | Required fields such as payment terms, liability for breach, and confidentiality period are all extracted correctly |
| Faithfulness to source | Every conclusion can be located in the original text; nothing is fabricated |
| Format | Output conforms to the agreed JSON structure |
| Conservatism | Content not specified in the original must be labeled “not specified” rather than guessed |
Once the criteria are explicit, a subjective “feels good/bad” becomes a computable, reproducible pass rate.
Step 3: Automated screening + spot-check review
- Rule checks: Field completeness, JSON parseability, and whether code compiles and passes unit tests can be fully automated.
- Model scoring: For open-ended generation, another model can score against a rubric. Watch for “position bias” (a tendency to prefer the earlier answer) and “verbosity bias” (longer answers tend to score higher); if needed, swap the answer order and score again.
- Human review: Sample 10%–20% of the model-scored results. Focus on boundary and disagreement samples, and review all samples for high-risk, low-tolerance scenarios (finance, healthcare, legal, etc.).
Step 4: Sample multiple times to quantify stability
- Repeat each question at least 3–5 times and record the distribution of success rates, rather than looking at a single run.
- If a question succeeds only once in five tries, it should not be used for automated production decisions, even if the average score looks acceptable.
- For long tasks, additionally record the number of turns needed, whether constraints were violated midway, and the growth curve of context length.
Step 5: Metric analysis and regression comparison
| Dimension | Reference metrics | Notes |
|---|---|---|
| Effectiveness | Task pass rate, first-pass rate | Core metrics, tracked per scenario |
| Stability | Success rate across repeated samples, result variance | Identifies “intermittent failures” |
| Long tasks | Average turns to completion, number of constraint violations | Reflects context-handling ability |
| Cost | Average input/output tokens, cost per task | Maps directly to the bill |
| Performance | Time to first token, total latency, failure rate | Affects user experience |
| Safety | Rate of over-permission actions, harmful-output block rate | A red line for enterprise deployment |
After a version upgrade, rerun the same question set with the same parameters and the same scoring criteria, and compare against historical results. When a regression appears, you can quickly determine whether the change came from the model or from the evaluation environment.
4.2 Sample code: a reproducible, multi-scenario evaluation script
import statisticsfrom collections import defaultdict# Question set structure: id is a unique identifier; scene is the business scenario; prompt is the anonymized real input;# verify is the acceptance function, which takes the model output and returns (passed, note); samples is the number of repeats per questionTASKS = [{"id": "sum-001", "scene": "Summarization", "prompt": "...", "verify": verify_summary, "samples": 5},{"id": "code-001", "scene": "Code", "prompt": "...", "verify": verify_code, "samples": 3},{"id": "doc-001", "scene": "Extraction", "prompt": "...", "verify": verify_extract, "samples": 5},]def evaluate(call_model, tasks):records = []for task in tasks:passes = []for _ in range(task["samples"]):output = call_model(task["prompt"]) # unified wrapper, so you can swap in different modelsok, _note = task["verify"](output) # rule checks + model scoring + human review resultspasses.append(ok)records.append({"task_id": task["id"],"scene": task["scene"],"pass_rate": sum(passes) / len(passes),"all_pass": all(passes), # stable only if all repeated samples pass})by_scene = defaultdict(list)for r in records:by_scene[r["scene"]].append(r["pass_rate"])report = {s: {"avg_pass_rate": round(statistics.mean(v), 3),"min_pass_rate": round(min(v), 3), # worst-question performance, more telling of weak spots than the average}for s, v in by_scene.items()}report["overall"] = {"avg_pass_rate": round(statistics.mean(r["pass_rate"] for r in records), 3),"stable_task_ratio": round(statistics.mean(r["all_pass"] for r in records), 3),}return report
In practice, you only need to swap the call_model implementation to compare different models. For every evaluation, pin down and archive the following; otherwise the results cannot be reproduced:
- Model version identifier;
- Sampling parameters such as temperature and maximum output length;
- System prompt and scoring rules;
- Evaluation-set version.
5. An Easily Overlooked Detail: Multimodal Training Objectives Determine Where Capability Lands
Public evaluations are mostly text-based, and multimodal ability is often reduced to “can it read an image.” But in training a multimodal model—typically made up of a vision encoder, a connecting module, and a large language model—what objective you optimize for directly determines what the model is good at.
In image-text contrastive training, the model maps images and their captions into the same semantic space. What it learns is mostly a global representation of “what the image is about overall,” which naturally suits tasks like “retrieve by image” and “image-text matching.”
Under a generative objective of “predict the next token from the image content,” the situation is different. The model must generate text that matches the image, so the vision encoder has to produce features that support language reasoning, including:
- Stroke shapes and glyph forms of text;
- Layout position and reading order;
- Structural relationships such as cells and hierarchy.
In other words, the language-modeling objective “pulls in reverse” on the vision side, driving it to extract fine-grained information. Public research supports this design idea: studies often freeze the vision encoder directly and train only the connecting module and the large language model, relying on exactly this reverse pull.
This explains a common experience gap: the same “multimodal model” may
- do well on “what is this image mainly expressing”;
- but lag noticeably on dense text recognition, table reconstruction, layout reconstruction, and receipt-field extraction.
So when selecting a model, don’t just look at “supports image input”—evaluate it separately with your own real image types (screenshots, scanned documents, tables, forms). Multimodal support may look like “it handles everything,” but what really determines the experience is its fine-grained performance on your kind of images.
6. Practical Recommendations: Make Evaluation an Ongoing Mechanism
- Don’t rely on a single leaderboard: Treat public benchmark scores as a “shortlisting reference,” and base the final decision on your own question set.
- Select per scenario: Evaluate Q&A, writing, code, and document processing separately. Within one system, you can perfectly well “use a lightweight model for simple tasks and a stronger model for complex ones,” with routing and fallback handled through a unified access layer.
- Establish a regression baseline: Rerun a baseline question set on every model upgrade to catch regressions early.
- Value observability: Log call records, response times, and failure reasons, and combine them with monitoring and alerting to spot anomalies quickly.
- Watch cost and compliance: Make decisions based on token consumption together with data-residency and privacy requirements. When data is sensitive, prefer model services that support private deployment or dedicated data isolation.
As an example of a common path for developers, you can use a model platform such as Baidu AI Cloud to access and orchestrate multiple models through a single interface, and consolidate your own evaluation question sets, call logs, and version information into one place—forming a closed loop of “evaluate -> deploy -> monitor -> re-evaluate.”
7. Conclusion
In the high-score range, the top models’ aggregate scores differ by only a point or two. At that point, arguing over “who is number one” no longer means much. What really decides success is four things:
- Whether the model is stable on your tasks;
- Whether it can complete multi-turn, long-context real work;
- Whether cost and latency are acceptable;
- Whether data is secure and compliant.
Leaderboards tell you “who is stronger”; your own question set tells you “who is the better fit.”
Rather than chasing benchmark numbers, spend half a day pulling a few dozen real tasks from your production logs and run a small-scale evaluation using the method in this article. The conclusions you get are often far more valuable than any leaderboard.
