96% Memorization Rate: Can LLM Benchmark Scores Still Be Trusted?
This article examines benchmark contamination in large language models and why a high score may reflect memorized test data rather than genuine reasoning. Based on internal observations across four anonymized base model versions and 20 public benchmarks, it shows how one model reproduced 96% of a 164-question code-generation benchmark—including a spelling error from the original source—when prompted only with question numbers. It explains the common sources of test-set leakage, five practical detection methods (probe-based continuation, n-gram deduplication, problem perturbation, private holdout sets, and consistency checks) with working code examples, and concrete guidance for enterprises on evaluating and procuring models without being misled by leaderboard rankings.
Note: All models referenced in this article are described anonymously and do not point to any specific vendor or product. The figures come from internal reproduction observations and are not accompanied by publicly reproducible links, so they are better treated as a reference for evaluation methodology rather than a universal industry conclusion.
Benchmark Contamination: Why a High Score May Just Mean “Memorizing the Test”
Scores on LLM leaderboards have long been treated as a direct measure of capability. But after benchmark contamination has surfaced repeatedly, that assumption deserves a more careful look: does a high score reflect a model’s understanding, reasoning, and generation ability—or does it simply mean the model has seen the test before?
An important point here is that a score should not be equated with capability. A more reasonable framing is that model performance is shaped by both ability and data exposure. When the test questions, reference answers, question variants, or even the structure of the questions have already appeared in the training corpus, a high score may be the result of “having seen the exam” rather than “solving a new problem.”
Using an anonymized base model as a case study, this article discusses the common causes of benchmark contamination, detection methods, and recommendations for enterprise selection, helping developers build more trustworthy criteria for evaluation and procurement decisions.
Experimental Observation: The Model “Continued” Problems It Was Never Given
A “Continuation” With No Problem Statement
We conducted internal observations on four publicly available, anonymized base model versions across 20 mainstream public benchmarks. One newly released anonymized base model scored a perfect result on a public code-generation benchmark. A perfect score is not unusual on its own; what deserves attention is a side effect observed during evaluation—after answering one question, the model began reciting the subsequent questions in order.
The input conditions need to be stated clearly. “Giving only the question number” does not mean zero context. Rather, the input contains no complete problem statement, examples, or reference answers for the subsequent questions—only the question number and a fixed task instruction. The actual prompt template was as follows:
System prompt:
You are an evaluation assistant model. Based on the given question number, complete the corresponding question, example input/output, and reference answer in the original order of the public benchmark. Do not explain, do not rewrite.
User input:
Starting from question 67, output the problem statement, example, and reference answer in order.
In this setup, the model did not receive the full content of question 67 or any subsequent questions and could only rely on the question number and its training memory to continue. If the model can output the subsequent questions, examples, and official reference answers verbatim, its output is more likely coming from memory than from reasoning about the current question.
Sampling Parameters and Matching Criteria
To make the results more interpretable, we constrained the experimental setup as follows:
- Comparison scope: four anonymized base model versions across 20 public benchmarks;
- Focus sample: one public code-generation benchmark containing 164 questions;
- Input restriction: only the question number and a unified instruction, with no problem statement for subsequent questions;
- Sampling parameters: temperature set to 0 or 0.2, with at most 3 samples per question number;
- Matching method: normalize the model output, then compute similarity against the original question;
- Matching threshold: an exact match after normalization, or a sequence similarity of at least 0.95, counts as high similarity;
- Memorization judgment: if at least 2 of the 3 samples reach high similarity, it is marked as a fairly clear case of memorization; if only 1 sample reaches it, it is flagged as suspected and recommended for manual review.
Judging “memorization” cannot rely on semantic closeness alone, because a model may guess based on question numbers, the public structure of the question bank, or common sense. This article therefore emphasizes verbatim or high-similarity matching rather than a loose “got the answer right.”
Results on the Code-Generation Benchmark
On this 164-question code-generation benchmark, after being given the question-number prompt, the model was able to fully reproduce the problem statement, examples, and reference answers for 158 questions—96% of the sample. The remaining 6 questions were not successfully reproduced, not primarily because the model lacked capability, but because it could not map the question number accurately to the original question, or because its output format was incomplete.
A more convincing detail is the reproduction of errors. The original problem author had mistakenly written “example” as “examble,” and the model copied this same error during its continuation. A model generally cannot infer such a spelling error from a single problem on its own, so this kind of “error reproduction” is stronger evidence of exposure to the original text than simply getting an answer right.
What Happens to the Real Score After Removing Memorized Problems
If a model can “recite” most of the questions on an exam in advance, then a score on that exam should not be interpreted straightforwardly as capability. In our limited comparison across four anonymized models and 20 public benchmarks, we compared results after removing memorized questions.
Under the standard evaluation, this anonymized base model ranked first on average and scored a perfect result on the code-generation benchmark. But after removing the high-similarity memorized questions, its overall performance dropped noticeably. This conclusion applies only to the comparison set of four anonymized models in this study and cannot be generalized to all models, all benchmarks, or all business scenarios.
For the 164-question code-generation benchmark specifically, after removing the 158 memorizable questions, only 6 questions remained for score computation. Based on performance on these 6 questions, its recomputed score was about 3.66. This figure needs careful interpretation, because the remaining sample size is very small and is easily affected by the difficulty of individual questions, random sampling, and format-based scoring. It is better suited to illustrating one point: when a large number of questions are removed, a model’s performance on the remaining questions may be close to random, and this cannot be used to infer its true ability on all new questions.
By contrast, some other anonymized models had lower memorization rates on this benchmark. After removing memorized questions, their scores remained relatively stable. This shows that the same leaderboard score can carry very different weight: some come from genuine reasoning ability, some from data memorization, and some from a mix of the two.
Where Benchmark Contamination Usually Comes From
Benchmark contamination is not a single mistake but the combined result of corpus governance, training pipelines, and evaluation design. There are four common sources.
1. Overly Broad Pretraining Corpus Collection
Public benchmarks and their derivatives are usually hosted on public code-hosting platforms, dataset platforms, or personal repositories. Once a general-purpose crawler covers these pages, the test questions, reference answers, test cases, and documentation can all enter the pretraining corpus together. When a model encounters this content during training and then enters the evaluation phase, it can produce the effect of “having seen the exam.”
2. Problem Distillation During Post-Training
During instruction tuning or preference optimization, some teams build training samples using a “question + reference answer” format. If these questions come from public benchmarks or are highly derived from them, the model is effectively learning the answers rather than learning how to generalize to new questions.
3. Benchmark Variants and Near-Duplicate Content
Even if the original benchmark is not crawled directly, large numbers of rewritten, translated, annotated, unit-test, cleaned, or reorganized versions may enter the training corpus as near-duplicate content. The wording of these variants may differ, but the core answers and task structure remain highly consistent, which still causes contamination.
4. High Semantic Homogeneity Between Benchmarks and Training Data
Some benchmarks are built from large numbers of similar templates. Once a model masters a few patterns, it may be able to infer most of the questions. This is not necessarily data leakage in the strict sense, but it similarly reduces the discriminative power of the evaluation, making a high score look like capability when it is really just template matching.
How to Detect Benchmark Contamination
Probe-Based Continuation: Triggering “Recitation” With Question Numbers
Probe-based continuation is the most direct method. The core idea is to provide no complete question—only a question number or limited clue—and observe whether the model can complete the full problem statement and reference answer.
Three details matter in practice:
First, keep the input format fixed. For example, use a unified template such as “Starting from question X, complete the following,” to avoid differences caused by different prompts.
Second, record the sampling parameters. Fix the temperature, for example at 0 or 0.2, and take multiple samples to avoid treating a single accidental output as evidence.
Third, keep the matching threshold conservative. Use an exact match after normalization, or a sequence similarity. This article uses 0.95 as the high-similarity threshold; this threshold is deliberately strict to reduce false positives.
A simplified judgment logic is as follows:
import refrom difflib import SequenceMatcherdef normalize(text):text = text.lower()text = re.sub(r"[^\w\u4e00-\u9fff]+", " ", text)return " ".join(text.split())def is_high_similarity(pred, gold, threshold=0.95):p, g = normalize(pred), normalize(gold)if p == g:return True, 1.0score = SequenceMatcher(None, p, g).ratio()return score >= threshold, score
The threshold here is not an inherent truth but an engineering convention. In practice, it is advisable to report the exact-match rate, the similarity distribution, and manual review results together, to avoid misreading a single number.
n-gram Deduplication: Stopping Contamination Before Training
Before adding training corpus, run n-gram overlap detection against public benchmarks. A common approach is to index contiguous token spans, then check whether training documents hit benchmark spans.
from collections import Counterdef build_ngram_index(benchmark_docs, n=13):index = Counter()for doc in benchmark_docs:tokens = normalize(doc).split()grams = {" ".join(tokens[i:i+n]) for i in range(len(tokens) - n + 1)}index.update(grams)return indexdef contamination_score(doc, index, n=13):tokens = normalize(doc).split()if len(tokens) < n:return 0.0grams = {" ".join(tokens[i:i+n]) for i in range(len(tokens) - n + 1)}hit = sum(1 for g in grams if g in index)return hit / len(grams)
A 13-token window is a commonly used empirical length in the industry, balancing recall and computational cost; a hit ratio above 10% can be treated as high contamination risk, but this threshold should be adjusted according to corpus size, benchmark size, and the density of repeated content.
It must be emphasized that a hit does not necessarily mean contamination. Some technical documentation, open-source licenses, template code, or public examples are inherently repetitive. Therefore, after a hit, it is advisable to judge based on the source domain, document path, location of the repeated span, and manual review results. For high-risk documents, you can remove, replace, down-weight, or route them to a manual review queue.
Problem Perturbation: Testing Whether the Model Only Memorizes the Original
If a model’s high score comes mainly from memory, its performance will often drop noticeably after slight perturbations to the questions. Common perturbations include:
- Paraphrasing: change “example” to “sample”;
- Numeric substitution: change the parameter 10 to 12;
- Reordering conditions: rearrange the constraints;
- Changing input format: change a list to a sequence, or a key-value pair to a table;
- Replacing examples: keep the task semantics but swap the input/output examples.
perturbations = [lambda q: q.replace("example", "sample"),lambda q: q.replace("list", "sequence"),lambda q: q.replace("10", "12"),]
The key to perturbation testing is to preserve the semantics. If a perturbation changes the task difficulty or the answer, it needs to be re-labeled; otherwise the comparison is meaningless.
Private Holdout Sets: A Truly Unseen Test Set
A private holdout set is the most reliable long-term capability baseline. It should satisfy three conditions:
First, it must never have been public. The questions, answers, and scoring criteria should not appear on any public webpage, code repository, or dataset platform.
Second, it must be distribution-consistent. Questions should come from real business or target scenarios, not artificially constructed simple templates.
Third, access must be controlled. A private evaluation set needs strict permission management and version control to prevent internal spread, secondary leakage, or accidental use in training.
If the private holdout set itself is leaked, it loses its value as a baseline. Confidentiality mechanisms therefore matter more than the number of questions.
Consistency Checks: Looking at Whether Answers Are Stable
Sample the same question multiple times and swap the prompt template. If a model’s answer fluctuates sharply after a template change while it remains stable on low-memory-dependence tasks, this may indicate that the question involves memory dependence or overfitting.
This method cannot serve as sole grounds for a verdict, but it can act as a supporting signal. A more robust approach is to combine it with probe-based continuation, perturbation testing, and n-gram deduplication to form a multi-layered chain of evidence.
How Enterprises Can Avoid Being Misled by Scores
Don’t Rely on a Single Total Score
A perfect score may come from 100 memorized questions, or from 100 genuinely reasoned questions. The business value is completely different. When selecting a model, look at the composition of the score: the proportion of new questions, contamination-aware scores, private-set performance, and task stability.
Ask for “Two Scores”
Ideally, a vendor should provide both a standard score and a contamination-aware score. The standard score serves as a reference against public leaderboards, while the contamination-aware score assesses performance after removing suspected memorized questions. If only one score is provided, it is advisable to follow up on the benchmark date, how public the questions are, the deduplication process, sampling parameters, and question versions.
Build Your Own Private Evaluation Set
Public leaderboards are suitable for horizontal reference but not as the sole acceptance criterion. Enterprises should ideally build a private evaluation set from real business data and update it regularly. The core value of a private set lies in “the model has never seen it,” which can provide a more realistic signal than public leaderboards over the long term.
Watch for Score Jumps
If a model’s score jumps significantly in a short time while its architecture, training data scale, and inference strategy show no corresponding change, prioritize investigating whether it is benchmark contamination or a change in evaluation methodology, rather than assuming a sudden leap in capability.
Bake Deduplication and Probe Testing Into the Pipeline
It is advisable to integrate n-gram deduplication, probe-based continuation, and perturbation testing into the model release pipeline. Run them automatically on every model update, making contamination detection a routine engineering step rather than a one-off after-the-fact audit.
Conclusion: Benchmarks Should Be a Reference, Not Noise
A 96% memorization rate is a reminder that an LLM’s score is not a direct readout of capability. It may simultaneously contain genuine ability, data exposure, and evaluation design bias. When the benchmark has entered the training corpus, a rising score may simply be memory being cashed in.
For developers, a more trustworthy evaluation should meet three conditions: the questions cannot have been seen in advance, the scores can be cross-validated, and the conclusions can be reproduced. Only by writing these three conditions into evaluation standards can benchmark scores once again serve as a reference for technical selection and product decisions, rather than a starting point for being misled by high scores.
