How to Verify LLM Benchmark Screenshots and Build a Reproducible Evaluation Pipeline

    A redacted benchmark screenshot went viral in a developer community, with two models rumored to sit only 0.1 percentage points apart. This article explains why pixel-level image "restoration," small-sample percentage conversions, and second-hand paraphrases cannot establish a trustworthy result, and shows how to check masked screenshots, recognize synthetic images, reason about small-sample percentages, and assess paraphrased evidence. It then lays out a reproducible evaluation system: the fields to record per run, sandboxing and contamination-check practices, the pass@1 / pass@k / pass^k metrics, and cost/latency analysis, so developers can turn rumors into verifiable facts and make sound model-selection decisions.

    How to Verify LLM Benchmark Screenshots and Build a Reproducible Evaluation Pipeline

    A redacted benchmark screenshot went viral in a developer community, with two models rumored to sit only 0.1 percentage points apart. The discussion quickly split into several camps: some tried to recover the obscured model name through font comparison, others converted 60-point scores into percentages and treated them as an exact ranking, and still others stitched a paraphrased remark about “capacity expansion” and “rotating launches” into a rumor about a new gray-release version. The same model produced scores more than 10 percentage points apart at different times; everyone claimed to be “testing,” yet the disagreements were not about the model itself but about the evidence chain. This article is written for developers and evaluation engineers. It provides methods for checking redacted benchmark screenshots, identifying synthetic images, judging small-sample percentages, and assessing paraphrased evidence, and it presents a reproducible evaluation system that helps you extract verifiable facts from rumors and reach conclusions you can actually use for model selection.

    1. Post-mortem: How One Screenshot Split Into Three “Versions”

    Recently, a screenshot of a third-party coding benchmark circulated in a developer community. To push back against the claim that “open-weight models can no longer keep up with closed-source frontier models on complex engineering tasks,” the benchmark’s author posted several records, three of which were redacted. The username prefixes and scores were hidden; only the suffixes “Pro,” “Med,” and “Small” remained visible. No model was named, but the wording of the reply led many developers to believe the tested subject might be a new series about to be released by a model vendor.

    The “restoration” that followed unfolded in roughly three steps, each corresponding to a typical kind of misjudgment.

    The first step was pixel comparison. Someone reassembled a placeholder name using a similar font, applied Gaussian blur, and compared it against the original pixel by pixel, claiming the residual was below 1/255. A clearer version of the leaderboard that later circulated gave specific names, and another reading appeared in the comments. The three-tier suffix was always legible, but the candidate prefixes kept multiplying. This kind of operation is essentially generating a synthetic image and using its similarity to the original to infer the obscured content. But similarity is not the same as truth.

    The second step was score conversion. The original was scored out of 60 across 6 coding tasks. The community converted it to a percentage on its own. Because the 6 tasks were equally weighted and the total came from the number of tasks passed, the only legal percentages were 0%, 16.7%, 33.3%, 50.0%, 66.7%, 83.3%, and 100%. Therefore 59.4% was not a legal value; likewise, the 70.0% and 70.1% that appeared in another supposedly “genuine” image could not have been real conversions under a 6-task scheme. If the scoring system includes partial credit, it must state the scorer version, the field mapping, and the partial-credit rules; otherwise these decimals can only be treated as paraphrased estimates, not exact readings. The supposed “overtake” existed only in the paraphrase—the corresponding row in the original image was still redacted.

    The third step was clue stitching. When asked about capacity-expansion progress, the developer of a coding tool mentioned that the previously capacity-constrained service was now running smoothly, and “the next turn goes to a certain vendor.” Pressed on whether the lightweight tier or the full version would go first, he replied with one word: “full.” A sentence that only discussed capacity expansion and never mentioned a version number, when stitched to the leaderboard screenshot, turned into inferences like “the next-generation mid-tier is about to enter gray testing” and “last week’s old post already foreshadowed it.” Such inferences seem to have a timeline, but they lack a traceable, reproducible, and falsifiable evidence chain.

    These three steps correspond to image forensics, numerical conversion, and semantic inference, respectively. Each looks “grounded” on its own, yet strung together they form an unfalsifiable chain of rumor. When facing the task of verifying a redacted benchmark screenshot, the real work is not to pick a side quickly but to break down the evidence chain and clarify which information can be trusted and which can only be held as doubtful.

    2. The Technical Roots of the Three Distortions

    2.1 Visual restoration is not information restoration

    Comparing fonts and blur parameters on a redacted region is a standard idea in image forensics. But treating it as a conclusion requires clearing at least three hurdles.

    First, rendering chains differ. How the original image was produced is usually unknown: UI rendering, export tools, and a social platform’s second compression all vary. Quantization error, chroma subsampling, and scaling/resampling all alter edge pixels. So a “0-or-1 per-pixel residual” only shows that a simulation is close enough to some copy; it does not prove it is the original content.

    Second, the candidate space is hard to close. To make the comparison meaningful, you need a complete candidate list—including the full naming scheme, date suffixes, and internal codenames. Candidates outside the list can always produce an equally good match, so a recovered name is at best “one of the most likely candidates.”

    Third, the minimum-change principle also applies. The person who produced the clearer version, and the person who forwarded it, may hold preconceptions. The more smoothly a synthetic image is produced, the more easily it is taken as “restoration evidence.”

    The illustrative code below shows how strict the conditions for rendering, blurring, quantization, and residual comparison really are:

    1. import numpy as np
    2. from PIL import Image, ImageDraw, ImageFilter, ImageFont
    3. def render_candidate(text, font_path, box, font_size=14, blur_radius=0.6):
    4. """Render candidate text in the given region, simulating the screenshot pipeline."""
    5. canvas = Image.new("L", box[2:], 255)
    6. draw = ImageDraw.Draw(canvas)
    7. draw.text((2, 1), text, font=ImageFont.truetype(font_path, font_size), fill=0)
    8. canvas = canvas.filter(ImageFilter.GaussianBlur(radius=blur_radius))
    9. return np.asarray(canvas, dtype=np.float32) / 255.0
    10. def pixel_diff(ref_crop, candidate):
    11. """Return the mean absolute residual and fraction of differing pixels; ref_crop is a crop of the same region."""
    12. ref = np.asarray(ref_crop.convert("L"), dtype=np.float32) / 255.0
    13. if ref.shape != candidate.shape:
    14. raise ValueError("Size mismatch; align the canvas before comparing")
    15. diff = np.abs(ref - candidate)
    16. return float(diff.mean()), float((diff > 1 / 255).mean())
    17. # A and B are two candidate spellings of equal length with similar glyphs
    18. # a_score = pixel_diff(crop, render_candidate("A", font_path, box))
    19. # b_score = pixel_diff(crop, render_candidate("B", font_path, box))

    What matters is not how small the residual number is, but the comparison logic: “A wins” is only worth discussing when candidate B’s residual is clearly larger than candidate A’s, and that difference reproduces stably across multiple candidates.

    A safer approach is to trust only high-contrast, large-font, non-recompressed parts—such as suffixes and version numbers—and to explicitly mark the prefix, date, and scores as “unknown” or “unverified.”

    2.2 Small-sample percentages have almost no resolving power

    Converting raw scores into a percentage does not create resolution out of thin air. Suppose there are N tasks, each with the same maximum score and no partial credit; then the total score S is just the number of tasks passed, K. Percentage = K / N, and adjacent values differ by exactly 100/N percentage points:

    Tasks N Gap between adjacent values Available values between 59% and 70%
    6 16.67 pp 66.7
    20 5.00 pp 60.0, 65.0, 70.0
    50 2.00 pp 60.0, 62.0, 64.0, 66.0, 68.0, 70.0
    200 0.50 pp ~20 values

    This data shows that with only 6 tasks, 59.4%, 70.0%, and 70.1% are not legal percentage values. If a leaderboard claims precision to one decimal place, readers should first ask whether the scorer allows partial credit, whether it applies secondary standardization, and whether there are hidden tasks or weight changes. Without this information, a supposed “0.1 percentage point” lead cannot form a credible ranking.

    You also need to distinguish a “single-run score” from “capability level.” Treating the pass rate p as the model’s true level over a task distribution, a single-run score X is the sum of n Bernoulli trials. With n=6 and p=0.6, for example, each evaluation has a 27.6% chance of scoring 50.0% (3/6) and a 31.1% chance of scoring 66.7% (4/6). The gap between the two is a full 16.7 percentage points, so swapping ranks across two runs is completely normal.

    Illustrative code:

    1. from math import comb
    2. def possible_scores(n):
    3. """All legal percentage values for N equally weighted tasks."""
    4. return [round(100 * k / n, 1) for k in range(n + 1)]
    5. def binom_pmf(k, n, p):
    6. return comb(n, k) * p ** k * (1 - p) ** (n - k)
    7. if __name__ == "__main__":
    8. print(possible_scores(6))
    9. for k in (3, 4):
    10. print(k, round(100 * k / 6, 1), round(binom_pmf(k, 6, 0.6), 3))
    11. # 3 50.0 0.276 4 66.7 0.311

    Two takeaways:

    First, a more reasonable way to read the leaderboard is as coarse-grained capability tiers—something like “around sixty percent”—rather than a ranking precise to one decimal place. Rankings should be compared under the same tasks, the same parameters, and the same scorer, with repeated runs and confidence intervals; when intervals overlap, the rank difference itself has no statistical meaning.

    Second, watch for selection effects beyond the screenshot. Which comment the author chose to respond to, which image was shown, and which rows were obscured all change what the reader sees. The danger of small-sample percentages is that they look like precise measurement while actually being a random draw from discrete values.

    2.3 Second-hand paraphrasing is not verifiable evidence

    The three statements “it ran well after capacity expansion,” “the next turn goes to a certain vendor,” and “full” are not enough to support the conclusion that “a next-generation tier of some model is about to enter gray release.” A provider’s capacity arrangements only reflect resource scheduling, not the existence of a new model; the choice between the lightweight tier and the full version also says nothing about which generation. Treating an older post with a similar timeline as a “preview” is a classic confirmation bias: people only notice clues that match their expectations.

    When judging a rumor, you can apply a standard similar to intelligence analysis and check whether it satisfies all of the following:

    • Traceable: Is the original source public, and can the full context be traced?
    • Reproducible: Can a third party reach the same result with the same method?
    • Falsifiable: What kind of evidence would refute it?

    Screenshot residue, stitched paraphrases, and after-the-fact correlation all fail these tests. The most reasonable stance is to mark it “unverified” rather than to fill in the missing link for it.

    3. Making Conclusions Credible: From Screenshot to Evaluation Record

    The purpose of a benchmark is to set a rough priority order during selection; it is not responsible for answering “will it work in my business?” Rather than chasing the next screenshot, it is better to make evaluation into a reproducible evidence chain.

    3.1 What to record for each evaluation

    A defensible evaluation record should contain at least the following fields. You can fix them into a unified result structure to make cross-batch comparison easy:

    • Model identifier: precise to a specific version snapshot, not a series name;
    • Sampling parameters: temperature, top_p, max output length, inference budget, etc.;
    • Dataset and scoring standard version: task content or hash, scorer version, data-cleaning rules;
    • Prompt version: to avoid score drift caused by prompt changes;
    • Failure-sample labels: timeout, syntax error, test failure, invalid output, etc., to aid review;
    • Execution environment: resource specs, timeouts, dependency image version;
    • Raw output and error messages: to enable re-checking and appeals.
    1. {
    2. "run_id": "eval-20260827-001",
    3. "model_id": "model-a@snapshot-20260820",
    4. "prompt_version": "prompt-v1.2",
    5. "sampling": {"temperature": 0.2, "top_p": 1.0, "max_output_tokens": 8192},
    6. "scorer": "pytest-wrapper@1.4.0",
    7. "dataset": {
    8. "name": "internal-task-v3",
    9. "version": "2026-08-01",
    10. "size": 240,
    11. "data_cleaning_rules": "dedup_13gram",
    12. "leak_check": "public_ngram_overlap_lt_0.08"
    13. },
    14. "runtime": {
    15. "cpu": "8c",
    16. "memory_gb": "32g",
    17. "timeout_s": 600,
    18. "image": "eval-runner@sha256:..."
    19. },
    20. "repeats": 3,
    21. "failure_labels": ["timeout", "compile_error", "assert_failed", "no_code_block"],
    22. "metrics": {
    23. "pass_at_1": 0.61,
    24. "ci95": [0.55, 0.67],
    25. "median_latency_s": 34.2,
    26. "tokens_per_task": 18500
    27. }
    28. }

    Two of these are the most easily overlooked: complete output and task-prompt protection. A screenshot only carries the conclusion, whereas an evaluation record must keep the raw replies, execution logs, and failure reasons; otherwise disagreements cannot be reviewed. Once such a task set leaks, it will be optimized against specifically, and subsequent scores will lose their reference value.

    3.2 Reproducible engineering practices

    To turn evaluation from a “one-off demo” into “auditable evidence,” you need at least fixed parameters, contamination checks, sandbox execution, automated judging, and result retention.

    First, fix and align parameters. Use the same sampling parameters for all candidate models. When vendors’ defaults differ, state this clearly in the report. Only models of comparable cost make a meaningful capability comparison.

    Second, check for test-set contamination. Use n-gram overlap to filter out tasks highly similar to public question banks, and rotate tasks regularly. Common approaches combine 13-grams, longest common substring, semantic similarity, and manual review.

    Third, sandbox execution. Model-generated code should not run directly on a bare machine. Production should use isolated containers and restrict network, filesystem, and resources:

    1. docker run --rm \
    2. --network none \
    3. --read-only \
    4. --cpus 1 \
    5. --memory 2g \
    6. --pids-limit 256 \
    7. --tmpfs /tmp:rw,noexec,nosuid,size=64m \
    8. --user nobody \
    9. eval-runner:sha256-example

    The configuration above is illustrative: network disabled, read-only root filesystem, limited CPU and memory, limited process count, limited temp-directory size, and a non-root user where possible. Real systems should also add timeouts, audit logs, least-privilege images, and dependency allowlists.

    Fourth, automated judging. Prefer executable signals such as unit tests and static checks; if you use a model as a judge, fix the judging prompt and order, and do AB/BA swaps to cancel out position bias.

    Fifth, result retention. In a cloud evaluation pipeline, you can harden this flow into automation: use object storage to host tasks and evaluation records, use a container platform to carry the isolated execution environment, and use logging plus monitoring/alerting to track metric fluctuations, so every conclusion can be traced back to raw data.

    3.3 A minimal skeleton for task-level evaluation

    The code below shows the basic structure of task-level evaluation, including prompt construction, code extraction, test execution, retries, and aggregation. It is suitable as a prototype and does not represent production-grade isolation.

    1. import json
    2. import subprocess
    3. import tempfile
    4. from pathlib import Path
    5. from statistics import mean
    6. from concurrent.futures import ThreadPoolExecutor
    7. def build_prompt(case):
    8. return (
    9. "Fix the failing case below and output only the complete target file:\n"
    10. f"Case:\n{case['spec']}\n\nCurrent implementation:\n{case['code']}\n"
    11. )
    12. def extract_code(text, filename="solution.py"):
    13. """Extract the code block from a reply; treat a reply without a code block as invalid output."""
    14. if "```" not in text:
    15. return None
    16. return text.split("```", 2)[1].split("\n", 1)[1].rsplit("```", 1)[0]
    17. def run_tests(code, case, timeout_s=60):
    18. """Illustrative execution in a local temp directory; production should use container isolation."""
    19. with tempfile.TemporaryDirectory() as tmp:
    20. root = Path(tmp)
    21. (root / "solution.py").write_text(code, encoding="utf-8")
    22. for name, content in case.get("fixtures", {}).items():
    23. (root / name).write_text(content, encoding="utf-8")
    24. try:
    25. proc = subprocess.run(
    26. ["python", "-m", "pytest", "-q"],
    27. cwd=root, capture_output=True, text=True, timeout=timeout_s,
    28. )
    29. return proc.returncode == 0, 0.0, (proc.stderr or proc.stdout)[-800:]
    30. except subprocess.TimeoutExpired:
    31. return False, float(timeout_s), "timeout"
    32. def call_with_retry(client, model, prompt, params, max_retries=3):
    33. """Retry on transient failures (rate limiting, timeouts) and record the retry count."""
    34. last_err = None
    35. for attempt in range(1, max_retries + 1):
    36. try:
    37. resp = client.chat.completions.create(
    38. model=model, messages=[{"role": "user", "content": prompt}], **params
    39. )
    40. return resp.choices[0].message.content, attempt - 1
    41. except Exception as exc:
    42. last_err = exc
    43. raise RuntimeError(f"Call failed: {last_err}")
    44. def evaluate(client, model, cases, params, repeats=3, workers=8):
    45. def one(job):
    46. case, rep = job
    47. try:
    48. text, retries = call_with_retry(client, model, build_prompt(case), params)
    49. code = extract_code(text)
    50. ok, elapsed, err = False, 0.0, "no_code_block"
    51. if code:
    52. ok, elapsed, err = run_tests(code, case)
    53. return {
    54. "case": case["id"],
    55. "rep": rep,
    56. "passed": int(ok),
    57. "retries": retries,
    58. "error": err,
    59. }
    60. except Exception as exc:
    61. return {"case": case["id"], "rep": rep, "passed": 0, "error": repr(exc)}
    62. jobs = [(c, r) for c in cases for r in range(repeats)]
    63. with ThreadPoolExecutor(max_workers=workers) as pool:
    64. records = list(pool.map(one, jobs))
    65. return {
    66. "model_id": model,
    67. "repeats": repeats,
    68. "pass_at_1": mean(r["passed"] for r in records),
    69. "records": records,
    70. }

    A small reminder: calls involving external interfaces must be placed inside the retry logic. Transient failures such as rate limiting and timeouts, mixed together with genuine failures, will also pollute the metrics. In production, the temp-directory execution should be replaced with container execution, with boundaries set for network, filesystem, and permissions.

    3.4 Three metrics you must be able to compute

    • pass@1: The expected pass rate when submitting once per task, aggregated over “number of samples × number of repetitions.”
    • pass@k: The fraction of tasks where at least one of k submissions passes, used to measure “can it be done after trial and error.”
    • pass^k: The fraction where all k runs pass, used to measure stability—especially important for automated pipelines with high reliability requirements.

    The formulas are as follows; the illustrative code can be used directly for evaluation aggregation:

    1. from math import comb
    2. def pass_at_k(n, c, k):
    3. """Unbiased estimate of the probability of at least one success given c successes in n samples."""
    4. if n - c < k:
    5. return 1.0
    6. return 1.0 - comb(n - c, k) / comb(n, k)
    7. def pass_hat_k(n, c, k):
    8. """Unbiased estimate of the probability that all k samples succeed."""
    9. return comb(c, k) / comb(n, k)
    1. from collections import defaultdict
    2. from statistics import fmean
    3. def summarize(records, k=3):
    4. """Group by case and compute pass@1, pass@k, and pass^k."""
    5. groups = defaultdict(list)
    6. for r in records:
    7. groups[r["case"]].append(r["passed"])
    8. p1 = fmean(fmean(v) for v in groups.values())
    9. pk = fmean(pass_at_k(len(v), sum(v), min(k, len(v))) for v in groups.values())
    10. pkk = fmean(pass_hat_k(len(v), sum(v), min(k, len(v))) for v in groups.values())
    11. return {
    12. "pass@1": round(p1, 3),
    13. f"pass@{k}": round(pk, 3),
    14. f"pass^{k}": round(pkk, 3),
    15. }

    An output example might be:

    1. {"pass@1": 0.612, "pass@3": 0.789, "pass^3": 0.543}

    These numbers mean: the average single-run pass rate is about 61.2%; when each task allows up to 3 attempts, about 78.9% of tasks succeed at least once; but only 54.3% of tasks pass on all 3 attempts. This shows the model has a certain ceiling for trial and error, but its stability is still lacking.

    Note that pass@1, pass@k, and pass^k cannot be directly compared across different tasks. Tasks differ in difficulty, length, and dependency complexity; an increase in pass@1 may come from tasks becoming easier, and an increase in pass@k may also come from allowing more attempts. When comparing across models, prefer the same question bank and the same task set, and do paired analysis. If you want to compare different question banks, you must first prove their difficulty distributions are consistent; otherwise conclusions are easily distorted.

    1. def ngram_overlap(text, reference, n=13):
    2. """Return the fraction of n-grams in text that overlap with reference, for initial contamination screening."""
    3. grams = lambda s: {s[i:i + n] for i in range(len(s) - n + 1)}
    4. g1, g2 = grams(text), grams(reference)
    5. return len(g1 & g2) / len(g1) if g1 else 0.0

    On tasks like code generation, where there are objective acceptance criteria, a single-run pass rate is often insufficient to reflect real usability. A model with a similar single-run pass rate that only stabilizes after multiple attempts is suited to assist with human-in-the-loop correction; a model that can get it right on the first try again and again is the one suitable for an unattended pipeline.

    3.5 Cost, latency, and reward hacking

    On cost and latency, multiply the average output length per task by the unit price to estimate total spend. When billed by output length, verbose explanations directly inflate the bill. Then use P50 and P95 latency to check whether the pipeline can meet its SLA. When inference is delivered as a service, you can use message queues to smooth out traffic peaks and troughs and caches to reuse high-frequency requests, easing capacity pressure.

    On reward hacking, when only “tests passing” is measured, the model may learn tricks such as deleting assertions, hardcoding expected values, or renaming test functions. Adding assertion-deletion detection, differential testing against a reference implementation, and manual spot checks keeps the metric close to real quality.

    On pairwise comparison and significance, when comparing two candidates, prefer a same-task paired comparison—both run the same set of tasks—then bootstrap-resample the “differences” and observe whether the 95% interval crosses 0. If it crosses 0, it should be judged as “no measurable difference.”

    1. import numpy as np
    2. def paired_bootstrap(a, b, n_boot=10000, seed=0):
    3. """Paired bootstrap: resample the per-task score differences and return the 95% confidence interval."""
    4. rng = np.random.default_rng(seed)
    5. a, b = np.asarray(a, float), np.asarray(b, float)
    6. idx = rng.integers(0, len(a), size=(n_boot, len(a)))
    7. diffs = (a - b)[idx].mean(axis=1)
    8. return float(diffs.mean()), np.quantile(diffs, [0.025, 0.975])

    4. A General Strategy for Judging Model Tiers

    Pro/Med/Small-style tiering, and the choice between a lightweight tier and a full version, are common ways the industry organizes products. When choosing a tier, you do not need to rely on any leaked screenshot; instead, decide by task layering:

    • Hard tasks: complex engineering fixes, cross-module refactoring, long-context understanding. Use pass@k to measure the ceiling while watching cost.
    • Mid-frequency tasks: routine feature development, code review. Use pass@1 and P95 latency to measure everyday experience.
    • High-frequency tasks: completion, formatting, simple queries. Measure with throughput, unit price, and stability.

    The point of tiering is the trade-off between cost and capability, not just the score. If a full-version model leads by a tier on key tasks with stability, it may have lower total cost even at double the unit price, because it reduces rework; if a lightweight-tier model has a comparable single-run pass rate but a clearly lower rate of getting it right on the first try, putting it in an unattended pipeline will bring extra manual-review burden.

    In real business, the worst thing is to replace the whole decision with a single screenshot. A screenshot can point out a direction to watch, but it cannot prove a model is definitely usable in your scenario. Model selection should return to reproducible evaluation: fixed parameters, isolated execution, automated judging, multiple repetitions, interval estimation, and each round’s results settled into auditable records.

    5. Conclusion: The Endpoint of Evaluation Is a Reproducible Record

    Back to the screenshot at the beginning. The reasonable conclusion is: the existing evidence can only support that “a next-generation model series from a certain vendor may be in small-scale internal testing,” and even this judgment remains to be confirmed. Pixel residuals do not explain the original content, small-sample percentages do not explain the size of the lead, and paraphrases do not explain the release schedule.

    For developers, rather than waiting for the next screenshot, it is better to immediately turn evaluation into an engineering capability: fixed parameters, sandbox execution, automated judging, multiple repetitions, interval estimation, and each round’s results settled into a reproducible evaluation record. Whoever owns an evaluation set closest to their own business owns the ultimate basis for judging the value of new models.

    To put it into action, remember three points:

    1. Stay skeptical of rumors, and accept only traceable, reproducible, and falsifiable evidence.
    2. Do not treat a lead beyond the decimal point as a conclusion; give a confidence interval and judge whether the difference is significant.
    3. Use layered evaluation and stable pass rates to guide model selection and tier allocation.

    This is both the answer to benchmark chaos and the long-term foundation for enterprises to deploy large models stably.