Benchmark Screenshots Can't Be Trusted: A Practical Guide to LLM Evaluation and Model Selection

    A benchmark screenshot circulating in a tech community is often the weakest kind of evidence for deciding whether a new model can handle critical tasks. Using a recent viral example of a redacted leaderboard screenshot as a starting point, this article breaks down the evaluation methodology behind such claims: why converted scores hide information gaps, why a redacted model name still leaves the process verifiable, and why pixel comparison and AI image recognition can't establish facts. It then explains what benchmark scores can and can't tell you, how to route across Pro/Med/Small model tiers with cascade dispatch, how to run a five-stage gray release before onboarding a new model, and how to build continuous evaluation into a pipeline. Developers will learn to judge new models with their own real tasks and a reproducible process instead of relying on viral screenshots.

    Introduction: A Redacted Screenshot—What Can It Really Prove?

    A recent episode in the tech community illustrates the problem. Someone claimed that a new generation of an open-weight model might have entered limited internal testing. The evidence: a leaderboard screenshot posted by an open-source community maintainer while responding to someone else’s discussion, in which the model names and scores of three rows were redacted. The publicly recorded existing version had only scored 45.8%, while the three redacted rows—compiled by the community—ranked 2nd, 5th, and 9th, with the suffixes Pro, Med, and Small respectively.

    After the screenshot spread, two versions of the same “suspected new model” score emerged:

    • An enthusiastic user estimated from the redacted image that it scored 59.4%, 1.6 percentage points higher than the current strongest closed-source frontier model.
    • A later “clearer version” changed the score to 70.1%, only 0.1 percentage points above the leader.

    The identification process that followed was equally telling. Some retyped candidate names in a similar font and blurred them, then compared pixel by pixel against the original and claimed a residual below 1/255; others used AI image recognition and inferred that the first three letters were “GLM.” The person doing the pixel comparison wrote “GLM-Name,” the circulating clear version wrote “GLM-Nano,” and the comments section guessed “GLM-Next.” Meanwhile, a developer relayed that after a certain model scaled up, “next is GLM’s turn,” and when asked whether to do the small tier first or the full version, the other party replied with just one word—“full”—which immediately sparked speculation that “the next-gen version is about to undergo gray testing.”

    Setting aside whether the rumor is true, this incident reflects a common dilemma in engineering selection: when we want to use a single benchmark to judge whether a new model can take over a critical task, the evidence in hand is often just a screenshot, a relay, and a few seconds of visual inspection. This article doesn’t address the truth of any specific product. Instead, it dissects the evaluation methodology behind this episode to help developers build a reproducible, actionable process for judging similar news.

    Score Conversion: The Three Information Gaps Behind 59.4%

    A seemingly precise percentage on a leaderboard is often just a conversion of “raw score ÷ full marks.” Let’s do a quick calculation. This coding benchmark has 6 problems totaling 60 points. If each problem is scored only in whole numbers, there are just 61 possible raw scores, and after converting to percentages, the gap between adjacent levels is about 1.67 percentage points. The results are shown below.

    Raw score (/60) Percentage Gap from 70.0%
    35 58.3% -11.7
    36 60.0% -10.0
    37 61.7% -8.3
    42 70.0% 0
    43 71.7% +1.7

    There is no reading like 59.4% in this table. The Pro at 59.4%, Med at 57.1%, and Small at 54.0% in the community-compiled image cannot be derived directly from whole-number scores. This means the raw scoring likely involves finer rules such as partial credit, scoring by pass ratio across test cases, or weighted aggregation; and if these numbers were only “estimated” by users reading a redacted image, the error could be even larger. The 0.1 percentage point gap between “70.1%” and the leader’s “70.0%” can barely be expressed on the scale of a 6-problem score. Drawing a “surpass” conclusion from two numbers rounded to one decimal place in a screenshot is clearly not rigorous enough.

    Redaction Isn’t the Same as Missing Evidence

    A redacted model name doesn’t mean the evaluation process can’t be verified. Serious third-party evaluations typically publish the following:

    • Model version identifier;
    • Evaluation date;
    • Prompt templates and decoding parameters;
    • Scoring method;
    • Raw scores for each problem.

    Before releasing a new version, model vendors generally also run the candidate version on public leaderboards to confirm its position on public metrics. Once the raw details are available, “what the score is” turns from a guess into a fact.

    More importantly, the redaction itself carries information. Redacting only the model name while leaving the score and ranking untouched shows that the maintainer is confident about the evaluation data itself; the redaction is most likely a confidentiality measure for an unreleased version. For engineering teams, rather than pressing on what the redacted name is, the better move is to re-run the same set of problems immediately after the model is officially released and compare item by item against the leaderboard record. If they match, the evaluation process is transparent and trustworthy; if they don’t, then ask about its scoring rules and test conditions.

    Pixel Comparison and AI Image Recognition: Methodological Problems Behind the Hype

    Identifying redacted text through image comparison is essentially pattern matching. It cannot rule out differences caused by character rendering, anti-aliasing, image scaling, and lossy compression. Pushing the residual below 1/255 only shows that the generated candidate text is very close to the original image; it cannot rule out the possibility that a different font or layout combination produces a similar result. As for “letting AI read the text,” the hallucination risk is just as real: what an image-recognition model gives you is a guess, not a verification. Interestingly, in this incident it was precisely the recognition step that produced the relatively restrained answer, while it was the subsequent human propagation that amplified the “truth”—the person doing the pixel comparison wrote GLM-Name, the circulating clear version became GLM-Nano, and the comments section produced GLM-Next. The suffix got clearer, but the prefix kept multiplying.

    The word “full” in the developer’s relay is the same story. Whether it refers to the full version at the inference stage or the full model at the pre-training stage, whether it means priority adaptation or long-term planning, the relay never says. A conversation fragment detached from its context, after second- and third-hand transmission, easily turns into a definitive “announcement.”

    In methodological terms, this kind of reasoning can be summarized in three problems:

    1. Mismatch between evidence and the proposition to be proven: pixel residuals can only show that two images are similar, not prove that “a specific model is being tested.”
    2. Progressive information decay: relaying naturally involves summarizing, omitting, and even misreading.
    3. Confirmation bias: people first accept the expectation that “a new model is about to launch,” then interpret any later ambiguous information as supporting evidence.

    So when facing rumors, two principles should be followed:

    • Label every conclusion with its source and credibility;
    • For anything involving a major decision, you must find reproducible first-hand material.

    From Leaderboard to Decision: What Benchmark Scores Can and Can’t Answer

    The main role of a leaderboard is initial screening. It can help us quickly narrow down candidates, but it can’t answer “whether this model can reliably complete tasks in my business.” There are three reasons:

    1. Limited problem coverage: public leaderboards usually select only some representative tasks, while real enterprise needs are often highly specialized.
    2. Vastly different conditions: the same model, with different prompts, temperature, context length, tool permissions, and retry counts, can vary by several percentage points.
    3. Possible data contamination: if the solution to a public problem has been indirectly “seen” by the model, the score will be inflated and won’t reflect true generalization ability.

    So a more reliable evaluation needs to satisfy three conditions:

    • The evaluation set comes from real business scenarios;
    • The evaluation process is reproducible;
    • The metrics cover quality, cost, and stability.

    Tiered Rollout and Routing: How to Use Pro, Med, and Small

    The Pro/Med/Small suffix distribution in the image reflects the common “multiple tiers in one generation” strategy of modern large models:

    • Pro tier: pursues the ceiling for complex reasoning and long-chain tasks;
    • Med tier: balances capability and cost;
    • Small tier: targets lightweight, low-latency, high-concurrency scenarios.

    For engineering teams, rather than agonizing over which tier has the “higher total score,” it’s better to assign models by task characteristics.

    Task type Typical characteristics Recommended tier Key metrics
    Code completion, formatting Short context, frequent calls, latency-sensitive Small Latency, unit cost
    Single-file function generation, comment generation Clear task boundaries Med Unit test pass rate
    Cross-file refactoring, complex bug localization Long context, deep reasoning chains, high error cost Pro Compile & test pass rate, regression count

    Three tiers coexisting also leave room for cost optimization: you can design a cascade routing chain that first handles requests with a low-cost tier and escalates to a higher tier when results don’t meet the bar. Sample code:

    1. # Illustrative only: client, TASK_RULES, log_switch, invoke, etc. must be implemented per your actual platform
    2. # TASK_RULES: {task type: (tier, minimum pass rate threshold)}
    3. def dispatch(task_type, payload):
    4. tier, threshold = TASK_RULES.get(task_type, ("med", 0.8))
    5. quality = invoke(client, tier, payload)
    6. if quality < threshold and tier != "pro":
    7. # Log the reason before escalating, to avoid small-tier "confident errors" being silently passed through
    8. log_switch(task_type, tier, quality)
    9. tier, quality = "pro", invoke(client, "pro", payload)
    10. return tier, quality
    1. # Illustrative only: metric aggregation for batch evaluation; interface names and fields must be defined per your environment
    2. from statistics import fmean, pstdev
    3. import math
    4. def summarize(results):
    5. passed = [1 for r in results if r.tests_passed == r.tests_total]
    6. pass_rate = len(passed) / len(results)
    7. n = len(results)
    8. # Wald interval (for demonstration): p ± 1.96 * sqrt(p(1-p)/n)
    9. # Unreliable when very few samples pass; in engineering, prefer the Wilson interval
    10. half = 1.96 * math.sqrt(pass_rate * (1 - pass_rate) / n)
    11. return {
    12. "n": n,
    13. "pass_rate": pass_rate,
    14. "ci95": (max(0.0, pass_rate - half), min(1.0, pass_rate + half)),
    15. "avg_latency_ms": fmean(r.latency_ms for r in results),
    16. "latency_p95_ms": sorted(r.latency_ms for r in results)[int(n * 0.95) - 1],
    17. "retry_rate": sum(r.retries > 0 for r in results) / n,
    18. "edit_distance": fmean(r.edit_distance for r in results),
    19. }

    Cascade scheduling isn’t a new concept. Academia has long done extensive research on cascade-style inference that goes “small first, then large,” with the core idea always being to trade a small number of extra calls for a significant cost reduction. Two points must be noted when putting it into practice:

    • Escalation conditions must be measurable, e.g., not all unit tests pass, compilation fails, or static checks report errors;
    • Escalation logs must be fully retained, to make it easier to later adjust thresholds and tier allocation.

    Gray Release: Five Gates Before Onboarding a New Model

    The rumored limited testing, from an engineering perspective, is a typical gray release. For a new model or new model version, it’s recommended to proceed through these five steps:

    1. Scenario inventory: Sort through the task list currently handled by the existing model, classify it by call volume, failure cost, and latency requirements, and clarify which tasks are suitable for replacement first.
    2. Test set construction: Sample and de-identify from production logs to form an evaluation set covering typical scenarios and edge cases; too small a scale leads to unstable conclusions. Suppose the true pass rate is 60%—with only 10 cases, the 95% confidence interval is roughly 31%~83%, completely unable to support judgments about subtle differences; expanding to 1000 cases narrows the interval to about 57%~63%. A more cost-effective approach is paired design: run both the candidate model and the live model on the same batch of cases, compare only the differences, and the required sample size can usually drop substantially.
    3. Unified evaluation conditions: Fix the prompts, temperature, and context length; take averages or pass rates across multiple samples, avoiding “one run decides everything.”
    4. Multi-dimensional metrics: Besides pass rate, also track latency percentiles, retry rate, edit distance from human benchmark answers, and cost per task, to prevent “score up, experience down.”
    5. Traffic gray release and rollback: The new model first takes a small fraction of traffic, dual-tracked against the live model, with one-click rollback when monitoring flags an anomaly.
    1. # Illustrative only: traffic gray release using stable hashing, ensuring the same user always hits the same version
    2. import hashlib
    3. def pick_model(user_id, new_ratio=0.1):
    4. bucket = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 100
    5. return "candidate" if bucket < int(new_ratio * 100) else "baseline"

    If you integrate through Baidu AI Cloud’s large-model services, you can directly leverage general capabilities such as its model service platform, logging service, and monitoring alerts to bring call logs, latency distribution, and success rate under unified observability, reducing the cost of building and running your own infrastructure.

    Continuous Monitoring: One Evaluation Won’t Last Forever

    Models iterate fast, so evaluation must become routine. It’s recommended to turn evaluation into a pipeline: after a new version is released, automatically run the fixed test set and compare key metrics against the baseline; continuously collect failure samples from production and periodically feed them back to supplement the test set. This way the test set itself also “evolves,” continually covering newly emerging problem patterns.

    In addition, the evaluation method itself needs validation. For example, when using a model to score models, tendencies such as preferring longer answers and preferring its own model’s outputs are common. If you adopt such methods, you need to add human spot-checks and cross-model scoring as corrections.

    Conclusion

    Where that redacted screenshot ultimately points remains unverified for now. But it reminds us that in an era of rapidly iterating model capabilities, the least reliable basis is precisely the most accessible “benchmark screenshot.” For developers, there’s really only one reliable selection method: verify with your own real tasks, your own evaluation standards, and a reproducible process. While the community argues over 0.1 percentage points, engineering teams should be asking a different question: on my own codebase, can it reliably, safely, and cost-effectively get the job done?