Leaked Model Benchmark Scores: How to Read the Leaderboard Truth and Choose Models by Tier

    A leaked coding leaderboard screenshot — with redacted model names and two conflicting scores — set off a heated community debate. This article unpacks the three puzzles it raised (the score, the hidden model name, and the release timing), explains why secondhand screenshots get "sharper" as they spread and how to present credible evidence, and shows why a small-sample benchmark (6 questions, 60 points) can't be read literally. It argues that capability tiering matters more than any single rank, then gives developers a practical playbook: build a reproducible eval set from real tasks, dispatch requests across light/medium/strong tiers with cascade routing and canary rollouts, keep a cost ledger for tokens, storage, and latency, and confirm release signals only through official channels before migrating. Every model and score mentioned comes from community speculation, not official confirmation.

    1. What happened: a redacted leaderboard and three puzzles

    A screenshot of a coding-ability leaderboard recently circulated in technical communities. In the image, the model names and scores for three entries were redacted, leaving only the suffixes Pro, Med, and Small visible. Two contradictory readings soon appeared: one showed 59.4%, slightly below the top model; the other showed 70.1%, edging out the leader by 0.1 percentage point. The model-name prefixes were also disputed. According to reports, the leaderboard was posted by the maintainer of a coding tool while responding to a developer’s doubts — the developer had argued that on complex engineering tasks, open-weight models can no longer keep up with closed frontier models.

    The discussion breaks down into three questions:

    • The score puzzle: Why would the same model show two scores more than 10 percentage points apart?
    • The model-name puzzle: What is the redacted prefix?
    • The timing puzzle: Is a release window approaching?

    It’s worth stressing that every model and score discussed here comes from community speculation, with no official confirmation. Rather than chase one specific number, it’s more useful to build a reusable way of judging such claims.

    2. The score puzzle: why secondhand screenshots get “sharper” as they spread

    Start with the evidence chain. The original leaderboard was posted by the maintainer, but key fields were obscured. The “clear version” circulating in the community is secondhand work — some reconstructed by estimating from the redacted image, others with scores simply replaced. The gap between the two versions can’t be explained by rounding:

    • The 59.4% version was called fake in the comments;
    • The 70.1% version sits just 0.1 percentage point above the leader — a more dramatic position that spreads faster.

    Why can one image produce two answers? At least three reasons:

    1. Missing information: Once names and scores are obscured, reading the image relies on guessing character shapes, pixels, and layout.
    2. Transcription loss: Screenshots are forwarded, compressed, and redrawn; font rendering and anti-aliasing all affect legibility.
    3. Motivation bias: Controversial posts spread more readily, and a “just overtook the leader” narrative is inherently more shareable.

    How should credible evidence be presented? Reliable reporting follows these principles:

    • Include a full leaderboard link and the query time;
    • State the benchmark task set, version, and parameters (temperature, context length, whether reasoning mode is on, etc.);
    • Provide the raw output so it can be reproduced.

    The redacted screenshot, by contrast, conveys emotion but no reproducible information.

    Evidence type Example Credibility
    First-party, verifiable Vendor announcements, model cards, official docs High: traceable and reproducible
    First-party, unverifiable Unconfirmed leaderboard screenshots Medium: has a source, but fields are incomplete
    Secondhand processing Community “clear” leaderboards, pixel-reconstructed images Low: multi-hop, easy to distort
    Speculation Guesses about redacted prefixes and release timing Clues only, not a basis

    Some in the community did attempt “technical forensics”: rendering candidate names in a similar font, adding Gaussian blur, then comparing pixel by pixel with the original and claiming a residual under 1/255. Others used image-recognition tools to identify the prefix. The approach is reasonable, but has inherent limits:

    • The comparison assumes “the original was rendered in some font, then blurred.” If the screenshot was scaled by a screenshot tool, JPEG-compressed, or forwarded again, that assumption breaks — and no matter how small the residual, it can’t prove the text content.
    • The text in a redacted region is, in an information-theoretic sense, unrecoverable. A recognition tool only offers the “most likely candidate,” not a definitive answer.

    3. The leaderboard itself: how to read a 6-question, 60-point benchmark

    According to reports, this leaderboard tested only 6 coding questions, worth 60 points in total, with percentages converted from the score. Taking the community-compiled version, the rough picture is:

    • The Pro tier scored about 59.4%, placing second, roughly 10.6 percentage points behind the leader;
    • The Med tier scored about 57.1%, following another leading model;
    • The Small tier scored about 54.0%, around ninth place;
    • A previous-generation flagship scored about 45.8%, 8.2 percentage points below the Small tier.

    Three points stand out.

    First, a position gap isn’t a capability gap. In this score range, adjacent models often differ by only a percentage point or two — one or two points on a 60-point scale, which may come down to a single question.

    Second, the scale is coarse. With 6 questions, each carries about 10 percentage points. A small-sample benchmark’s score can only roughly reflect relative strength on these task types, and confidence intervals tend to be wide.

    Third, a single-dimension rank ≠ overall level. Coding ability can be split into at least five categories: standalone function completion, cross-file refactoring, bug localization and repair, multi-turn instruction following, and long-context repository understanding. One composite score can’t cover all of them.

    Typical coding benchmarks contain dozens to hundreds of questions. New models iterate faster and faster, and contact between questions and training data is hard to avoid, so many people have begun to doubt their discriminative power and fairness. This isn’t surprising:

    • With few questions, each carries too much weight, so rankings can be swung by a handful of items;
    • Public questions circulate over a long time, making it hard to rule out exposure during training.

    Benchmarks are therefore suited to coarse horizontal screening, not to a final go-live decision.

    4. Small models take the stage: the tiering signal matters more than any single score

    If this information is accurate, the most noteworthy point isn’t “who overtook whom,” but that three tiers of new models appeared on the same leaderboard, along with their position relative to the previous-generation flagship.

    Drawing on recent public discussion, the community broadly recognizes a few trends:

    • Smaller-parameter models have made marked progress on verifiable tasks such as math and code;
    • The gap between open-weight and closed models keeps narrowing;
    • Lightweight models aimed at low-latency, low-cost scenarios are becoming the workhorse for batch tasks.

    Against this backdrop, it’s no surprise that the new small tier’s score beats the previous-generation flagship. This is capability trickling down, not across-the-board superiority. On the same leaderboard, the new Pro tier still trails the strongest closed model, and complex engineering tasks remain the frontier models’ stronghold.

    For developers, a more useful reading is to draw a capability curve:

    Scenario Typical features Priority
    Completion, formatting, explanation Single file, short context, high call volume Light tier
    Single-file changes, scripting, test additions Medium context, clear acceptance criteria Medium tier
    Cross-file refactoring, bug localization, architecture design Long context, multi-turn reasoning, high fault tolerance Strong tier
    Data can’t leave the domain Private deployment, strict audit requirements Locally deployable models
    Edge and on-device Constrained compute, memory, power Light tier
    Long-running unattended tasks Needs fault tolerance and self-checking Strong tier, plus human review

    Note too that benchmark scores and deployment cost don’t scale linearly. Parameters, inference VRAM, concurrent throughput, and per-call cost all rise with the tier, while the distribution of task difficulty is long-tailed: most everyday requests fall into the light and medium tiers, and only a few critical tasks need the strong tier. The goal of tiered selection is to reserve the strong tier for the problems that genuinely need it.

    5. From benchmark to production: a four-step method for your own eval set

    A leaderboard is fine to watch, but go-live has to be judged on your own data. Build a reproducible eval set to answer “can we use it.”

    Step 1: Create tasks. Pull tasks from your team’s repositories, requirement tickets, and defect tickets, covering:

    • Bug fixes: provide reproduction steps and a failing case;
    • Feature development: provide a requirement description and acceptance cases;
    • Refactoring: require unchanged behavior, with all existing tests passing;
    • Code review: plant typical defects in a commit and see whether they’re caught;
    • Instruction following: include hard constraints on format, length, forbidden items, etc.

    Step 2: Define scoring. Favor objective acceptance over subjective “looks good” ratings:

    • Whether generated code compiles;
    • Hidden-test pass rate (e.g., pass@1);
    • First-pass rate on real requirements;
    • Extra time and extra cost.

    Step 3: Control contamination. Keep eval tasks out of training and unpublished, rotate them periodically, and re-run the baseline whenever the model version changes.

    Step 4: Wire it into the pipeline. Make the eval a scheduled job, write results together with cost data to a logging service, and track regressions with monitoring and alerts.

    Sample code:

    1. # eval.py: Minimal implementation of a self-built coding benchmark (illustrative)
    2. import json
    3. from pathlib import Path
    4. from statistics import mean
    5. TASKS = Path("tasks.jsonl")
    6. RESULTS = Path("eval_results.jsonl")
    7. def load_tasks():
    8. """One task per line: id / prompt / test_cmd / hard_rules"""
    9. with TASKS.open(encoding="utf-8") as f:
    10. return [json.loads(line) for line in f if line.strip()]
    11. def evaluate(model_client, tasks):
    12. rows = []
    13. for t in tasks:
    14. out = model_client.chat(
    15. messages=[
    16. {"role": "system", "content": "Output code only, no explanation."},
    17. {"role": "user", "content": t["prompt"]},
    18. ],
    19. temperature=0, # fixed sampling params during eval to guarantee reproducibility
    20. )
    21. # 1) Objective acceptance: does the output pass the task's built-in check command
    22. passed = run_checks(t["test_cmd"], out) # implemented by the team, returns bool
    23. # 2) Hard constraints: length, forbidden words, output format, etc.
    24. rules_ok = all(rule(out) for rule in t["hard_rules"])
    25. rows.append({
    26. "task": t["id"], "model": model_client.model_id,
    27. "pass": passed, "rules_ok": rules_ok,
    28. "latency_ms": model_client.last_latency_ms,
    29. "tokens": model_client.last_usage.total_tokens,
    30. })
    31. with RESULTS.open("a", encoding="utf-8") as f:
    32. for r in rows:
    33. f.write(json.dumps(r, ensure_ascii=False) + "\n")
    34. print("pass@1 =", round(mean(r["pass"] for r in rows), 3))
    35. return rows
    36. if __name__ == "__main__":
    37. evaluate(model_client=None, tasks=load_tasks()) # replace with the actual model-call client

    The model client, test-command runner, and similar parts are illustrative. The exact interface should follow your chosen model service or self-hosted inference service.

    6. Tiered dispatch: routing, feedback, and canary

    With your own eval data, you can organize the three capability tiers (strong, medium, light) and two deployment forms (local or cloud) into a dispatch strategy.

    1. Cascade routing: easy before hard

    • Let the light tier go first, with rule checks, compilation, and tests as the first gate;
    • When a task fails or produces an unacceptable result, automatically escalate to the medium or strong tier and retry;
    • Cap the maximum number of escalations so repeatedly failing tasks don’t keep burning the strong-tier budget.

    2. Dynamic confidence: don’t send everything to the strong tier

    The strong tier has higher latency and cost. If the model service returns confidence signals such as log probabilities, use them to set an escalation threshold: when output confidence falls below the threshold, or the output repeats or is truncated, hand it to a stronger model.

    3. Multi-dimensional acceptance

    For structural tasks, check whether it compiles; for logic tasks, check whether it passes tests; for security tasks, check static analysis. Output that only “looks right” doesn’t advance.

    4. Rollbackable canary release

    When a new model goes live, bucket traffic by user or task type for canary. During the canary window, run old and new models in parallel and compare results, and keep a one-click switch back to the old model.

    Sample code:

    1. # router.py: Tiered model dispatch (illustrative)
    2. import hashlib
    3. from dataclasses import dataclass
    4. TIERS = ("small", "medium", "large")
    5. def pick_tier(task: dict) -> str:
    6. """Select a tier by task features; default to the light tier, auto-upgrade for complex tasks."""
    7. if task.get("cross_file") or task.get("high_risk"):
    8. return "large"
    9. if task.get("files", 1) > 3 or task.get("diff_lines", 0) > 200:
    10. return "medium"
    11. return "small"
    12. def assign_new_tier(task: dict, ratio: float, model_id: str) -> str:
    13. """Canary: a new model version covers only a fraction of requests, for comparison and rollback."""
    14. bucket = int(hashlib.sha256(f"{task['user_id']}:{model_id}".encode()).hexdigest(), 16) % 100
    15. return model_id if bucket < ratio * 100 else task["current_model"]
    16. def dispatch(task: dict, clients: dict):
    17. tier = assign_new_tier(task, ratio=0.1, model_id="candidate-v2")
    18. client = clients[tier]
    19. out = client.generate(task["prompt"], temperature=0.2)
    20. # if validation fails, upgrade and retry, at most two levels
    21. for _ in range(2):
    22. if check(out, task):
    23. break
    24. tier = TIERS[min(TIERS.index(tier) + 1, len(TIERS) - 1)]
    25. out = clients[tier].generate(task["prompt"], temperature=0.2)
    26. log_call(task["id"], tier, out, check(out, task)) # write to log for later review
    27. return out

    The model client and acceptance functions are illustrative and need to be wired to your actual model service or self-hosted inference service.

    5. Keep an eye on the resource ledger

    Cost doesn’t come only from model calls. If you add a vector store for repository retrieval, or a message queue for asynchronous task scheduling, storage and compute grow as well. Before going live, estimate three kinds of resources:

    • Input and output tokens;
    • Storage and retrieval overhead;
    • Concurrency and queueing latency.

    6. Cloud API or local deployment

    Local deployment suits data-sensitive, stable-call, long-term high-frequency scenarios, but VRAM and operations are the barriers. Scenarios with large elastic swings fit cloud on-demand calls better, or a hybrid of both. For example, with Baidu AI Cloud, an enterprise can call general large-model capabilities on demand through model services, or host its own inference service on general cloud products such as containers and function compute, and use logging plus monitoring/alerting for call auditing and cost aggregation.

    7. Release cadence: how to read the “scale-up → rotation → canary” chain

    Back to the timing puzzle. According to community accounts, a coding tool’s developer mentioned scaling progress: the previous model’s scale-up was running well, and “next it’s someone’s turn.” When asked whether a lightweight or full version would come first, he answered “the full version.” The remark was quickly read as a signal that a new round of canary testing is near.

    How should you read such signals?

    • Signals with reference value: scale-ups and listings on an inference service platform, platform-side call logs after integration, and doc updates on official channels — these usually come before a public release.
    • Signals that shouldn’t be treated as conclusions: leaderboard screenshots, redacted reconstructions, and over-interpretation of a single sentence.

    Keep a list of questions to verify:

    1. What are the parameter scale and context length?
    2. Are the weights open? Does the license permit commercial use?
    3. Is a standard API provided? What are the rate limits and quotas?
    4. What are the response latency, throughput, and price?
    5. Is it compatible with your existing call chain?

    Only after these are confirmed through official channels should you move into selection and migration.

    8. Action checklist for developers

    1. Preserve evidence: save the original screenshot and source link, and distinguish first-party from secondhand information.
    2. Build a baseline: create your own eval set from real team tasks and fold it into CI.
    3. Tier your dispatch: route by task features, easy before hard, escalate on failure.
    4. Control cost: track tokens, storage, and retrieval overhead; set budget caps and fallback strategies.
    5. Release steadily: roll out via canary, keep a rollback switch, and monitor key metrics.
    6. Respect boundaries: don’t send unpublished data, keys, or sensitive personal information to services you haven’t evaluated.

    9. Conclusion

    A redacted screenshot yielding two different scores is itself a reminder: secondhand information amplifies bias along the chain of transmission. For developers, what’s truly reliable isn’t a rank on some leaderboard, but three things:

    • A reproducible evaluation;
    • A rollbackable architecture;
    • Measurable cost.

    In an era where capability tiering is becoming the norm, whoever converts “benchmark” into “business metrics” fastest can judge, the moment a new model ships, whether it’s worth integrating.