Unreleased LLM Coding Benchmark Scores: How to Read Leaked Leaderboard Data and Verify It Yourself

    Screenshots of an unreleased large language model scoring 59.4% and then 70.1% on a coding benchmark sparked debate. This article explains how to interpret multi-tier scores, why leaked numbers often conflict, and how developers can validate a new model with a fixed task set after it ships.

    A redacted leaderboard screenshot drew attention

    Screenshots recently circulated in a technical community showing an unreleased large language model scoring 59.4% on a coding-capability benchmark, 1.6 points above the previous flagship. Another screenshot later put the same model at 70.1%, 0.1 points above the then-first-place entry. The gap between the two figures triggered a debate about whether the numbers were real.

    Regardless of authenticity, “unreleased models appearing on public boards” is worth developers’ attention. It shows how intense competition has become in code generation, and how easily benchmark figures get distorted as they spread. This piece uses the incident to cover three questions: how to read leaderboard data, what a suspected multi-tier lineup implies, and how to build your own verification process once a model is actually available.

    Reading the scores: three tiers, three signals

    Community-compiled records (16 entries, percentages scaled to a full-score basis) place the suspected new model in three sizes:

    • Pro (flagship): 59.4%, second place, just behind 70.0% at the top, and ahead of the previous flagship’s 57.8%.
    • Med (mid-size): 57.1%, immediately after the previous flagship and ahead of several mainstream flagships.
    • Small (lightweight): 54.0%, ninth, between the previous flagship’s light variant and another small model, with several well-known models still behind it.

    Three signals stand out.

    First, the flagship target is to match or beat the current leader. A 1.6-point lead over 57.8% is modest, but top ranks often differ by only a few points, so that margin can change the order. The 70.1% screenshot, if genuine, would be a generational jump—but it lacks cross-checks and is more likely a misreport or a different normalization.

    Second, the mid-size tier is positioned as the value sweet spot. 57.1% is already close to the last-generation flagship. With lower price and latency, a mid-size model could absorb most day-to-day coding work. That is a common industry pattern: one family covering everything from bulk completion to complex architecture design.

    Third, the light-tier gain is the most practical. A prior light model (an older GLM-series figure used only as a reference) scored 45.8%; this Small score of 54.0% is 8.2 points higher. For models that run locally or on cheap inference, eight points can move tasks such as multi-file refactoring and tool-call chaining into a usable range.

    Why two screenshots conflict

    The 59.4% and 70.1% images look contradictory. They actually illustrate three typical problems in how benchmark data spreads:

    1. Inconsistent protocols. Different suites, sampling temperatures, and prompt templates produce very different scores. 70% on board A and 59% on board B may not conflict; they may not measure the same thing.
    2. Different normalization. Some boards use pass rate, some weighted difficulty, some strict “all-or-nothing” scoring. Comparing raw percentages from different sources is comparing incomparable quantities.
    3. Redaction and editing. Three rows on the original board were redacted—typical for unreleased models. Anyone can edit a screenshot in secondary circulation; authenticity cannot be proven.

    Treat leaked figures as clues to verify, not confirmed facts. Useful validation starts when the model is public and callable, using a consistent toolchain.

    A verification workflow after the model opens

    If Small is the next-generation family’s light variant, developers can close the loop as follows once it is available.

    1. Freeze a task set

    Pull representative work you already give current models: single-function implementation, cross-file refactor, test completion, log analysis, tool-call chaining. Freeze prompts, context, and expected outputs so evaluation does not drift.

    2. Compare with a single evaluation script

    1. # Sketch: parallel multi-model eval
    2. tasks = load_tasks("my_daily_tasks.jsonl") # frozen task set
    3. for model in ["baseline_model", "new_small"]:
    4. results = run_eval(model, tasks, temperature=0) # fixed sampling
    5. report = aggregate(results) # pass rate, mean latency, mean token cost
    6. print(model, report)

    Control variables: temperature 0 or a fixed value, aligned context length, identical tool definitions. Otherwise the comparison is meaningless.

    3. Track three dimensions, not a single score

    • Pass rate: share of tasks that succeed in one shot—closest to leaderboard style.
    • Repair cost: average extra turns to fix failures—hidden cost boards do not measure.
    • Latency and cost: a light model’s value is capability per dollar. A 54% score with much cheaper inference can beat a 59% flagship in practice.

    Closing note

    Leaderboards are a starting point, not the finish. Community scores can go viral in minutes; whether a model fits your codebase, stack, and budget only shows up on your own frozen tasks. Stay interested in these leaked numbers, but do not treat them as settled. When the model ships, run the work you currently give the old version and see how much it can take—that is always more persuasive than forwarding a screenshot.