Decoding a Leaked Open-Weight LLM Benchmark Screenshot: From Redaction to Pixel-Level Reconstruction

    A redacted benchmark screenshot with three entries labeled Pro, Med, and Small triggered a community effort to reconstruct model names and scores. This article breaks down the font-recreation and pixel-residual methods used in that effort, explains what the visible score structure can and cannot tell us, and offers practical guidance for developers on interpreting benchmark leaks and community-circulated numbers.

    1. Event Recap: How a Redacted Leaderboard Sparked a Community Investigation

    The starting point was simple. While discussing whether open-weight models had fallen behind closed-source frontier models on complex engineering tasks, the maintainer of a benchmark leaderboard posted a screenshot containing 16 entries. The image did not name any unreleased model, but three entries were deliberately redacted: both the model-name prefix and the score were masked, leaving only the suffixes Pro, Med, and Small.

    Those three suffixes, together with the maintainer’s wording hinting that a new model might already be under test, set off a wave of community inference. Two score versions soon circulated: one estimate of 59.4% based on the redacted leaderboard, and another reaching 70.1%. The gap between the two numbers illustrates a key point: when official information is absent, community-circulated “benchmark scores” are inherently noisy.

    2. The Technical Path of Leak Reconstruction: How the Community Tried to Break the Redaction

    The most technically interesting part of the incident was the community’s attempt to reconstruct the redacted areas. The process can be divided into three steps:

    1. Font and layout recreation. Some users manually typed suspected prefix text using similar fonts and layout, then applied Gaussian blur to simulate the masking method used in the original image. The core of this step is constructing a “hypothesized original image.”

    2. Per-pixel residual comparison. After aligning the recreated image with the original redacted image, they calculated the difference pixel by pixel. According to the reported result, the pixel residual between the two images was less than 1/255—meaning that at 8-bit color depth, most pixels differed by less than one grayscale level, making the two images nearly visually indistinguishable.

    1. # Illustration: simplified logic for pixel residual comparison
    2. import numpy as np
    3. def pixel_residual(img_a, img_b):
    4. diff = np.abs(img_a.astype(int) - img_b.astype(int))
    5. return diff.max(), diff.mean()
    6. # If max < 1, the residual is within quantization error

    3. Cross-checking with image-recognition tools and multiple candidates. Others used image-recognition tools to guess at the blurred region and arrived at different prefix candidates. Interestingly, the person doing pixel comparison wrote one name, a later “clear version” of the leaderboard showed another name, and the comments produced a third guess. All three suffixes were visible, but the prefix became more uncertain the more people guessed.

    This exposes an important fact: a pixel residual below 1/255 only proves that “the recreated image is highly consistent with the original,” not that “the guessed text is the real text.” If the blur is strong enough, multiple texts with similar length and similar stroke distribution can produce close residuals. This is the methodological limitation of all image-forensics-style analysis.

    3. The Leaderboard Itself: How to Read the Scores

    Setting aside the prefix dispute, the structural information in the leaderboard is relatively reliable. The leaderboard contains 6 coding problems with a total score of 60 points, and the percentages shown are calculated against the full score. According to community-compiled data, the three redacted entries were roughly arranged as follows:

    • Pro tier: about 59.4%, second on the leaderboard, behind a closed-source flagship model at about 70.0% and ahead of the third place by about 1.6 percentage points;
    • Med tier: about 57.1%, immediately after the third place and ahead of several mainstream international closed-source models;
    • Small tier: about 54.0%, ranked ninth, positioned between two generations of lightweight models.

    A more noteworthy reference point is that the previous-generation model in the same series on the same leaderboard scored about 45.8%, which is 8.2 percentage points lower than the inferred Small tier. If the Small tier is indeed the small model in the next-generation series, this would mean that even the smallest new model may significantly outperform the full version of the previous generation. That is the real information value in this data.

    4. Corroborating Signals from Release Timing: The Developer Ecosystem

    Beyond the leaderboard, the community also picked up corroborating signals from activity in the developer toolchain. The maintainer of a mainstream coding tool recently mentioned that an open-weight model performed well after scaling up, and that “next up” was a new model in the same series. When asked whether a lightweight version or the full version would be offered first, he gave a clear answer: “full.”

    The exchange itself did not mention any version number, but the timeline was suggestive: first someone reconstructed three model suffixes from the leaderboard, and shortly afterward came the statement that it was “this series’ turn.” The community naturally interpreted this as a signal that staged testing was approaching. Some users even looked through the evaluator’s historical posting patterns to estimate the release window.

    5. Three Practical Recommendations for Developers

    First, distinguish “official benchmark scores” from “community-circulated scores.” In this incident, both 59.4% and 70.1% circulated at the same time, both claiming to come from the same model. Before a model is officially released and the evaluation protocol is public, any specific score should be treated as low-confidence information. A genuinely reliable comparison must use the same evaluation set, the same prompt template, and the same inference parameters.

    Second, beware of over-interpretation in pixel-level analysis. Gaussian blur is inherently a lossy operation. Residual comparison can narrow the candidate space, but it cannot uniquely determine the original text. The community’s result—multiple prefix versions coexisting—already proves this. A reasonable framing for such analysis is “circumstantial evidence that narrows the range,” not “conclusive evidence.”

    Third, use reproducible private benchmarks for acceptance testing. For engineering teams, leaderboard scores are only a reference for model selection. A more practical approach is to wait until the model is officially available, then turn high-frequency task samples from daily business operations into an internal evaluation set, run them with a unified scoring standard, and see how much of the existing workload the new model can handle. The value of such private benchmarks is far higher than any public leaderboard percentage.

    6. Conclusion

    This leak incident offers a complete view of an interesting technical-community phenomenon: a redacted screenshot was decoded layer by layer through font recreation, pixel comparison, AI recognition, timeline inference, and other paths, ultimately producing “suspected but not conclusive” intelligence. It was both a miniature exercise in image forensics and a public lesson in information confidence—before an official release, all scores are only hypotheses; and the best way to verify a hypothesis is always to wait until the model is in hand and run it on your own data.