Small-Model Internal Scores Differ: Reading a Code Benchmark Dispute
Two screenshots from the same code benchmark reportedly scored an unreleased model at 59.4% and 70.1%. This article reconstructs the dispute, explains how the six-task, 60-point benchmark converts raw scores into percentages, and outlines how developers should treat leaked rankings when choosing models for real engineering work.
Recent community discussion claimed that a vendor had opened a limited internal test of a new model generation. Two evaluation screenshots then showed sharply different results: a sample that appeared to be the new model scored only 59.4% in one ranking and 70.1% in another, a gap of more than 10 percentage points. Both images were said to come from the same code-capability benchmark, and some model names on the original board were deliberately obscured. The “one model, two scores” pattern quickly raised two questions among developers: which number is real, and where should the new model be placed? This article reconstructs the episode from the benchmark rules, dataset design, and model-engineering perspective, then looks at the technical issues behind the gap.
A score dispute that drew attention
The discussion began with a developer’s everyday observation: on complex engineering tasks, open-weight models already felt unable to keep up with closed frontier systems. The developer then shared a traffic curve from a programming-assistant project, which drew a reply from a maintainer of the benchmark. That reply included a capability board with 16 entries and said test data for a new model had already been obtained. The evaluator did not name the model, so readers inferred from the wording that it was an unreleased next-generation product from a particular vendor.
What pushed the topic further was not the model itself but conflicting versions of the ranking image. The first image, estimated by users from a redacted board, put the new model’s Pro tier at 59.4%, 1.6 percentage points above a closed flagship. In the comments, someone immediately posted a second screenshot showing 70.1%, even 0.1 percentage points above the closed flagship that then led the board. A later ranking compiled by the community from a clearer screenshot again listed the Pro tier at 59.4%, in second place, while the closed flagship remained first at 70.0%. The large gap made many people question whether the scores were genuine: different evaluation conditions, or an edited image?
Technically inclined users then tried pixel-level analysis of the original image, reconstructing obscured text through font rearrangement and Gaussian blur. One claim was that the residual after reconstruction was below 1/255. Others tried AI image recognition on the covered regions. Those efforts widened the story’s reach.
What the benchmark and the percentages mean
Understanding the dispute starts with the benchmark’s published rules. The evaluation contains six programming tasks and a raw total of 60 points. Percentages on the board are raw scores scaled to a perfect score. Each percentage point therefore corresponds to 0.6 raw points. On a code task, 0.6 points can be the difference between passing and failing a single test case, so small scoring changes can move the displayed percentage. The benchmark emphasizes complex engineering work such as cross-file edits, type inference, and test completion. Those tasks demand context understanding, multi-step reasoning, and tool use. They separate models more clearly than simple code-generation items, but they also add uncertainty: sampling temperature, the number of runs, and even code-formatting style can change the final score.
From a redacted image to a model name
Three entries on the original board were covered. Their suffixes were Pro, Med, and Small, while the prefixes were fully blurred. That style of redaction was not enough to hide identity. Users tried several “unredaction” methods, including manual font rearrangement, pixel-by-pixel comparison after Gaussian blur, and OCR guesses from vision models. The candidate prefixes differed, and none received official confirmation. Notably, the board also showed the previous generation’s mid-tier model from the same series at 45.8%, 8.2 percentage points below the new model’s Small tier. If Small really is a next-generation small-parameter version, that gain over the previous mid-tier is substantial. Whether it can directly take on the previous mid-tier workload still needs more evidence.
Where small-parameter models still fall short
On the ranking, all three tiers of the new model looked competitive. Pro placed second at 59.4%, ahead of several closed models. Med followed closely at 57.1%. Small sat in the middle of the pack at 54.0%, ahead of several open and lightweight closed models. The closed flagship at the top still held a clear lead at 70.0%. On very complex engineering tasks, a noticeable gap remains between small-parameter models and top closed models. That gap has been observed repeatedly in practice: larger models are better at retaining constraints across long context and at updating related call sites when a function changes. Small models often do well at local generation and completion, but on repository-wide refactors they more often miss pieces or break logic. They fit IDE assistance such as completion, unit-test generation, and comment writing. For large architectural changes and cross-module dependency edits, frontier models remain the more reliable choice.
How developers should read evaluation data
For teams in companies, leaked “limited internal test” information should be treated cautiously. A benchmark score is only a first reference and cannot be copied directly into a business workload. Practical checks include:
- Check how open the evaluation set is and how well it resists leakage. If the set is not public, or if the model may have seen similar tasks during training, a high score does not prove real capability.
- Look at subtask scores, not only the total. Complex engineering work covers code correctness, compile pass rate, test coverage, and runtime performance. Two models can share a total while one is stronger at refactoring and the other at completion.
- Build an internal regression set. Sample historical code changes from the company, assemble dozens to hundreds of problem–patch pairs, and score them with fixed parameters. That measures behavior on real work.
- Balance quality and cost. Small-parameter models have clear advantages in inference speed, deployment cost, and latency, but quality may have to give way. A mixed setup can let a small model filter candidates and a larger model make the final choice, holding quality while controlling cost.
A simplified evaluation flow looks like this:
def evaluate(model, dataset):total = 0for task in dataset:result = model.generate(task.prompt)total += validate(result, task.tests)return total / len(dataset)
Here, validate can combine static checks, unit tests, and capability tests so the result stays as objective and reproducible as possible.
Conclusion
Taken together, the limited-internal-test rumor cannot be confirmed, and the gap between the two boards is far from settled. Whether the new model is a cross-generation improvement depends on an official evaluation report and a reproducible setup. For developers, the model name and a leaderboard score are not the main selection criteria; stable output on real work is. Rather than chasing unverified internal numbers, it is more useful to build an evaluation system on models that are already available and use an engineering process to pick the one that fits the current task.
