Benchmarking Evolution: Why Private Test Sets Are Replacing Public Leaderboards in AI Evaluation

The Technical Logic Behind the Benchmarking Shift

The latest version of the intelligence benchmark system introduces a key change: 60% of the evaluation weight now falls on private test sets or test sets with withheld answers. On the surface, this looks like a simple adjustment of weighting ratios. In practice, it represents a fundamental correction to how the entire industry evaluates models. The long-standing reliance on public benchmark datasets has exposed a serious systemic flaw—when training data overlaps heavily with test sets, a model’s leaderboard score reflects its memorization capacity more than its reasoning ability.

From a technical evolution standpoint, public benchmark data has been repeatedly used in model training and fine-tuning, creating a difficult-to-break cycle: models achieve high scores on public test sets, developers make model selection decisions based on those scores, and then discover during real-world deployment that model performance falls far short of expectations in actual business scenarios. The benchmark system’s decision to shift most weight toward private test sets is a direct response to this pain point. The dual-track design—retaining some public questions while introducing a large volume of unseen ones—more realistically simulates a model’s generalization ability in unfamiliar scenarios.

It’s worth noting that the weighting change is not merely a proportional adjustment. It comes with a complete anti-leakage mechanism. The generation of private test questions uses both dynamic template composition and manual review to ensure that the content of each evaluation round has no direct or indirect semantic overlap with previous versions. The question bank also incorporates a periodic refresh mechanism—certain publicly exposed questions are automatically downweighted, while newly added private questions gradually gain more weight. This dynamic balancing strategy means evaluation results are no longer a static snapshot but a comprehensive assessment of a model’s continuous learning capability and stability.

The Widening Capability Gap Between Open-Source and Closed-Source Models

The most notable change after the evaluation criteria tightened is that the score gap between open-source and closed-source models expanded from 3 points under the old system to 12 points. This widening gap does not mean the new evaluation standard is biased toward either side. Rather, it reflects a fundamental difference in how the two technical approaches handle unseen problems.

Closed-source models invest more resources in engineering deployment, including larger-scale training datasets, more refined human feedback alignment processes, and stricter testing and validation. For reasoning problems that never appeared during training, closed-source models demonstrate stronger robustness in chain-of-thought capabilities and multi-step reasoning mechanisms. Some open-source models, while performing well on public test sets, show significantly larger score drops when facing questions with transformed semantics and shifted scenarios.

Several technical factors drive this result. First, differences in training data scale directly affect a model’s coverage of long-tail knowledge. When a model encounters a problem type it has never seen before, it must rely on analogical transfer and compositional generalization—capabilities closely tied to the scale and diversity of training data. Second, alignment precision plays a critical role. Closed-source models typically use multi-round iterative reinforcement learning optimization, which makes them better at rejecting incorrect reasoning paths and thinking more carefully. Third, inference-time compute investment cannot be ignored. Closed-source models allow larger thinking budgets during inference, enabling longer chain-of-thought reasoning, while some open-source models face constraints on reasoning length and depth due to deployment cost considerations.

For the industry, this widening gap is not bad news. It provides a clearer reference point. For business scenarios with extremely high accuracy requirements and low error tolerance—such as financial risk control, medical diagnostic assistance, or code review—choosing a closed-source model remains the more reliable strategy. For cost-sensitive scenarios with relatively standardized tasks, open-source models combined with fine-tuning and knowledge distillation can still deliver competitive cost-performance.

Revisiting the “Only 3 Points Behind” Claim

The claim that domestic models lag leading closed-source models by only 3 points has been circulating in the industry. Under the new v5 evaluation standard, this assertion no longer holds. It’s important to recognize that this claim was never a pure technical conclusion—it was a product of a specific evaluation framework. When the constraints of that framework change, the old conclusion naturally loses validity. This does not negate the technical progress of open-source models. It suggests that the true capability dimensions of models—particularly their generalization ability on unseen problems—may differ from what public leaderboards have shown.

Looking at the score composition under the v5 standard, scores are derived from four components: private reasoning questions (30% weight), private knowledge questions (20% weight), public reasoning questions (25% weight), and public knowledge questions (25% weight). The first two categories together account for 50% of the total, directly determining the overall score trajectory. The design principle for private questions is “grounded in public knowledge, but distant from public test questions”—the knowledge points being tested remain within the model’s training coverage, but the question phrasing, compositional logic, and solution paths are entirely different from public versions. This approach effectively tests whether a model truly understands the internal structure of knowledge rather than simply memorizing surface-level question-answer mappings.

For developers, the key takeaway is that no single public leaderboard should serve as the sole basis for model selection. A more reliable approach involves multi-dimensional evaluation—independent testing on small-scale private test sets, targeted performance tests on typical business scenarios, and stability testing under adversarial inputs. A complete evaluation framework should consist of three layers: a general capability baseline, scenario-specific testing, and competitive benchmarking. Only this combination can produce an accurate enough model profile within controllable costs.

Model Selection Strategies for the New Evaluation Era

Faced with this significant shift in evaluation standards, developers need to revisit their model selection methodology. Under the v5 standard, high-scoring models typically exhibit two notable characteristics: architecturally, they adopt more efficient attention mechanisms and sparse activation strategies, allowing the model to carry more knowledge within limited compute; in training strategy, they emphasize frequency balancing and data cleaning to ensure the model is not simply memorizing high-frequency patterns but developing pattern recognition and rule generalization capabilities.

For application-layer developers, the recommendation is to start from actual business problems and establish a three-layer mapping of “task-data-metric.” First, identify the natural language task types involved in your business, such as information extraction, semantic retrieval, content generation, or dialogue reasoning. Second, collect private evaluation data highly relevant to your business. This data does not need to be large-scale, but it should cover typical edge cases. Third, define clear evaluation metrics, including both hard metrics like task accuracy and soft metrics related to user experience such as response latency and answer stability.

For infrastructure builders, the new evaluation standard sends an equally clear signal: model capability assessment cannot rely solely on external public leaderboards. Building a continuously evolving private evaluation system is an essential investment. Such a system should have core capabilities including automated question generation, automated answer comparison, and dynamic difficulty adjustment, allowing rapid iteration as business requirements change. Evaluation result interpretation also requires professional analysis and visualization tools to help algorithm teams quickly identify which types of problems expose capability gaps, enabling targeted adjustments to the composition and weighting of the next round of training data.

The ongoing evolution of AI evaluation systems reminds us that a model’s true capability must be tested and measured across richer scenarios and stricter standards. For developers, the ultimate criterion for judging a model is always the results it delivers against actual business objectives—not the number on any single leaderboard.