AI Intelligence Index v5: Why the Leaderboard Changed So Dramatically

In 2026, an AI evaluation platform released Intelligence Index v5, describing it as the largest redesign since v4. After the announcement, the leaderboard changed dramatically. Several domestic models that had consistently ranked near the top fell significantly.

The first reaction from many observers was that “the new questions must be much harder.” However, a closer look at the evaluation methodology suggests that the absolute difficulty of the questions may not have increased substantially. The bigger change was the way answers were evaluated.

1. Locking the Test in a Drawer: Private Test Sets and Hidden Answers

The most important change in v5 is that approximately 60% of the weighting is now based on private test sets or hidden answers.

Under the previous evaluation system, questions remained public for long periods. Any team could download and study them, or even include them in training data. There were many ways to optimize for a higher score, but a high score did not necessarily represent genuine intelligence.

The new system effectively locks the test in a drawer. Models can learn what types of capabilities are being tested, but they cannot access the exact questions. This makes it much harder to earn points simply by memorizing answers.

At the same time, the change can cause major leaderboard swings. Models that previously relied heavily on tuning against public questions will naturally perform closer to their actual generalization ability once those questions are replaced with private ones.

2. Nine Evaluations, Four Weighting Categories, and Two New Measuring Sticks

Version 5 includes nine evaluations divided into four categories, with each category accounting for 25% of the overall score. Although this may appear to be a straightforward redistribution of weights, the most significant changes are the complete replacement of the scoring logic for two types of tasks.

Real Code Integration Evaluation

The first task category evaluates whether generated code can be integrated into a real repository.

Questions are extracted from real open-source repositories. The model must act like a developer and generate a patch for a specific issue. The standard is no longer whether the model claims that its code should work. Instead, the question is whether repository maintainers would be willing to merge the patch.

Any blocking failure results in a score of zero for the entire task. Examples include conflicts with the existing code style, unhandled edge cases, or changes that break related tests.

This approach is closer to real software engineering than conventional code-generation benchmarks. It also exposes how well a model understands the broader context of a large codebase, rather than only whether it can produce locally plausible code.

Scientific Terminal Workflow Evaluation

The second task category evaluates scientific workflows in a simulated terminal environment.

Approximately 70 tasks require the model to complete multi-step research workflows. These may involve installing dependencies, writing scripts, processing data, checking intermediate results, and correcting errors before reaching the final conclusion.

A task receives full credit only when all of its substeps are completed successfully.

By replacing a single final answer with end-to-end execution, this evaluation measures planning, tool use, intermediate-result validation, and error recovery. It is designed to test more than the appearance of intelligence in a single conversational turn.

3. Why Did the Open-Source and Closed-Source Gap Suddenly Reach 12 Points?

The platform stated in its announcement that the average score gap between open-source and closed-source models increased from 3 points in the previous version to 12 points in v5.

The change is substantial, but it should not be interpreted simply as evidence that open-source models have become weaker. Several factors may explain the result.

• Different levels of overfitting to public benchmarks. Open-source models and their surrounding training pipelines often depend heavily on public data. When developers iterate around public leaderboards, it becomes easy to optimize for benchmark performance rather than capability itself. Closed-source models may use stricter data governance, reducing the probability that evaluation sets enter their training data.
• A shift in task formats. Real code integration and terminal workflow tasks require long-horizon planning, tool use, and result verification. Closed-source models often benefit from larger investments in training data scale and quality, which may help them adapt to these complex tasks.
• Greater robustness from private evaluations. Private test sets can reduce scores based on memorization, but they cannot completely eliminate distribution differences. Open model weights are publicly available, allowing the community to perform extensive analysis. However, improving real-world capability still requires substantial investment in data engineering and evaluation governance.

4. Does a Leaderboard Drop Mean a Model Has Become Less Capable?

Not necessarily.

A change in leaderboard position only shows that a model’s effective score was recalibrated under a new evaluation protocol. If a model falls from fourth place to ninth, that does not mean the model itself has become weaker. It may mean that a larger portion of its previous score came from factors that did not reflect general capability.

Public questions in the previous version allowed models to encounter answers directly through training data. As a result, their scores may have overstated their actual performance. Replacing those questions with private ones can make scores more reliable and comparable.

However, private evaluations also require caution. Because the test set is not transparent, it is impossible to completely rule out indirect leakage or evaluation criteria that favor the habits of a particular type of user. Evaluation results remain an important signal, but they should not be treated as the final definition of intelligence.

5. How to Use Private Evaluations for Model Selection and Optimization

For developers and enterprise users, it is more useful to build a small, reliable internal evaluation set than to focus exclusively on leaderboard positions.

The following steps provide a practical starting point:

1. Sample from real business tasks. Extract 50 to 100 representative tasks from logs, support tickets, code repositories, customer-service conversations, or other operational data. Avoid common prompt templates found online.
2. Package the private data. For each task, define the input, expected output, and evaluation logic. Run the test set in a private evaluation environment that is not publicly exposed.
3. Use multidimensional evaluation. In addition to checking whether the final answer matches expectations, evaluate required tool calls, intermediate results, exception handling, and other process-level requirements.
4. Rotate the questions regularly. Replace approximately 20% of the test set each month to reduce the risk that a model or service provider develops an effective memory of the evaluation data.
5. Track cost and latency. Accuracy is not the only relevant metric. Usability per unit cost, response speed, and failure rate also determine whether a model is valuable in production.

The following example illustrates a lightweight approach to building a private evaluation set:

def make_private_testset(business_samples):
    testset = []
    for item in business_samples:
        testset.append({
            "input": item["prompt"],
            "expected": item["expected_output"],
            "checks": item["verifiers"]  # Includes forbidden conditions and multi-step checks
        })
    return testset

6. Conclusion: Bringing Evaluation Back to Real-World Tasks

The value of Intelligence Index v5 is not simply that it produces a new ranking. Its greater contribution is that it forces the industry to reconsider what effective capability actually means.

AI models are becoming increasingly capable on standardized question-answering tasks. Real-world problems, however, are often ambiguous, multi-step, and constrained. A model must do more than provide a standard answer. It must be able to modify a line of code in a repository, complete an end-to-end process in a terminal, and solve a previously unseen user problem in a business context.

For developers, the smarter strategy is not to chase leaderboard positions. It is to take control of evaluation by building private test sets around real tasks and updating them regularly. “Usable” should be the baseline, while “reliably useful” should be the goal.

Only when evaluation logic moves closer to real-world work can an intelligence index provide meaningful guidance.