How to Evaluate Breakthroughs in New General-Purpose AI Models Rationally

Every time a new generation of general-purpose AI models is released, public discussion tends to polarize quickly into two extreme views: one holds that artificial general intelligence has already arrived, while the other attributes all improvements to evaluation tricks or marketing hype. For developers and enterprise users, both judgments are overly simplistic.

Measuring the real value of a model upgrade cannot rely solely on the top score on some leaderboard, nor should the limitations of benchmarks be used to deny all progress. A more reasonable approach is to break public results down into verifiable technical questions: what exactly does the test measure, under what operating conditions was it run, has the model seen similar data, can the results be independently reproduced, and can these capabilities transfer to real business scenarios?

1. Near-Perfect Scores Do Not Mean the Problem Is Solved

When a model achieves near-perfect results on abstract reasoning, research-level mathematics, or vulnerability analysis tests, the first thing it demonstrates is that the model performs well on specific tasks under specific evaluation conditions. But this conclusion cannot be automatically extrapolated to mean “the model already possesses general intelligence.”

A complete evaluation result is determined by at least the following factors:

Final score = base model capability × prompting and interaction strategy × inference budget × tool environment × scoring rules × data independence

A change in any one of these can significantly affect the final result. For example, the same model may show notable score differences under different input formats, retry counts, and answer extraction rules. If the evaluation allows the model multiple attempts, environmental feedback, and iterative correction, what is being measured is not one-shot reasoning ability but the combined capability of the model, the execution system, and the feedback mechanism.

Therefore, when seeing a high score, the following questions should be confirmed first:

1. Is it a single-attempt score or the best score after multiple samples?
2. Is the model allowed to call code execution, retrieval, calculation, or verification components?
3. How much compute and inference time can each question consume?
4. Is intermediate feedback provided, allowing the model to re-plan based on results?
5. Is scoring done by fixed rules or judged by another model?
6. Is the test set public, and could the training data contain similar content?
7. How are failures, timeouts, format errors, and refusals scored?

Without this information, a seemingly precise percentage is really only an observation under limited conditions.

2. The Evaluation Framework Changes Model Performance

Many controversies do not originate from the model itself but from the evaluation framework. Model output typically goes through prompt assembly, context management, result parsing, and answer matching. Some models tend to output short answers, while others provide longer derivations; if the answer parsing rules favor one format over another, the latter may be misjudged due to formatting issues.

Interactive tasks are even more affected by the framework. Suppose a task requires the model to observe states, execute actions, and make further decisions based on feedback in an unfamiliar environment. The evaluation system must then define at least:

• What environmental information the model can see;
• Whether the full history is retained at each step;
• The maximum number of steps allowed per task;
• Whether invalid actions can be retried;
• Whether output format is automatically corrected;
• Whether a task can be restarted after failure;
• Whether results across multiple runs are averaged or take the maximum.

If different models use different context compression strategies, action repair rules, or retry limits, their scores cannot be directly compared.

Developers can abstract the evaluation process into pseudocode like this:

for each task in evaluation_set:
    state = initialize(task)
    history = []
    budget = fixed_limit
    while not finished(state) and budget > 0:
        observation = observe(state)
        request = build_request(observation, history)
        action = model_generate(request)
        if not valid(action):
            record_format_error()
            break
        state = execute(action)
        history.append(observation, action)
        budget = budget - actual_cost(action)
    score = evaluate_final_state(state)
    save(score, cost, latency, error_type)

A truly fair comparison requires not only fixed tasks but also fixed budgets, termination conditions, error handling, and scoring methods. Otherwise, what is being evaluated may be two system engineering approaches rather than two models.

3. How to Interpret Leaps in Abstract Reasoning

Novel rule learning and abstract reasoning tests typically require a model to induce patterns from a small number of examples and then apply those patterns to new inputs. These tasks are closer to “learning on the spot” than ordinary knowledge Q&A, so they carry higher reference value.

If a new model shows substantial improvement on such tasks, it may indicate that the following capabilities have been strengthened:

1. Stronger State Modeling

The model no longer generates answers based solely on surface vocabulary but attempts to build internal objects, relationships, and transformation rules for the task. This is important for process automation, fault analysis, and complex operation planning.

2. More Efficient Use of Feedback

The model can adjust its strategy based on failure results rather than repeatedly generating similar answers. Intelligent systems in real work rarely succeed on the first try, so error-correction ability is often more critical than single-turn accuracy.

3. Improved Long-Horizon Planning

Complex tasks require breaking an overall goal into multiple steps and maintaining constraints during execution. If a model can reliably complete longer task chains, it shows progress in context management and plan updating.

But abstract reasoning tests are still compressed experimental environments. They usually have clear goals, finite states, and computable success conditions, whereas real business scenarios often involve ambiguous requirements, conflicting constraints, dirty data, and outcomes that cannot be immediately confirmed. Therefore, high scores in the lab cannot be directly equated with reliable autonomy in production.

4. What High Scores on Research-Level Mathematics Mean

The value of mathematics tests lies in the fact that answers are usually highly objective and reasoning processes are easier to inspect. If a model can consistently achieve excellent results on difficult math tasks, it is reasonable to infer that its symbolic manipulation, constraint tracking, problem decomposition, and conclusion verification abilities have improved.

However, “solving test problems” and “completing mathematical research” remain distinctly different.

Research activity requires not only giving answers but also:

• Discovering worthwhile problems;
• Determining whether a problem already has a known conclusion;
• Choosing appropriate definitions and theoretical paths;
• Constructing new methods and explaining their necessity;
• Checking implicit assumptions in proofs;
• Establishing relationships with existing results;
• Persistently adjusting direction through long periods of failure.

Test sets typically complete the hardest step on the model’s behalf: defining the problem. The model only needs to find a solution within given boundaries, whereas real research also includes posing, evaluating, and reformulating problems.

In addition, attention should be paid to whether scoring only checks the final answer. A model may accidentally reach a correct conclusion while providing a derivation with logical gaps. For high-stakes scenarios such as scientific research, finance, and engineering computation, answer accuracy alone is far from sufficient; proof checking, numerical recomputation, and independent verification are also needed.

5. A Perfect Safety Test Score Does Not Mean “Absolutely Safe”

A high score on vulnerability analysis or attack simulation tests usually indicates that the model can identify flaws, construct exploitation paths, or complete preset security tasks. This represents defensive value, but it also means the risk boundary needs to be reassessed.

Enterprises in particular need to distinguish three capabilities:

1. Vulnerability identification: discovering potential flaws and explaining trigger conditions;
2. Exploit construction: turning flaws into executable attack steps;
3. Scaled operation: automatically searching for targets and carrying out actions in batches.

The first two capabilities can assist security auditing, while the third may significantly lower the barrier to malicious operations. Even if a model scores highly in closed tests, it cannot be concluded that it can handle all real network environments. Real systems involve version differences, access control, network isolation, monitoring policies, and human intervention, making them far more complex than standardized tests.

Production deployment should include at least:

• Least privilege and short-lived credentials;
• Network access scope controls;
• Secondary confirmation for high-risk actions;
• Parameter validation for tool calls;
• Retention of input, output, and execution logs;
• Alerts for abnormal frequency and batch behavior;
• Independent sandboxes and resource quotas;
• Emergency stop and human takeover mechanisms.

The stronger the security capability, the higher the governance requirements typically are.

6. Identifying Data Contamination and Overfitting Risks

High scores may also be affected by training data contamination. Contamination does not only mean the model has seen the exact same questions; it also includes more subtle cases:

• Having seen rewritten versions of questions;
• Having seen detailed solutions or answer discussions;
• Having seen large numbers of similar questions generated from the same template;
• Having seen the generation rules for test tasks;
• Having been specifically optimized for a certain type of evaluation through subsequent training.

The most effective way to determine whether a model has truly mastered a capability is not to keep using the same public questions but to build dynamic, private, and variable evaluation sets. For example, keep the task principle but change the data, constraints, expression, and target combinations. If the model performs well only on fixed question surfaces and degrades noticeably with slight changes, what it has mastered is more likely pattern matching than stable capability.

Internal enterprise evaluation should also follow an important principle: separation of test sets and optimization sets. If the development team repeatedly reviews all failure samples and modifies prompts accordingly, the internal test set will gradually become a training set. The correct approach is to reserve a portion of final acceptance samples that remain invisible for a long time and to regularly add new tasks.

7. Judging Proximity to General Intelligence Requires More Dimensions

There is no universally accepted single threshold for general intelligence. Beating certain benchmarks only shows that existing tests are losing discriminative power; it does not by itself prove that general intelligence has been achieved.

A more complete judgment should cover at least the following dimensions:

Generalization

Can the model transfer methods learned in one domain to another, rather than relying on fixed templates?

Continual Learning

When facing new rules, can the model form a stable strategy through limited interaction while avoiding degradation of existing capabilities?

Long-Task Reliability

Even if single-step accuracy reaches a high level, errors can accumulate and cause failure in tasks spanning dozens of steps. The completion rate of the entire task chain must be examined.

Expressing Uncertainty

A reliable system should know when information is missing and proactively ask for clarification. Always giving confident answers does not indicate intelligence; it may instead amplify risk.

Causal Understanding

Does the model truly understand the influence relationships between variables, can it distinguish correlation from causation, and can it handle counterfactual questions?

Autonomous Error Correction

When results are inconsistent with expectations, can the model locate the source of the error rather than simply repeating the previous path?

Environmental Adaptability

When tools are unavailable, inputs are incomplete, or rules change temporarily, can the model adjust its plan while maintaining goal consistency?

Only when these capabilities simultaneously reach high stability in open tasks does discussing general intelligence become more meaningful.

8. Enterprises Should Build Their Own Validation Systems Rather Than Chase Leaderboards

For enterprises, the most important question is not “is this the strongest model in history” but “can it improve our business processes at acceptable cost and risk?”

A four-tier validation structure is recommended.

Tier 1: Offline Task Evaluation

Extract desensitized samples from real business operations, covering routine cases, long-tail cases, and adversarial inputs. Metrics should include not only accuracy but also refusal quality, format compliance rate, factual error rate, and consistency.

Tier 2: Shadow Running

Let the new model receive real requests without directly affecting production results. Compare its output with the existing process and observe stability, latency, and resource consumption.

Tier 3: Limited-Traffic Trial

Open access only to low-risk users or low-risk business functions, and restrict executable actions. Detect issues promptly through monitoring alerts, logging services, and human review.

Tier 4: Graduated Expansion

Determine the degree of automation based on task risk. Content organization can increase the proportion of automated execution, while changes to funds, permissions, security, and critical configurations should retain human confirmation.

A practical composite evaluation formula can be written as:

Net business benefit
= labor cost saved
+ new business value
- inference and infrastructure cost
- human review cost
- error remediation cost
- security and compliance risk cost

If a model is more capable but the cost per task rises sharply, latency is unacceptable, or substantial human review is still required, it may not be more suitable for production systems than a smaller model.

9. Which Engineering Metrics Developers Should Focus On

Public leaderboards usually emphasize accuracy, while engineering deployment should focus more on the following metrics:

• Task completion rate: whether the user’s goal is ultimately achieved;
• First-attempt success rate: the proportion of successes without retries;
• Average retry count: measuring stability and additional cost;
• End-to-end latency: including model generation and tool execution time;
• Cost per task: rather than only per-unit input/output pricing;
• Tool call success rate: whether parameters, permissions, and execution results are correct;
• Error recovery rate: whether failures can be self-corrected;
• Human takeover rate: how many tasks ultimately still require human handling;
• High-risk error rate: distinguishing general errors from unacceptable ones;
• Version regression rate: whether older tasks show capability degradation after an upgrade.

Be especially wary of “average scores masking severe failures.” A few percentage points of improvement on ordinary tasks cannot offset a single irreversible error in a critical task. Therefore, enterprises should report statistics by risk level rather than giving only an overall average.

10. Rational Conclusion: Acknowledge Breakthroughs While Retaining the Right to Verify

Faced with high scores from a new generation of models, the most reasonable attitude is neither to immediately declare that general intelligence has been achieved nor to reflexively assume that all results are untrustworthy.

If a model improves simultaneously across abstract reasoning, complex mathematics, and security analysis, this usually means that important advances have indeed occurred in underlying reasoning capability, training methods, or systems engineering, and these deserve serious study. But public scores are only the starting point of the evidence chain, not the end. Whether the evaluation framework is fair, whether inference budgets are consistent, whether data is independent, whether third parties can reproduce the results, and whether real tasks are stable—these questions together determine the actual meaning of a high score.

For ordinary users, the focus can be on whether new capabilities genuinely improve daily tasks, without being led by grand slogans. For developers, models should be treated as more capable but still error-prone system components, with risk controlled through constraints, verification, and monitoring. For enterprises, deployment strategy should start from business value, total cost, and responsibility boundaries, using small-scale validation, gradual rollout, and rollback readiness at any time.

What is truly worth looking forward to is not a near-perfect number appearing on some leaderboard, but whether a model can consistently deliver verifiable, controllable, and accountable results in complex, open, long-running, and high-risk real-world environments. Until these conditions are fully demonstrated, any conclusion that “a new era has arrived” should be met with technical openness and engineering prudence.