How to Rationally Evaluate Frontier LLMs and AGI Claims

Whenever a frontier large language model makes clear progress in mathematical reasoning, scientific tasks, software engineering, or security analysis, the question of whether society has entered the AGI era quickly becomes a focal point of discussion.

However, high scores presented at model launch are only the beginning of evaluation. They cannot directly establish that a model has general intelligence, nor can they prove that it can reliably take responsibility for production workloads.

A rigorous assessment must separate the model itself, the external execution system, the evaluation setup, and the business environment. Only by identifying where performance gains come from, where failures occur, and whether capabilities transfer across tasks can we determine whether a release represents a local breakthrough, better systems engineering, or a meaningful step toward general intelligence.

1. Separate Model Capability From System Capability

The final result of an LLM task is usually not produced by the model alone. It is the outcome of several interacting components. A complete intelligent task system may include:

1. A foundation model for understanding problems, generating plans, and making judgments.
2. A context management layer for preserving historical state and compressing long conversations.
3. A retrieval component for accessing documents, code, or knowledge bases.
4. An execution environment for running programs, operating on files, or calling business interfaces.
5. Feedback mechanisms for checking results, detecting errors, and retrying.
6. An orchestration policy that decides when to continue reasoning, roll back, or terminate.
7. Safety controls that limit permissions, isolate resources, and record actions.

As a result, the same foundation model can perform very differently under two conditions: answering once with a fixed prompt, versus operating with persistent state, execution tools, feedback, and repeated correction.

System performance can be represented conceptually as:

System Performance =
Model Reasoning Capability × Environment Feedback Quality × Orchestration Strategy × Available Resources × Verification Mechanisms

This is not a strict mathematical formula. It highlights an important fact: a high score may reflect the capability of a mature intelligent system, but it does not necessarily represent the native capability of the foundation model alone.

At a minimum, evaluations should report three types of results:

• Native results: Model performance under fixed prompts, limited context, and a single response.
• Augmented results: Performance after adding memory, retrieval, execution tools, and self-checking.
• End-to-end results: Overall performance in a real workflow, from receiving an objective to delivering an outcome.

If only the end-to-end result is shown without disclosing the system configuration, it becomes difficult to determine whether improvement came from the model itself or from additional compute, more attempts, and more sophisticated workflow orchestration.

2. Interpret Multi-Attempt Results Correctly

Many complex evaluations allow a model to make multiple attempts, treating the task as passed if any one attempt succeeds. This approach is useful for measuring whether a system can eventually solve a problem, but it should not be treated as a direct measure of single-call reliability.

Suppose the success probability of one attempt is (p), and attempts are independent. If the system is allowed (k) attempts, the theoretical probability of at least one success is:

[
P = 1 - (1-p)^k
]

Even a system with a relatively modest single-attempt success rate can achieve a strong final pass rate when it is allowed to generate multiple candidates and automatically select one that passes validation.

In real environments, however, attempts are often not independent. A model may repeatedly make the same type of mistake. Conversely, it may quickly correct itself after receiving execution feedback. Therefore, the formula above is still insufficient for characterizing real-world performance.

Enterprise evaluations should examine all of the following:

• First-attempt success rate
• Average number of attempts
• Total compute required to achieve success
• Whether human selection is needed to identify the correct answer
• Whether automated validators can accurately detect errors
• Whether repeated attempts introduce additional security risks
• Recovery cost and business impact after failures

For low-risk tasks with reliable automated validation, repeated attempts can be a reasonable strategy. Code compilation, format conversion, and some mathematical calculations can often be checked using deterministic rules.

For high-risk tasks such as financial operations, permission changes, or production configuration updates, the assumption that “more attempts will eventually succeed” cannot replace reliability requirements.

3. High Benchmark Scores Do Not Equal General Intelligence

Frontier models are improving rapidly on difficult tasks, especially in mathematics, scientific reasoning, program analysis, system diagnosis, and long-horizon planning. These results are important, but they are not enough to determine whether a system is approaching AGI.

Several additional dimensions must be tested.

3.1 Transfer to Unfamiliar Tasks

A genuinely general intelligent system should not only excel at familiar task types. It should be able to infer new rules from a small number of examples and transfer existing knowledge into unfamiliar environments.

Testing should not merely change numbers or names in existing questions. It should alter the structure of the task itself. For example:

• Change the rules connecting inputs and outputs.
• Introduce combinations of constraints that have not appeared before.
• Remove part of the available prompt information.
• Add irrelevant distracting information.
• Require the model to request missing conditions proactively.
• Ask the model to explain why a particular method does not apply.

If a model scores highly only on familiar templates but degrades significantly when the problem is expressed differently, it may be better described as a powerful pattern-matching and reasoning system rather than a stable general-purpose problem solver.

3.2 Consistency in Long-Horizon Tasks

Real tasks rarely consist of answering one question. They often require dozens of sequential steps. The model must maintain the objective, track intermediate state, handle exceptions, and avoid contradictions across the process.

Long-horizon evaluation should not focus only on whether the final objective was completed. It should also record:

• Whether the plan covers critical dependencies
• Whether intermediate steps drift away from the main objective
• Whether the model can recognize ineffective attempts
• Whether it can roll back after errors
• Whether it repeats operations that have already been completed
• Whether it forgets early constraints as context grows
• Whether the final result is independently verified

Many models perform well in single-step reasoning, but errors can accumulate over long tasks. Even if each individual step has a high probability of being correct, end-to-end success can decline rapidly as the task length increases.

Long-horizon intelligence is therefore not only about planning. It also requires state management, error detection, and recovery capability.

3.3 Balanced Performance Across Domains

A model may make substantial progress in mathematics and scientific tasks while users see little noticeable improvement in ordinary writing, interface development, requirements clarification, or knowledge-based question answering. This is not necessarily contradictory.

The bottlenecks of everyday tasks are often not the upper limit of reasoning. They may instead include:

• Incomplete user requirements
• Subjective evaluation criteria
• The need for extensive domain context
• Constraints involving format, tone, or organizational standards
• Older models already being “good enough”
• New capabilities not being effectively invoked through the interaction workflow

As a result, major capability gains may appear first in research, engineering, and complex decision-making rather than in simple question answering. If high-capability models are evaluated only with low-difficulty tasks, observers may conclude that benchmark gains are large while practical improvements appear limited.

4. Why Research-Oriented Intelligence May Be Closer to a Key Breakthrough

Compared with single-turn question answering, real scientific research and complex engineering demand a broader combination of capabilities. A model must not only understand natural language but also formulate hypotheses, analyze information, design experiments, operate in an environment, interpret results, and correct mistakes.

A typical research-oriented workflow can be represented as follows:

Receive a research objective
  ↓
Identify known conditions and unknown variables
  ↓
Propose multiple candidate hypotheses
  ↓
Design verifiable experiments or computational steps
  ↓
Execute in an isolated environment
  ↓
Analyze outputs and identify anomalies
  ↓
Revise hypotheses or parameters
  ↓
Repeat validation
  ↓
Produce a conclusion supported by an evidence chain

This is much more difficult than simply generating an answer because it requires sustained action under uncertainty. If a model can complete multi-step analysis in an unfamiliar tool environment and adjust its strategy based on feedback, its capability is expanding from language generation toward goal-directed problem solving.

Caution is still necessary. Mathematical conclusions, experimental outputs, and security findings produced with model assistance must be independently verified. A model may generate a formally complete but subtly flawed derivation, or it may misinterpret experimental output.

The value of research capability should not be confirmed because “the model says it proved something.” It should be confirmed through reproducible processes, formal checks, peer review, and independent experiments.

5. What Is Required to Approach AGI?

There is currently no single, universally accepted, directly executable criterion for determining whether a system has achieved AGI. Treating one benchmark score as the boundary for AGI creates obvious problems: benchmarks can saturate, training data may contain similar patterns, and evaluation workflows may be optimized for specific tests.

A more reasonable approach is to use a multidimensional evaluation framework.

5.1 Breadth

Can the system cover multiple domains, including language understanding, mathematical reasoning, software engineering, scientific analysis, visual understanding, planning and decision-making, and social common sense? Or does it only perform strongly on a narrow set of tasks?

5.2 Generalization

Can the system handle task structures not seen during training and adapt quickly when rules change? Generalization is closer to the core of general intelligence than simply memorizing a large amount of knowledge.

5.3 Autonomy

Can the system formulate plans, choose actions, use feedback, and complete deliverables around a goal without requiring humans to prompt it step by step?

Greater autonomy also makes permission control and safety governance more important.

5.4 Reliability

The system must do more than occasionally produce correct results. It should know when it is uncertain, when more information is needed, and when it should stop acting.

For enterprises, predictable and stable performance is usually more valuable than a small number of impressive demonstrations.

5.5 Efficiency

If a system requires extremely high compute costs, long waiting times, and many repeated attempts to complete a task, its capability may not yet be practical for broad deployment.

Evaluation of general intelligence should include compute consumption, response latency, and resource usage.

5.6 Verifiability

The system should provide enough process information for auditing, such as cited data, execution records, result validation, and explanations of anomalies.

It is important to note that a model’s natural-language explanation is not necessarily an accurate representation of its internal reasoning process. A fluent explanation should not be treated as direct evidence of trustworthiness.

5.7 Safety

Whether a system can perform unauthorized actions, expose sensitive data, generate dangerous procedures, or behave unpredictably when objectives conflict cannot be ignored in AGI discussions.

The more capable a system becomes, the more important safety evaluation becomes.

Only when a system demonstrates stable performance across these dimensions and can solve a broad range of new problems in open environments is there a stronger basis for discussing whether it has reached a level of general intelligence.

6. Why High-Scoring Benchmarks Can Lose Their Ability to Differentiate Models

When models approach perfect scores on a test, that test often becomes less useful for comparing stronger systems. This may happen for several reasons:

• The number of questions is limited, increasing statistical variation.
• The task type is too narrow.
• Test data may resemble training materials.
• The scoring rule checks only final answers rather than the process.
• Multiple attempts and external validation inflate final scores.
• The test difficulty is below the model’s current capability ceiling.
• The system can exploit unintended signals in the evaluation environment.

Next-generation evaluations should place greater emphasis on dynamic and adversarial conditions. Examples include generating new rules at evaluation time, hiding part of the environment information, randomly changing task interfaces, and using private test sets that were unavailable during training.

Models should also face decision pressure in situations where errors cannot be automatically recovered. Evaluators can then observe whether the model is able to stop proactively and request authorization.

However, adversarial evaluation does not mean deliberately designing traps. A high-quality evaluation should measure the intended capability, not whether the model can guess the evaluator’s preferences.

7. How Developers Can Design Their Own Evaluation Systems

Enterprises should not select models based solely on public leaderboards. Instead, they should create internal evaluation sets around their own tasks.

Step 1: Build a Task Matrix

Classify tasks by business value and risk level. Examples include:

• Information extraction and classification
• Document summarization and question answering
• Code generation and code review
• Data analysis and reporting
• Customer support assistance and process recommendations
• System diagnosis and operations tasks
• Scientific computing and knowledge discovery

Each category should include standard samples, boundary cases, abnormal cases, and adversarial cases. This prevents the evaluation set from covering only ideal conditions.

Step 2: Define Multidimensional Metrics

Do not rely on accuracy alone. At a minimum, record:

• Task completion rate
• First-attempt success rate
• Percentage of output requiring human modification
• Factual error rate
• Whether refusals are appropriate
• Average response time
• Average resource consumption
• Number of high-risk errors
• Long-task interruption rate
• Reproducibility of results

For generative tasks, rule-based checks, model-assisted scoring, and human review can be combined. However, high-risk conclusions should always retain human review.

Step 3: Distinguish Model Configurations

Evaluation reports should clearly record context length, prompt templates, whether retrieval is used, whether state is retained, the number of permitted attempts, available resources, and termination conditions.

Without this information, different results cannot be compared fairly.

Step 4: Use Hidden Test Sets

Development teams should not continually tune prompts against public samples, as this can lead to internal overfitting. Hidden test sets should be updated regularly and include new business scenarios, new ways of expressing requests, and real failure cases.

Step 5: Conduct Staged Validation

Before entering production, models should initially operate in read-only and low-permission modes. They may provide recommendations without directly taking action.

Only after accuracy, stability, and auditability meet requirements should limited permissions be gradually introduced.

8. Security Challenges Created by More Capable Models

As models become stronger at program analysis, system diagnosis, and vulnerability discovery, they can help defenders but may also lower the barrier to using high-risk technical capabilities.

Evaluation should therefore focus not only on whether the system can complete a task, but also on the permissions under which it completes that task.

Organizations should follow these deployment principles:

1. Least privilege: Grant only the minimum permissions required for the current task.
2. Environment isolation: Run generated programs and operations in controlled environments.
3. Sensitive data minimization: Avoid placing credentials, personal data, or core business data directly into the context.
4. Operational auditing: Preserve records of inputs, outputs, resource calls, and result changes.
5. Approval for high-risk actions: Require human confirmation for deletion, publication, transfers, permission changes, and similar actions.
6. Resource limits: Restrict compute, network access, file access, and task runtime.
7. Exception termination: Stop immediately when the model repeatedly fails, drifts from its objective, or shows intent to exceed its permissions.
8. Independent validation: Use separate rules or independent systems to check critical results.

These governance measures are not a rejection of intelligent capability. They are necessary conditions for deploying powerful systems in production environments.

9. A Rational Conclusion on “Entering the AGI Era”

If frontier LLMs achieve major advances in unfamiliar-rule induction, complex mathematics, scientific workflows, program analysis, and long-horizon execution, they do indicate that AI is moving from content-generation tools toward general-purpose problem-solving systems.

In particular, when a model can plan proactively, operate within an environment, and revise its strategy based on feedback, it is structurally closer to an intelligent agent than a traditional question-answering system.

However, “approaching AGI” and “having achieved AGI” are not the same conclusion.

A near-perfect score on a single benchmark may mean that the benchmark has saturated. A high system-level score may include the contribution of memory, execution tools, verification, and repeated attempts. Laboratory success does not necessarily translate into stable reliability in production environments.

The most rational response to AGI claims is therefore neither immediate acceptance because a demonstration is impressive nor wholesale dismissal because everyday question-answering feels unchanged.

A more professional evaluation asks:

• Does the result come from the model itself or from the complete system?
• Are multiple attempts and human selection allowed?
• Does the evaluation genuinely cover unfamiliar tasks?
• Can the capability transfer across domains?
• Is long-horizon execution stable?
• Can failures be detected, recovered from, and audited?
• Does the cost support deployment at scale?
• Are powerful capabilities subject to effective safety controls?

If these questions can be answered transparently, reproducibly, and through independent validation, then discussions of AGI will have a stronger foundation.

For developers and enterprises, the more important priority is not debating whether a specific moment marks the beginning of the AGI era. It is building internal evaluation, validation, and governance systems that turn the upper bound of model capability into controlled, reliable, and genuinely valuable productivity.