Evaluating Frontier Models’ AGI Claims: A Practical Framework for Assessing General Intelligence

In recent years, the capability gains of frontier general-purpose models have increasingly shown up in complex reasoning, interface operation, code generation, and coordination with professional software. When a new model scores near the top on multiple difficult benchmarks and is described as “entering the AGI era,” the public can easily jump to a straightforward conclusion: machines have already achieved general intelligence approaching human level.

From a technical evaluation standpoint, however, a high score on a single benchmark, or leadership on a leaderboard, does not directly equal AGI. Whether a model’s capabilities have undergone a genuine qualitative shift requires examining evaluation design, runtime environment, tool permissions, task distribution, cost efficiency, stability, and safety boundaries at the same time. In agentic tasks especially, the evaluation framework can have an enormous impact on model performance. Without consistent conditions, the numbers themselves may not even be comparable across systems.

1. First, Recognize That AGI Is Not a Single Metric

AGI generally refers to a general-purpose intelligence system capable of understanding problems across tasks and domains, learning in unfamiliar environments, reasoning, planning, and completing goals. It includes at least the following categories of capability:

1. Broad generalization

   A system should not only solve problems within its training distribution but also handle unseen rules, interfaces, and task combinations. True generality should not depend on preparing large numbers of similar cases in advance, nor should it work only on fixed templates.

2. Continual learning

   When facing unfamiliar tasks, a system needs to adjust its strategy through trial and error, feedback, and environmental change, rather than starting completely over after every failure. The key here is not “remembering a longer conversation” but whether reusable intermediate experience can be formed.

3. Long-horizon planning

   Many real tasks involve dozens or even hundreds of steps. A system must maintain goals, constraints, and progress, identify deviations, and replan after partial failures. Generating a correct answer in one pass and executing reliably over a long period are two different capabilities.

4. Cross-modal environment understanding

   Models need to jointly process text, images, interface states, file structures, and operational feedback. Recognizing screen elements is not enough; the system must also understand how those elements relate to business goals.

5. Reliability and controllability

   If a system can occasionally complete highly difficult tasks but frequently makes unpredictable errors, it is hard to entrust with critical production responsibilities. Evaluating general intelligence must include failure probability, error types, and the cost of human takeover.

6. Safety and responsibility boundaries

   The stronger the capability, the greater the potential impact. Whether the system respects permission constraints, can identify high-risk operations, and provides audit records are all important conditions for judging whether it can enter real business operations.

AGI is therefore better understood as a multidimensional set of capabilities, not a label automatically earned when some benchmark reaches a certain score.

2. Why “Near-Perfect Benchmark Scores” Still Require Careful Interpretation

2.1 Benchmark Sets May Already Be Saturated

When a model approaches a perfect score on a benchmark, the first conclusion is that the model has adapted well to that task—but not necessarily that the problem itself has been fully solved. Benchmark saturation usually has three possible explanations:

• The model’s capability has genuinely broken through;
• The benchmark is too small to distinguish stronger models;
• The task format substantially overlaps with the model’s training distribution.

Once a test set has been publicly analyzed many times, model development pipelines may optimize around its characteristics. Even without directly using test answers, scores can be indirectly raised through similar tasks, prompting strategies, and tool configurations.

A more reasonable approach is to introduce continuously updated hidden tasks, use test environments with different difficulty levels, domains, and interaction mechanisms, and observe whether the model can transfer stably.

2.2 The Evaluation Framework Can Significantly Change Results

Agent evaluation is not just comparing models; it is comparing complete systems composed of a model plus an evaluation framework. The following factors all affect final scores:

• Whether the full message history is retained;
• Whether the model is allowed to read previous reasoning summaries;
• Whether automatic retries after failure are supported;
• Whether structured information about page elements is provided;
• Whether additional tools are allowed;
• Whether a dedicated prompt template is used;
• How much compute a single task may consume;
• Whether parallel exploration of multiple paths is allowed.

If the same model shows dramatically different scores under a standard framework versus a customized one, that usually means systems engineering contributes significantly to the result. In that case, not all improvements can be attributed to model parameters or reasoning capability.

A fair comparison should at minimum specify four conditions: the same tasks, the same tools, the same resource budget, and the same success criteria. Otherwise, scores from a system that retains full context and allows multiple retries are not directly comparable to those from a system with no memory and a low budget.

2.3 A Single Best Result Does Not Represent Stable Capability

Model evaluation is stochastic. Temperature settings, initial prompts, task order, and environment latency can all change results. Production scenarios care less about “what the best single run can do” and more about “how reliable the system is under continuous operation.”

At minimum, the following statistical metrics should be reported:

• Average success rate across multiple runs;
• Range of score variation;
• Median task completion time;
• Failure cases at the worst percentile;
• Average compute cost per successful task;
• Proportion of tasks requiring human intervention.

Only when capability, efficiency, and stability are presented together do evaluation results have engineering decision value.

3. Why Computer-Operation Capability Deserves Close Attention

Compared with text-only output, the ability to operate a computer means a model begins to shift from a “suggestion generator” to a “task executor.” It can recognize interface states, click controls, edit files, invoke professional software, and continue adjusting based on execution feedback.

The significance of this shift is that a large amount of enterprise knowledge work has no complete API and depends on graphical interfaces and manual operation. For example:

• Organizing data and generating reports in office systems;
• Modifying engineering files in design software;
• Completing data entry and review in business backends;
• Locating errors and running tests in development environments;
• Placing components, checking connections, and exporting manufacturing data in electronic design workflows;
• Reconstructing 3D structures from multi-view images.

Traditional automation usually relies on fixed scripts that break as soon as an interface or process changes. Agents with visual understanding and dynamic planning may be able to adjust actions based on real-time state.

However, “can operate” does not equal “can run unsupervised.” Real environments also include pop-ups, network fluctuations, insufficient permissions, interface updates, and data anomalies. A task completed successfully in a demo is still a long way from stable handling of long-term business operations.

4. Which Metrics Matter for Evaluating Computer-Operation Capability

4.1 Task Success Rate

Whether the final business goal is achieved should be verified by environment state or result files, not by the model’s own claim that it is “done.” For example, after generating an engineering file, one must also check whether the file opens correctly, whether the structure is valid, and whether constraints are satisfied.

4.2 Operational Efficiency

A model may complete a task through extensive trial and error, but that is not necessarily commercially valuable. The following should be measured:

• Number of operation steps;
• Number of page round trips;
• Proportion of invalid clicks;
• Total elapsed time;
• Reasoning resource consumption;
• Anomaly recovery time.

High success rate with excessive time consumption may only be usable for low-frequency tasks and cannot directly replace manual processes.

4.3 Error Recovery Capability

A mature agent needs to recognize whether an operation took effect. For example, if the page does not change after a click, it should determine whether the cause is delay, occlusion, or a permission issue, rather than repeating the same action. A good recovery process typically includes:

1. Reading the current environment state;
2. Comparing expected state with actual state;
3. Determining the source of deviation;
4. Choosing to retry, roll back, or replan;
5. Requesting human confirmation when risk increases.

4.4 Safety Constraints

Before irreversible operations such as deletion, payment, publishing, file overwriting, or permission changes, the system should enter a confirmation flow. Tiered control can be used:

• Low-risk actions execute automatically;
• Medium-risk actions display a plan before execution;
• High-risk actions require human authorization;
• Extremely high-risk actions are prohibited by default.

The importance of such mechanisms is no lower than the model’s own reasoning capability.

5. Do High Scores on Professional Tasks Mean Experts Can Be Replaced?

In engineering design, 3D modeling, software development, and data analysis, models may generate structured outputs from text or images. If evaluation only checks final appearance, model capability is often overestimated. Professional deliverables must satisfy many more constraints at the same time.

Take 3D design as an example: a result that “looks similar” does not mean it has production value. One should also check:

• Whether the geometry is closed;
• Whether dimensions and proportions meet requirements;
• Whether the object hierarchy is reasonable;
• Whether materials and lighting settings are maintainable;
• Whether there are duplicate faces or abnormal topology;
• Whether subsequent modification is convenient;
• Whether the output file meets delivery specifications.

Electronic design is similar: it is not enough to check whether connections exist. Electrical rules, component spacing, thermal management, signal integrity, and manufacturing constraints must also be considered. A model-generated draft may significantly improve efficiency, but final review still requires professional responsibility.

In the near term, therefore, frontier models are more likely to change the division of labor in professional work: models handle information organization, draft generation, and repetitive operations, while experts handle goal definition, constraint confirmation, exception handling, and final sign-off. What they compress first is low-value operation time, not professional responsibility itself.

6. Exploit Capability Is Both a Breakthrough and a Risk Signal

When a model can more accurately identify system flaws, construct test paths, and verify issues, it has positive value for defensive work. Security teams can use it to assist with code review, attack surface analysis, and fix verification.

But this capability is clearly dual-use. Without restrictions, it could equally be used for automated probing and attacks. Evaluation of such capabilities should therefore not look only at completion rate but also at:

• Whether obviously malicious requests are refused;
• Whether active operations against real targets are restricted;
• Whether the test environment is isolated;
• Whether tool calls are audited;
• Whether network and file permissions are controlled;
• Whether sensitive data can be identified;
• Whether immediate abort and rollback are supported.

When deploying such models, enterprises should adopt the principle of least privilege so the model can access only the resources necessary to complete the task. All high-risk calls should record request parameters, execution results, initiator, and approval status, forming a traceable audit chain.

7. How to Build a More Reliable AGI Evaluation Framework

A relatively complete evaluation system can be divided into six layers.

Layer 1: Foundational Capabilities

Assess language understanding, knowledge Q&A, mathematical reasoning, code generation, and multimodal recognition. This layer is suitable for measuring the lower bound of capability but is insufficient to prove autonomy.

Layer 2: Generalization to Unfamiliar Tasks

Provide unpublished rules and new environments, and observe whether the model can master the task through a small amount of feedback. Testing should control for the influence of similar training samples.

Layer 3: Long-Horizon Agentic Tasks

Require the model to run continuously for an extended period and complete tasks containing multiple subgoals. Focus on plan maintenance, state management, failure recovery, and context compression.

Layer 4: Real Professional Work

Have domain experts design tasks and accept results according to industry quality standards, rather than comparing only text similarity or visual similarity.

Layer 5: Reliability and Economics

Combine success rate with resource cost to calculate cost per effective task. Even if a model is more capable, it will be difficult to deploy at scale if cost and latency are unacceptable.

The following general metrics can be used:

Cost per effective task = Total operating cost ÷ Number of tasks that succeed and pass quality acceptance
Labor savings rate = 1 - Human time in agent workflow ÷ Human time in original workflow
Risk-adjusted return = Automation benefit - Expected loss from errors - Review and governance cost

Layer 6: Governance and Social Adaptability

Test whether the system complies with permissions, privacy, law, and organizational policy. If AGI is to become productivity infrastructure, it must have operating mechanisms that are auditable, interruptible, explainable, and accountable.

8. How Enterprises Should Validate a Model’s Real Value

Enterprises should not decide whether to adopt a model based directly on public leaderboards. Instead, they should build their own task sets.

The first step is to select real business samples covering common tasks, complex tasks, and high-risk boundary tasks. The second step is to define verifiable success conditions, such as whether a file is generated, whether data is correct, and whether a process is closed-loop. The third step is to set a unified resource budget to prevent models from obtaining inflated scores through unlimited retries. The fourth step is to introduce blind human evaluation to avoid judging complex deliverable quality solely by automated metrics.

A basic evaluation process can be represented as:

Input real task
  → Set permission and resource limits
  → Record model plan and tool calls
  → Execute in isolated environment
  → Automatically check results
  → Expert quality review
  → Measure success rate, time, cost, and risk
  → Decide: automatic execution, assisted execution, or prohibited execution

During deployment, a gradual strategy should also be adopted: first observe read-only, then allow suggestion generation; afterward open low-risk operations, and only finally consider limited autonomous execution. All stages should retain a human takeover entry point.

9. A More Accurate Judgment on “Entering the AGI Era”

If frontier models show clear improvements simultaneously in unfamiliar interactive tasks, mathematical reasoning, computer operation, and professional work, this may indeed mean the technology paradigm is changing. Models are no longer just generating content; they are beginning to understand environments, invoke tools, and complete multi-step goals.

From a rigorous perspective, however, a more appropriate statement is: intelligent systems are entering a “high-capability agent” phase, rather than AGI having been fully achieved based on a few near-saturated benchmarks.

Truly convincing evidence should include:

• Stable generalization on unseen tasks;
• Low failure rates in long-horizon execution;
• Autonomous adaptation to environmental change;
• Efficiency advantages maintained under a unified budget;
• Compliance with permission and safety constraints on critical tasks;
• Quality that remains high after acceptance by domain experts;
• Capabilities that can be reproduced continuously, not only in selected cases.

Conclusion

Breakthroughs in frontier models deserve attention, especially computer-operation and professional-tool coordination capabilities, which may change work methods more profoundly than question-answering scores alone. However, a high leaderboard score is only a starting point, not final proof that AGI has arrived.

For developers, the focus should shift from “can the model answer” to “can the system complete tasks reliably.” For enterprises, the focus should shift from “does it achieve the highest score” to “can it create sustained value under cost, quality, and risk constraints.” For the industry as a whole, transparent, unified, and reproducible evaluation standards are needed.

Judging whether AGI has arrived cannot rely on a declaration. It should rely on system evidence that withstands repeated testing, real environments, and long-term operational validation.