Evaluating General-Purpose AI Models: Capabilities, Benchmarks, and AGI Claims

In recent years, release notes for general-purpose AI models have increasingly used phrases such as “most intelligent” or “close to AGI.” For developers, the important question is not a marketing phrase but whether a model can complete complex tasks reliably: whether it can understand long-term goals, call tools, correct failures, handle real data safely, and reproduce these capabilities across different platforms, tasks, and runtime frameworks.

Therefore, evaluating a new model should not depend only on the single highest score reported by its vendor, nor should it rely on one composite leaderboard. A more reasonable approach is to split model capability into multiple layers and examine test design, runtime environment, task cost, stability, and safety boundaries separately.

1. Distinguish Three Types of Model Results

Three types of data usually appear when a model is released.

The first is vendor self-testing.
This type of data often selects tasks that best demonstrate the new model’s advantages. For example, in a computer-operation benchmark, a new model reached a 72.6% completion rate, compared with 65.7% for the previous generation. In terminal science tasks, the new model reached 64.6%, while the previous generation reached only 22.4%. In high-difficulty mathematical reasoning tasks, the score was close to 97.6%, while previously published results were around 22%. In an ultra-long-context retrieval task, accuracy in the 512K to 1M length range reached 96.3%, compared with about 73.8% for the previous generation.

These numbers can demonstrate clear progress in the corresponding directions, but they do not directly prove “highest general intelligence.” The reason is that a single benchmark covers only one type of capability, and test samples, prompts, tool permissions, retry counts, and scoring strategies can all affect the result.

The second is an independent organization’s composite index.
A composite index usually combines knowledge question answering, mathematical reasoning, code generation, long context, visual understanding, tool calling, and other items into one score. One new model scored 61 on an independent comprehensive evaluation, tied with the previous-generation model. A competing model scored 66, and other models reached 62.1 and 59 respectively.

This reveals a key issue: a jump in a single capability does not necessarily translate into a corresponding increase in the composite index. A new model may lead in computer operation and mathematical reasoning but fail to show the same magnitude of improvement in code reliability, factual consistency, response cost, or complex-task stability. A composite score reflects the structure of capabilities, not the peak of any one capability.

The third is real workflow performance.
When developers actually use a model, they often do not submit an isolated question. They ask the model to execute dozens or even hundreds of steps in sequence: analyze a code repository, modify multiple files, run tests, locate errors, read logs, generate patches, and then adjust again based on test results. At this point, the model’s final value depends on the success rate of the entire task chain, not on whether one answer is correct.

2. Why Official High Scores and Third-Party Results Can Diverge Sharply

The most typical case is a model that achieves 99.9% on a specialized test but only 62.7% under a standardized runtime framework. This gap does not automatically mean that either side is falsifying results. It means that the two tests are measuring different things.

1. The Runtime Framework Can Change Task Difficulty

The same model can perform completely differently under different conditions:

• Whether the model is allowed multiple attempts;
• Whether it can automatically call search, code execution, or file-operation tools;
• Whether it can save intermediate notes;
• Whether it can read the full historical context;
• Whether an external program automatically plans the next step;
• Whether it can roll back and re-execute after a failure;
• Whether the final result or the entire process is scored.

If a test framework allows a dedicated task scheduler, additional state management, and customized prompts, then the final score reflects not only the model itself but also the combined capability of the model and the runtime system.

2. Task Distribution May Be Biased

High-score benchmarks often concentrate on areas covered by model training and optimization priorities. A model may be very familiar with certain question types but degrade rapidly when facing out-of-distribution tasks. Therefore, evaluation should not rely only on a fixed question bank. It should also include dynamically generated tasks, hidden test sets, and cross-domain tasks.

For enterprise applications, “non-standard input” deserves particular attention. Requirements provided by real users often contain ambiguity, missing context, inconsistent formats, and changing goals. A model that scores highly only on clearly structured questions may not be suitable for direct deployment in production.

3. Composite Scores Can Mask Weaknesses

A composite index is suitable for horizontal comparison, but it cannot replace scenario-specific evaluation. Suppose a model is near perfect on mathematical tasks and performs well on code tasks, but frequently misinterprets business rules. Its practical value to a software team may still be limited.

Enterprises should therefore break the total score into a capability matrix that covers at least the following:

Capability Dimension	Questions to Examine
Task understanding	Can it identify goals, constraints, and implicit conditions?
Planning ability	Can it split long-term tasks and arrange reasonable steps?
Tool calling	Can it select tools correctly and handle returned results?
Coding ability	Can it generate, modify, test, and maintain code?
Factual reliability	Does it fabricate nonexistent results or evidence?
Context handling	Can it remain consistent across long conversations and large projects?
Self-correction	After discovering an error, can it locate the cause and adjust its strategy?
Safety	Can it identify high-risk operations and apply constraints?

3. Which Capabilities Are Most Valuable to Developers?

1. From “Generating Code” to “Completing Engineering Tasks”

Code generation is only one part of software development. A truly valuable model should be able to understand repository structure, identify dependencies, modify multiple files, execute tests, analyze errors, and roll back changes when necessary.

To evaluate this type of capability, design a complete task:

1. Provide a real but desensitized project;
2. Specify a task that includes functional requirements and boundary conditions;
3. Ask the model to output an implementation plan first;
4. Allow the model to read files and execute tests;
5. Record the number of modifications, failed steps, human interventions, and the final pass rate;
6. Repeat the run multiple times and calculate stability.

The core metrics here are not “whether the first generation is correct” but:

• Task completion rate;
• Average repair rounds;
• Invalid tool-call ratio;
• Number of unintended file modifications;
• Test pass rate;
• Human takeover frequency;
• Time and resource cost per task.

2. More Efficient Context Management

Long context does not equal true understanding of long documents. A model can receive a large amount of content but still miss key constraints or forget earlier decisions after multiple rounds of operations.

More reliable systems usually adopt structured memory mechanisms, such as:

• Saving task goals and non-negotiable constraints;
• Recording completed steps;
• Saving test results and failure reasons;
• Creating traceable explanations for important decisions;
• Separating temporary logs from long-term knowledge;
• Summarizing when the context approaches its limit while preserving key evidence.

The value of such mechanisms is reducing information loss caused by context compression. For development tools, a memory system should not simply truncate old messages. It should form a task trajectory of “goal-action-result-correction.”

3. Clearer Safety Boundaries

If a model can execute terminal commands, access the network, modify files, or operate production resources, it cannot pursue only “completing as many tasks as possible.” When permission escalation, sensitive data, remote execution, credential handling, or critical infrastructure are involved, the model should have mechanisms to proactively block or request confirmation.

Safety evaluation should cover at least:

• Whether it identifies high-risk instructions;
• Whether it distinguishes test environments from production environments;
• Whether it leaks keys, tokens, or personal information;
• Whether it can refuse clearly malicious or unauthorized operations;
• Whether it explains the scope of impact before a tool call;
• Whether it supports least privilege and human approval.

“Completing more tasks” does not necessarily mean “more intelligent.” In high-risk scenarios, knowing when to stop, when to ask, and when to refuse is also an important capability.

4. How to Understand the “AGI Era” Rationally

“AGI” has different meanings in different contexts. Sometimes it refers to a model capable of handling a broad range of tasks; sometimes it refers to a system approaching human-level general cognition; and sometimes it is merely promotional language in a launch event. Without clear capability standards, the term is difficult to use directly for technical judgment.

From an engineering perspective, a more operational definition should include several conditions:

1. It can transfer across domains, rather than performing well only on fixed question types;
2. It can maintain goals and constraints over the long term;
3. It can plan autonomously in uncertain environments;
4. It can use tools to obtain information and verify results;
5. It can discover errors, explain errors, and correct them;
6. It can run stably at an acceptable cost;
7. It can take on real work within safety boundaries.

Under this standard, even if a model achieves new top scores on several specialized tests, that only shows a breakthrough in local capabilities. AGI cannot be declared on the basis of a single score.

In addition, legal, commercial-contract, and technical definitions must be discussed separately. Some historical agreements once treated “AGI” as a special condition, but the relevant clauses may later have been deleted, modified, or reinterpreted. Such changes do not equal a technical leap in model capability, and they cannot replace public, reproducible capability verification.

5. What Evaluation Process Should Enterprises and Individuals Adopt?

A three-layer evaluation system is recommended: public benchmarks, private task sets, and real workflows.

The first layer is public benchmarks, used to understand a model’s approximate position in mathematics, code, knowledge, long context, and other areas. This layer is suitable for initial screening but should not be used directly as a procurement basis.

The second layer is a private task set. Enterprises can extract desensitized samples from historical tickets, code reviews, document processing, and data analysis tasks to build a representative internal test set. The test set should be updated regularly to prevent models from overfitting to fixed questions.

The third layer is a real workflow trial. Let the model complete full tasks in a controlled environment, and record success rates, human intervention, resource consumption, safety incidents, and user satisfaction. Only when a model performs stably across all three layers is it suitable for broader use.

Ultimately, evaluating a new generation of models should follow three principles: do not look only at the highest score, do not ignore the runtime framework, and do not treat promotional terminology as a technical conclusion. Whether a model is “the most intelligent” is not an absolute judgment detached from context. It depends on what tasks it performs, at what cost, and under what constraints it can complete work reliably and consistently.