How to Evaluate the Capability Limits of Frontier LLMs Beyond Public Benchmarks

When a new generation of general-purpose large models achieves higher scores on mathematical reasoning, software engineering, computer operation, cybersecurity, and professional knowledge tasks, it is easy to conclude that it is “the most intelligent model available.” From a technical evaluation perspective, however, “highest intelligence” is not a proposition that any single score can directly prove.

A model’s lead on a public test shows, at most, that it performs well under specific data, specific rules, and specific operating conditions. To determine whether it truly has higher overall intelligence, evaluators also need to examine task coverage, result stability, environmental adaptability, execution cost, and safety boundaries. For enterprise users, the more important question is not where a model ranks on a leaderboard, but whether it can consistently complete real business tasks within an acceptable budget and risk range.

1. “Most Intelligent” First Requires a Computable Definition

The capabilities of large models are clearly multidimensional and cannot simply be compressed into a single number. Model intelligence should be understood from at least the following perspectives.

1.1 Knowledge and Understanding

These capabilities are mainly demonstrated in reading long texts, extracting facts, understanding concepts, summarizing viewpoints, and processing professional materials. Evaluation should not only check whether the model produces a fluent answer. It should also check whether the model:

• Accurately distinguishes facts, assumptions, and inferences;
• Can trace the conditions on which a conclusion depends;
• Identifies contradictions when information conflicts;
• Proactively expresses uncertainty when evidence is insufficient;
• Uses professional terminology accurately.

Natural language expression does not equal correct understanding. Some incorrect answers can be highly persuasive, so fact verification and evidence checking should be conducted independently of text quality.

1.2 Reasoning and Problem Solving

Reasoning capability includes mathematical analysis, logical judgment, planning, constraint satisfaction, and multistep decision-making. Even if a model scores highly on a fixed question bank, that does not mean it can reliably solve structurally similar new problems.

A more reliable test method is to transform the problems—for example, by changing numerical values, reordering constraints, adding irrelevant information, or removing implicit conditions—and then observing whether the model reaches a consistent conclusion. If performance drops sharply after only minor changes to a problem, the high score may come mainly from pattern memorization rather than transferable reasoning ability.

1.3 Action and Environment Interaction

Answering questions and completing tasks are two different levels of capability. A computer-operation agent needs to form a closed loop of “observe—plan—execute—verify—correct.” A deviation at any step can cause the task to fail.

Consider updating a business record. The agent must not only recognize interface elements but also:

1. Understand the user’s goal and business constraints;
2. Find the correct record;
3. Determine which fields are allowed to be modified;
4. Perform input, selection, and submission operations;
5. Check whether the data was saved successfully;
6. Roll back or request human confirmation after detecting an anomaly.

Therefore, interface recognition accuracy reflects only perception capability and cannot fully represent end-to-end task capability.

1.4 Learning and Adaptation

Real business environments change continuously, including page layout adjustments, field name changes, permission changes, and data format changes. Whether a model can adapt to these changes matters more than its one-time performance in a fixed environment.

Adaptation evaluation should cover:

• Unseen page structures;
• Different resolutions and display scaling;
• Dynamic loading, pop-ups, and overlays;
• Multilingual or mixed-language interfaces;
• Replanning after an operation failure;
• Degradation handling when a tool returns an error.

1.5 Reliability and Controllability

Enterprise applications generally do not accept a system that is “occasionally very smart but occasionally seriously wrong.” A model needs to maintain low result variance across repeated runs and comply with permission, format, process, and compliance requirements.

For this reason, stable completion rate, critical error rate, and human takeover rate are often more business-relevant than the highest score achieved in a single test.

2. Why Public Benchmarks Alone Cannot Prove Overall Leadership

Public benchmarks have important reference value, but they provide only partial evidence. The benchmarks themselves also need to be examined.

2.1 Test Data May Be Contaminated

If training corpora contain test questions, explanations, variants, or discussion content, a model may obtain high scores through memorization. Even when the original question does not appear directly, the presence of many similar templates can reduce the difficulty of the test.

Methods for reducing contamination risk include:

• Using data newly constructed after publication;
• Applying semantics-preserving structural transformations to questions;
• Building internal private test sets;
• Introducing samples derived from desensitized real business logs;
• Rotating questions regularly to avoid long-term exposure.

2.2 High Scores May Come from Additional Inference Resources

Model results are closely related to runtime configuration. Longer inference time, more sampling attempts, external retrieval, code execution, and human filtering can all significantly improve final results. If different candidate models are tested under inconsistent conditions, horizontal comparison is not fair.

A credible evaluation report should at least specify:

• Whether access to external information is allowed;
• Whether an execution environment can be called;
• The maximum number of steps per task;
• Whether multiple generations and result selection are performed;
• How timeouts, refusals, and invalid outputs are scored;
• Whether human correction or task retries occur.

2.3 Average Scores Can Mask Critical Risks

The average score may be high, but the model may still make serious errors on a small number of high-risk tasks. For example, an occasional classification error in document classification may have limited impact, whereas a single error when deleting data, adjusting permissions, or executing a script may cause major losses.

Evaluation results should therefore show at least:

• Average task success rate;
• Worst-category performance;
• Critical error rate;
• Proportion of unrecoverable failures;
• Confirmation rate before risky operations;
• Deviation between output confidence and actual accuracy.

2.4 Benchmark Saturation Does Not Mean the Problem Is Solved

When a model approaches a perfect score on a fixed test, it may mean that the test can no longer effectively distinguish capabilities, not that the model has solved all problems in the field. At that point, task complexity should be increased—for example, by adding open environments, multi-objective constraints, long-horizon dependencies, and unexpected interference—rather than continuing promotional comparisons around saturated scores.

3. How Should Computer-Operation Capability Be Evaluated?

Computer operation is an important direction for general agents, but it is also one of the areas most easily misled by demonstrations. A screen recording of one smooth completion does not represent stable production capability.

3.1 Evaluate Perception, Decision-Making, and Execution in Layers

Tasks can be divided into four layers:

Perception layer: recognizing buttons, icons, input fields, tables, and status prompts.

Semantic layer: understanding the purpose of controls and distinguishing the business meaning of actions such as “save,” “submit,” and “delete.”

Planning layer: generating steps based on the goal and identifying dependencies between them.

Execution layer: completing clicks, input, scrolling, uploads, and confirmations while verifying operation results.

Testing only the perception layer cannot show whether the model can complete a full workflow. Conversely, recording only the final success rate cannot identify whether failures come from visual recognition, planning errors, or execution drift. Production evaluation should therefore retain both layered metrics and end-to-end metrics.

3.2 Focus on Task Recovery Capability

Real environments frequently experience network delays, failed page navigation, session expiration, and insufficient permissions. A good agent should not continue executing blindly after its first operation failure. It should re-observe the state and adjust its plan.

The following metrics are recommended:

• First-attempt completion rate;
• Completion rate after automatic retries;
• Average number of recovery steps;
• Proportion of repeated execution of the same operation;
• Error-page recognition rate;
• Accuracy in requesting human assistance.

3.3 Set Guardrails for High-Risk Operations

Tasks involving data deletion, funds, permissions, external sending, and production configuration should not be fully autonomous. Common controls include:

• Least-privilege authorization;
• Secondary confirmation for high-risk steps;
• Prioritizing reversible operations;
• Generating a change preview before execution;
• Retaining complete audit logs;
• Automatically transferring to a human after a risk threshold is reached.

The higher the intelligence level, the more actions a model can perform, and the greater the potential risk. Action capability and governance capability must therefore be developed together.

4. Software Engineering Capability Is More Than Code Generation

Code generation is only one part of software engineering. Real development tasks also include requirement understanding, code reading, defect localization, test design, change review, and deployment verification.

A relatively complete software engineering evaluation process can include:

1. Provide a desensitized code repository and defect description;
2. Restrict the model to accessing only necessary directories;
3. Require the model to first output the basis for problem localization;
4. Generate a minimal modification plan;
5. Automatically run builds, static checks, and tests;
6. Check whether regression issues are introduced;
7. Have developers evaluate maintainability and security.

The following metrics should be emphasized:

• First-fix pass rate;
• Regression test pass rate;
• Amount of unrelated code changes;
• Number of newly introduced security defects;
• Average resolution time;
• Proportion of developer-accepted changes;
• Whether the model can continue fixing based on logs after failure.

Generating a large amount of code does not mean engineering efficiency is higher. If developers need to spend more time reviewing and reworking, overall costs may increase instead.

5. Research and Professional Capabilities Require Evidence Chains

In tasks such as mathematics, data analysis, and professional consulting, providing a correct answer is not enough. The model also needs to form an inspectable reasoning and evidence chain.

For computational tasks, inputs, formulas, units, boundary conditions, and intermediate results should be verified. For literature-based tasks, the model’s existing knowledge should be distinguished from external retrieval results to prevent fabricated citations. For open research questions, the model should be required to clearly state assumptions, scope of applicability, and parts that cannot be determined.

Particular attention should be paid to claims that “the model solved a long-standing problem.” Such claims are extremely strong conclusions and cannot be based solely on the model’s self-report or a single output. They must be validated by independent experts, rigorous formal checks, or reproducible experiments before being considered reliable.

6. Build an Enterprise-Specific Composite Scoring System

Different businesses assign different weights to model capabilities, so public leaderboards should not be copied directly. Enterprises can build a weighted scoring model based on risk and benefit:

Composite value =
task success rate × business weight
− critical error rate × risk coefficient
− cost per task × cost coefficient
− human takeover rate × operations coefficient
− response latency × timeliness coefficient

This formula is not a unified industry standard; it is a design approach. The key is to translate “strong capability” into quantifiable business value.

For example, internal knowledge Q&A can place more emphasis on factual accuracy and citation completeness; interface automation can focus more on end-to-end success rate and recovery capability; software engineering can focus more on test pass rate and regression risk; high-risk professional scenarios should significantly increase the penalty weight for critical errors.

The evaluation set should also be divided into three layers:

• Foundation set: measures general understanding, reasoning, and format-following capability;
• Business set: derived from real workflows and typical tasks;
• Adversarial set: includes ambiguous instructions, malicious input, permission inducement, and abnormal environments.

Only after passing all three types of tests does a model meet the conditions for entering the pilot stage.

7. The Evaluation Process Must Be Fair and Reproducible

To avoid bias caused by differences in prompts, environments, and retry strategies, a unified experimental protocol is recommended.

First, use equivalent task instructions and the same context for all candidate models. Second, fix the maximum execution time, maximum number of steps, tool permissions, and retry count. Third, run each task multiple times to estimate both average value and variance. Finally, when possible, scorers should not know in advance which candidate model produced an output, in order to reduce subjective bias.

A simplified evaluation pseudocode is as follows:

for each candidate in candidates:
    for each task in evaluation_set:
        repeat N times:
            reset_environment()
            result = run(candidate, task, same_limits)
            verify_result(result, task.expected_state)
            record_success_cost_latency_and_risk()
generate_report(
    average_success_rate,
    variance,
    critical_error_rate,
    human_takeover_rate,
    average_cost,
    response_latency
)

Here, expected_state should not store only a text answer. It can also describe database state, page state, test results, or file changes. Only in this way can we determine whether the model truly completed the task rather than merely generating a plausible description.

8. From Model Capability to Production System Capability

The model itself is only one part of an intelligent application. A production system also requires infrastructure such as permission control, data isolation, log auditing, monitoring and alerting, task queues, human approval, and failure recovery.

Even if the underlying model performs well, a lack of engineering governance can still make the system unavailable. Conversely, a model whose single-task capability is not the highest but which is stable and easy to control may create higher practical value within a well-designed workflow.

Before launch, enterprises should confirm at least the following:

• Whether sensitive data is desensitized and isolated as required;
• Which tools and data the model can call;
• Whether every operation is traceable;
• Whether high-risk actions require approval;
• Whether tasks can be stopped immediately in the event of an anomaly;
• Whether regression evaluation is rerun after model updates;
• Whether human takeover and business degradation plans are in place.

9. Conclusion: A Leading Model Is Not Necessarily Absolutely the Most Intelligent

Determining whether a frontier model represents the current highest level of intelligence requires multidimensional, dynamic, and reproducible evidence. Leading on public benchmarks can prove technological progress, but it cannot alone prove that a model is superior to other options across all tasks, all environments, and all cost constraints.

A more rigorous conclusion should usually be written as: “The model leads on several controlled tests and specific task categories,” rather than directly claiming that it possesses the highest intelligence in an absolute sense.

For developers, the key focus should be whether the model can reliably understand requirements, call tools correctly, complete tasks, and handle exceptions. For enterprises, evaluation should further measure cost, latency, security, compliance, and human-takeover requirements. The model truly worth adopting is not necessarily the one that is most impressive in a demonstration, but the one that can continuously create value within clear boundaries, with discoverable errors, auditable processes, and controllable risks.