How to Evaluate the Intelligence of Next-Generation Large Language Models
New large language model releases often come with claims of top intelligence or near-AGI performance. This article explains how to assess such claims using vendor results, independent benchmarks, and real business testing, and it outlines six evaluation dimensions: core capabilities, complex task completion, tool use, long-context stability, cost and efficiency, and safety and alignment. It also covers the limits of high scores and composite indices, how to evaluate agentic task completion, and a four-stage enterprise selection process.
In recent years, large model releases have often been accompanied by claims such as “the highest level of intelligence” or “approaching artificial general intelligence.” Such promotion does not necessarily mean the data is fabricated, but it is usually based on specific test sets, specific runtime frameworks, and specific task-selection conditions. For developers, the real question is not whether a model has achieved the highest score, but whether it can reliably complete complex tasks in real business scenarios, reduce development costs, and provide controllable safety boundaries.
Therefore, evaluating a new model should examine at least six areas at the same time: core capabilities, complex task completion, tool use, long-context stability, runtime cost and efficiency, and safety and alignment performance. Only when these dimensions are placed in the same analytical framework can the conclusion be useful as a reference.
1. Distinguish Official Results from Independent Performance
Model evaluations usually have three types of data sources.
The first is results published by the model provider. The advantage of this type of result is that the test scale is often larger and the environment configuration is relatively clear, so it can quickly show the upper limit of model capability. The disadvantage is that the testing process, prompt design, sample selection, and runtime strategy may be decided by the model provider. This is especially true for intelligent coding, computer operation, and multi-step agent tasks, where the model itself and the accompanying runtime framework are often difficult to fully separate.
The second is comprehensive evaluations organized by independent institutions. They usually combine multiple indicators such as mathematics, knowledge, code, reasoning, visual understanding, and agent tasks into a composite index. A composite index is better for observing the model’s overall position, but it also has weighting design issues. For example, one index may emphasize knowledge question answering, while another emphasizes code execution, so the rankings they produce may naturally differ.
The third is real-world testing by users or enterprises in actual business scenarios. Real-world testing is closest to usage value, but samples are often small, and task difficulty and evaluation standards are not uniform. A model that performs well at automatically generating interface code does not necessarily perform equally well at long-document retrieval, complex deployment, or risk judgment.
Therefore, when seeing a “number one ranking,” the first step is to ask three questions: Who designed the test? Did the model use specialized tools? Can the results be independently reproduced? If these three questions do not have clear answers, a single ranking should not be treated as the final conclusion.
2. A Score Above 99% Does Not Mean Near Perfection
High scores are the easiest source of misunderstanding. A model reaching near-perfect results on a specific task may show that it is very well suited to that type of task, but it does not directly prove that it has comprehensive, stable, and transferable intelligence.
For complex reasoning tests, for example, scores may be affected by the following factors:
Concentrated task distribution
If the test samples mainly cover mathematical forms, operational processes, or code patterns that the model is good at, a high score only proves that the model performs strongly on that distribution.Different prompt and tool configurations
Some tests allow the model to call tools multiple times, execute code, read intermediate results, and even automatically correct errors; other tests allow only one answer. The two types of scores cannot be directly compared.Inconsistent success criteria
Some evaluations only judge whether the final answer is correct, while others also check whether the process is reliable, whether operations are safe, and whether results are reproducible. Looking only at the final answer may hide many risks in the process.Test set contamination or memorization effects
If public test questions have appeared in training data, the model may achieve a high score through memorization rather than through genuine generalization ability.Task selection bias
Tasks that one group of models is good at may be selected for evaluation, while tasks that another group is good at may not be covered. In this case, the final score is more like “the degree to which the test design matches the model’s strengths.”
Therefore, a near-perfect result should be interpreted as “strong ability to complete this type of task under specific conditions,” not as “the model is already close to the top human level across all cognitive tasks.”
3. Composite Indices Are More Useful, but They Should Not Be Treated as Absolute
The value of a composite index is that it reduces the randomness of a single indicator. It usually covers knowledge understanding, logical reasoning, code generation, tool calling, long context, multimodal tasks, and other directions. For enterprise selection, a composite index is more helpful than a single impressive score.
However, a composite index is not an absolute standard. The following points deserve close attention:
- Whether indicator weights are disclosed: If the weights are not transparent, users can hardly tell which capabilities dominate the total score.
- Whether model versions are distinguished: The same model may have different reasoning intensity, response speed, and context configurations, and these cannot simply be merged.
- Whether variance is reported: Publishing only an average score without reporting fluctuations across tasks can hide model instability.
- Whether failure rates are included: For agent tasks, a single failure may cause data damage or business interruption, and an average score cannot fully reflect this risk.
- Whether cost and latency are considered: In production environments, a model with slightly higher accuracy but several times higher time and cost may not be more suitable for large-scale use.
A more reasonable approach is to use composite indices as a screening tool, not as a procurement conclusion. Before a candidate model enters actual business use, enterprises should conduct a second evaluation using their own de-identified task set.
4. Agent Capability Should Be Judged by Task Completion, Not Conversation
An important change in the new generation of models is the expansion from generating text to executing multi-step tasks. For example, a model may need to read requirements, modify multiple files, run tests, locate errors, call tools, and produce a deliverable.
When evaluating this type of capability, it is not enough to look only at single-turn answer quality. The full task chain should also be examined:
- Whether it correctly understands the goal and constraints;
- Whether it can break down the task and create an execution plan;
- Whether it can select appropriate tools;
- Whether it can locate the cause after an error occurs;
- Whether it can preserve key intermediate states;
- Whether it can avoid repeated execution and ineffective loops;
- Whether it can proactively pause when permissions are insufficient or risk is too high;
- Whether the final result can pass automated tests.
Among these, “memory and state management” is especially important. During a long task, if the model relies only on the context window, historical information can easily be compressed, key decisions can be lost, and work can be repeated. A more reliable engineering approach usually stores task goals, constraints, completed items, failure reasons, and pending steps in a structured form, rather than endlessly accumulating the entire conversation history in context.
A task state structure similar to the following can be used:
{"goal": "Complete the specified feature and pass tests","constraints": ["Do not modify external interfaces", "Preserve compatibility logic"],"completed": ["Completed code scan", "Added basic tests"],"failures": ["Boundary condition tests failed"],"next_step": "Locate input validation logic"}
The value of this method is not in a specific format, but in giving the model a work state that can be inspected and recovered.
5. Long Context Does Not Mean “The More It Reads, the Smarter It Gets”
Long-context capability is often presented as an important breakthrough, but context length and effective utilization capability are not the same thing.
What really needs to be evaluated is:
- Whether the model can accurately find distant information;
- Whether it can distinguish relevant content from noise;
- Whether it can maintain consistency with earlier and later constraints;
- Whether it can build structured summaries from long documents;
- Whether it can avoid being distracted by irrelevant instructions later in the text;
- Whether accuracy drops significantly after the context exceeds a certain scale.
In actual systems, blindly putting all documents into the context often increases cost and latency and may even reduce answer quality. More reasonable approaches usually include document splitting, keyword retrieval, vector retrieval, hierarchical summaries, and result reranking. The model is responsible for understanding and reasoning, while the retrieval system is responsible for narrowing the information scope. Combining the two is usually more stable than simply expanding the context window.
6. Safety Capability Should Be Treated as a Core Indicator
The stronger the model, the greater the potential risk. This is especially true when system operations, cybersecurity, code execution, and sensitive data processing are involved. A model should not only pursue “completing the task as much as possible”; it must also have refusal, degradation, and human confirmation mechanisms.
Safety evaluation should cover at least:
- Whether it will execute high-risk commands;
- Whether it will leak system prompts or sensitive data;
- Whether it can identify permission boundaries;
- Whether it can be affected by malicious text injection;
- Whether it can distinguish test environments from production environments;
- Whether it clearly states uncertainty when unsure;
- Whether it supports human approval and operation auditing.
A model actively blocking a high-risk task does not necessarily mean its capability is insufficient. On the contrary, it may indicate a more mature risk-judgment mechanism. What enterprises really need is “controllable high capability,” not a system that unconditionally executes all instructions.
7. A Practical Selection Method for Enterprises
Enterprises can establish a four-stage evaluation process.
Stage 1: Build a task set.
Extract de-identified samples from real business scenarios, covering code development, document processing, data analysis, customer support assistance, and workflow automation. Tasks should include normal cases, boundary cases, and failure cases.
Stage 2: Define quantifiable indicators.
In addition to accuracy, record task completion rate, human edit ratio, average response time, number of tool calls, failure recovery rate, cost per task, and safety interception rate.
Stage 3: Conduct blind testing.
Hide model names where possible to avoid evaluators being influenced by brand impressions. For code tasks, use automated tests; for document tasks, use structured scoring rubrics; for agent tasks, record the full execution trace.
Stage 4: Validate through small-scale rollout.
First run trials in low-risk, rollback-capable business scenarios, and observe model performance under real data, real permissions, and real concurrency. Only when stability, safety, and cost all meet requirements should the scope of use be expanded.
Conclusion: Do Not Look for the “Absolute Strongest”; Look for the Model That Best Fits the Task
Evaluating large models cannot rely on a single phrase such as “the most intelligent in the world,” nor should it be directly convinced by a near-perfect number. Model capability is multidimensional: some models are good at mathematical reasoning, some are good at code agents, some are more stable in long-document processing, and others are better suited to enterprise deployment in terms of cost, latency, and safety control.
A more mature judgment approach is to combine promotional data, independent evaluations, and real business testing, and to clearly distinguish the contributions of the model itself, the runtime framework, the toolchain, and human workflows. For developers, the focus should be on task completion rate, debugging efficiency, and recoverability; for enterprises, the focus should be on stability, cost, safety, and audit capability.
Therefore, “currently the most intelligent” is usually not a question that can be answered without considering the scenario. A more accurate statement is: a model has achieved leadership in certain specific directions, but whether it suits your task still needs to be determined through transparent, reproducible, real-business-oriented evaluation.
