The Truth About AGI Evaluation: Where Benchmarks End and Marketing Narratives Begin

1. When “Near-Perfect Scores” Are Equated with “AGI Has Arrived”

A cloud provider recently announced that “humanity has entered the AGI era” upon releasing a new generation large model. The core argument: the model, paired with a proprietary runtime harness, achieved a 99.9% score on an abstract reasoning benchmark. Yet that benchmark series was designed specifically to measure “how far we still are from AGI.”

This is a fundamental logical inversion: a tool built to measure a gap is being presented as evidence that the gap has been eliminated. To use an analogy, it is like measuring “close to room temperature” with a thermometer and then declaring that the human body has evolved beyond the need for thermoregulation. The measurement instrument is not the capability itself; a high score does not mean the construct has been satisfied.

This article approaches the topic from an evaluation methodology perspective, systematically examining the design logic, construct validity, contamination risks, and interpretive challenges introduced by the “model + proprietary harness” evaluation paradigm. The goal is not to dismiss any technical progress, but to help developers build a more reliable assessment framework for determining what “a model’s score on a benchmark” actually means.

2. Core Concepts in Evaluation Methodology

Before analyzing individual benchmarks, let us establish a shared vocabulary:

Field	Meaning
Identity	Full benchmark name, issuing institution, release date, public paper identifier
Construct	What capability the benchmark claims to measure; examples of task types
Composition	Number of tasks, data sources, evaluation pipeline
Metrics	Scoring metrics used and their statistical meaning
Setup	Runtime harness / prompt configuration / tool configuration during model execution
Contamination	Training set leakage risk, known benchmark flaws, version staleness
Controversy	Existing community/academic critiques; conflicts with other leaderboard conclusions

Among these, “construct validity” is the most easily overlooked yet most critical concept: does a benchmark actually measure what it claims to measure? If a benchmark’s design goal is “to measure the gap to AGI,” then a perfect score means “the gap is zero along this particular measurement dimension”—not “AGI has been achieved.” Bridging that leap requires additional argumentation.

Furthermore, “harness” refers to the external configuration under which a model is evaluated: prompt templates, tool-calling interfaces, multi-turn interaction strategies, result parsing methods, and so on. The same model can produce dramatically different scores under different harnesses, so reporting a score without disclosing the harness configuration renders it neither reproducible nor comparable.

3. Typical Design of Computer-Use Benchmarks

3.1 An Agent “Ultimate Exam” Benchmark from an Academic Institution

Identity: Released by a research institute at a well-known university, made public in mid-2026, accompanied by a preprint paper.

Construct: Tests an agent’s ability to complete complex professional workflows in real software environments. Tasks span financial modeling (spreadsheets/financial terminals), engineering design (CAD software), media production (video/audio editing), and other domains, emphasizing multi-step, cross-tool, long-horizon tasks that require professional judgment.

Composition: Uses a tiered structure ranging from basic terminal tasks to the highest difficulty level. Evaluation runs in real desktop software environments; tasks are designed by industry experts, and results are verified by automated scripts or experts. Only about 2.6% of tasks at the highest difficulty level can be solved by any model.

Metrics: Pass rate, scored by task completion. Because the pass rate at the highest difficulty level is extremely low, existing scores concentrate in the mid-to-low difficulty tiers.

Setup: The publisher did not specify the exact harness version used during evaluation; the reported 59.3% is a “highest effort tier” score; no automated review or confirmation strategy was used in comparisons.

Contamination and controversy: The benchmark was released in 2026, so training-data leakage risk is relatively manageable, but version updates to real software environments may render tasks stale. Additionally, the statistical treatment of pass rates—whether partial completion counts, and how partial completion is weighted—significantly affects comparability.

Key takeaway: The value of this benchmark lies in its attempt to measure “autonomous completion capability in real professional workflows,” which is a direction with high construct validity. But the 2.6% pass rate at the highest difficulty level shows that current agents still face enormous gaps on long-horizon professional tasks. When citing such benchmarks, difficulty tiers must be reported alongside scores; otherwise “59.3% pass rate” will be misread as “approaching human level.”

3.2 The Amplification Effect of Evaluation Configuration on Scores

The “model + proprietary harness” evaluation paradigm raises a fundamental question: how much of a score improvement comes from the model itself, and how much from engineering optimizations in the harness?

For example, if a model is paired with a purpose-built prompt chain, multi-turn self-correction, tool-calling orchestration, and result-verification loops, the final score reflects the joint capability of “model + harness,” not the model’s standalone capability. This is reasonable from an engineering standpoint—real deployments do pair models with harnesses—but when comparing and making claims, the distinction must be made explicit.

More dangerously, if the harness has been overfit to a specific benchmark (e.g., hardcoding solution templates for task types), the score loses its generalization meaning. The community has discussed such “benchmark tuning” extensively, and the core consensus is: reporting a score requires disclosing the harness configuration, and the harness should be general-purpose across benchmarks rather than customized for a single one.

4. The Special Nature of Abstract Reasoning Benchmarks

The abstract reasoning benchmark series has a very clear design goal: to measure “the gap to AGI.” Its tasks typically involve novel pattern reasoning, sequence completion, rule induction, and similar problems that require models to reason without prior knowledge.

The special characteristics of such benchmarks are:

1. The construct is the “gap” itself: The benchmark designers publicly state that a perfect score means “the AGI gap is zero along this measurement dimension,” not “AGI has been achieved.” This is a basic distinction in measurement theory: the operationalization of a construct is not the construct itself.

2. Extremely high contamination risk: Abstract reasoning tasks have relatively fixed problem patterns; once public, they are easily absorbed into training data. If a model has seen similar problems during training, the score reflects memorization rather than reasoning.

3. The harness has outsized influence: Abstract reasoning tasks typically require multi-step trial and error and search; pairing with a dedicated harness (e.g., program synthesis, exhaustive search, verification loops) can substantially boost scores. But this boosts “the harness’s search capability,” not “the model’s reasoning capability.”

4. A perfect score is not AGI: Even if a model reaches 99.9% on an abstract reasoning benchmark, it may still have enormous gaps in natural language understanding, commonsense reasoning, cross-domain transfer, causal inference, and other dimensions. AGI is a multi-dimensional construct that no single benchmark can cover.

Therefore, when encountering “a model reached 99.9% on an abstract reasoning benchmark, so AGI has arrived,” the correct interpretation is: the model performs near-perfectly along that benchmark’s measurement dimension, but that benchmark is only a necessary—not sufficient—condition for AGI.

5. How to Rationally Evaluate “AGI Milestone” Claims

Based on the analysis above, developers should adopt the following framework when evaluating similar claims:

5.1 Check Construct Validity

• What does the benchmark claim to measure?
• Does its task design truly cover that construct?
• Does a perfect score equal construct satisfaction?

5.2 Check Contamination Risk

• What is the relationship between the benchmark’s release date and the model’s training data cutoff?
• Could the tasks have leaked into the training set?
• Have version updates rendered tasks stale?

5.3 Check Harness Disclosure

• Is the score a standalone model score or a joint “model + harness” score?
• Has the harness been customized for this benchmark?
• Is the harness configuration fully disclosed to support reproduction?

5.4 Check Statistical Methodology

• How is the pass rate calculated? How is partial completion weighted?
• Are difficulty tiers reported?
• Is the score comparable to other models under the same methodology?

5.5 Check Multi-Dimensional Evidence

• Is the single-benchmark score consistent with other benchmarks?
• Are there conflicts with conclusions from other leaderboards?
• Has an independent third party reproduced the result?

6. Practical Recommendations for Developers

1. Do not use a single benchmark for technology selection. Build your own evaluation set covering your actual business scenarios; it is more reliable than any public leaderboard.

2. Report scores with full configuration. If you used specific prompt templates, tool-calling strategies, or result-parsing logic in internal evaluations, record them all; otherwise scores are not reproducible.

3. Distinguish “model capability” from “systems engineering.” Joint optimization of model + harness is a reasonable engineering choice, but attribution must remain clear: attribute improvements to where they originate.

4. Focus on difficulty tiers, not total scores. A model with a 90% pass rate on low-difficulty tasks and 2% on high-difficulty tasks versus another with 70% on low and 5% on high—which is stronger? It depends on which difficulty range your business falls into.

5. Maintain measurement humility about “AGI.” AGI is a multi-dimensional, fuzzy, poorly bounded construct. A perfect score on any single benchmark is insufficient to claim AGI has been achieved. A reasonable statement is: “Along a specific measurement dimension of a specific benchmark, this model performs near human level.”

7. Conclusion

Benchmarking is an important tool in engineering practice, but it is not a truth machine. A benchmark score has interpretive power only within that benchmark’s construct scope, under a disclosed harness configuration, and with contamination under control. When “near-perfect scores” are directly equated with “AGI has arrived,” the problem lies not with the benchmark itself but with the logical leap from measurement results to grand claims.

As developers, our advantage is that we can build evaluations ourselves, reproduce results ourselves, and verify claims ourselves. Rather than debating whether we have entered the AGI era, it is better to run an evaluation in our own business scenarios and see what the model can do, what it cannot do, and under what configurations it can do it. That is the genuine engineering response to marketing narratives.