How to Evaluate AGI Claims from Large Models: A Practical Guide to Benchmarking

1. “Entering the AGI Era”: Where Does This Conclusion Come From?

Not long ago, an overseas AI company claimed at a flagship model launch that “humanity has entered the AGI era.” The core evidence was quite specific: with an officially customized evaluation framework, the model achieved a near-perfect score of 99.9% on an abstract reasoning benchmark designed to “measure the gap between AI and AGI.”

At first glance, the logic seems coherent. But the key issue is a subtle shift in meaning: the benchmark was designed to quantify the gap, yet the launch presented it as a certificate of arrival. A tool for measuring distance and a tool for certifying completion have very different evidentiary value. The academic community still has no unified definition of AGI, and a saturated score on a single benchmark may only indicate that the current technical approach has mastered that particular task distribution — not that general intelligence has truly arrived. For technical practitioners, a more valuable question than “Has AGI arrived?” is: faced with overwhelming capability claims, how do we build a rational evaluation framework?

2. Why High Benchmark Scores Do Not Simply Equal AGI

2.1 Inherent Limitations of Construct Validity

Every benchmark measures a “construct” — the abstract capability its designers intend to capture. Abstract reasoning benchmarks typically target “grid-puzzle-style pattern induction and pattern completion.” Such tasks have two characteristics:

• Narrow and enumerable task distribution: The puzzle space is finite. Once a model learns similar pattern induction techniques, scores inflate rapidly, while real-world open-ended problems such as long-horizon planning, physical common sense, and continual learning remain uncovered.
• Obvious ceiling effects: From “distinguishing frontier models” to “being maxed out by frontier models,” the useful lifecycle of such a benchmark is often only one or two versions.

Therefore, a saturated benchmark only shows that the model performs well on that specific construct. It cannot be generalized as “comprehensively reaching human-level performance.”

2.2 Training Data Contamination and Measurement Failure

Large model training corpora cover the public internet, so public benchmarks and their variants can easily enter the training distribution. Even without deliberate cheating, a “student who has seen the original questions” renders the measurement invalid. This is why serious evaluation organizations periodically refresh question banks, maintain private holdout sets, and introduce contamination detection methods such as canary strings.

2.3 Score Elasticity from Evaluation Setup

The same model, paired with different agent frameworks, tool-call budgets, retry strategies, and prompt templates, can produce scores that differ by several times. A so-called “model score” is essentially a joint score of “model + runtime environment,” yet promotional materials often amplify only the former. When you see a high score, the first question should be: under what configuration was this score produced?

3. A Seven-Dimension Framework for Benchmark Credibility

When you see “Model X tops Benchmark Y,” systematically ask seven questions:

1. Identity: Who developed the benchmark and when was it released? Is there any conflict of interest with the party being evaluated?
2. Construct: What capability is actually being measured? What task types are included? Is it a single dimension or a composite capability?
3. Composition: How many items are there? Is the data source human-designed or program-generated? Is scoring automated or expert-reviewed?
4. Metric: Pass rate, partial credit, pass@k? How is the score calculated? Are retries allowed?
5. Settings: What agent framework, prompts, and tool configurations were used at runtime? Was it the “max effort” tier?
6. Contamination: How high is the risk of training data leakage? Does the benchmark have known flaws or an outdated version?
7. Controversy: What objections have been raised by the community and academia? Does the conclusion conflict with other leaderboards?

This review framework can be formalized as an engineering metadata checklist:

# Benchmark credibility review checklist (template)
benchmark:
  identity:
    org: "Academic institution / independent third party"     # Check for conflicts of interest with the evaluated party
    release: "Release date and current version"
  construct:
    ability: "e.g., multi-step abstract reasoning"
    task_types: ["pattern induction", "pattern completion"]
  composition:
    n_items: 120
    source: "Human-designed / program-generated"
    process: "Automated scoring + sampled human review"
  metric:
    name: "pass@1"
    note: "Whether partial credit is given, whether retries are allowed"
  settings:
    harness: "Agent framework, tool list, max steps"
    prompt: "System prompt template"
  contamination:
    risk: "Whether public, whether variants may have entered training data"
    check: "n-gram overlap rate, canary strings"
  controversy:
    notes: "Community objections, consistency with other leaderboards"

4. Case Analysis: Agent Evaluation in Real Software Environments

A “computer operation” benchmark frequently cited in recent release materials provides a complete demonstration of why the seven-dimension review is necessary.

• Construct: It evaluates whether an agent can complete complex professional workflows in real desktop software environments, such as spreadsheet-based financial modeling, engineering drafting, and audio/video editing. The tasks are multi-step, cross-tool, and require professional judgment over long horizons.
• Composition: It uses tiered difficulty design, from basic terminal operations to the hardest tier. Public information shows that the hardest tier once had only about 2.6% of tasks solvable by any model. Tasks are designed by industry experts, and results are verified through automated scripts or expert review.
• Metric: Pass rate is calculated by task completion. Since the hardest tier is nearly unsolvable, existing scores are actually concentrated in a mixed distribution across easy, medium, and hard tiers — meaning the same nominal score may represent entirely different task compositions.
• Settings: Release pages often do not specify which agent framework version was used. Scores labeled as “max effort” are not directly comparable to others’ default configurations. Some validation strategies were not enabled during evaluation and were only recorded in footnotes.

The lesson is straightforward: the same “approximately 60% pass rate” could result from “easiest difficulty tier + maximum tool calls + multiple retries,” or from “hard difficulty tier + minimal configuration.” Without the seven-dimension review, the number itself is meaningless.

5. Common Score Presentation Techniques and Cognitive Traps

Based on community discussions, common practices in promotional materials include:

1. Selective reporting: Only reporting high scores on favorable benchmarks while omitting unfavorable results.
2. Opaque frameworks: Not disclosing the agent framework, retry count, or prompts, making results impossible to reproduce.
3. Effort tier confusion: Comparing “max effort” scores against competitors’ default configurations.
4. Difficulty baiting: Claiming “near-perfect on a benchmark” without noting that it is the basic tier rather than the hardest level.
5. Outdated versions: Citing superseded or already-saturated benchmark versions.
6. Unauditable internal benchmarks: Relying heavily on internal datasets that cannot be externally verified, providing only conclusions without details.

The way to identify these is simple: ask about configurations, ask about task composition, and ask for third-party reproduction. If any of these three is missing, reduce the trust weight assigned to the claim.

6. Building an In-House Model Evaluation System: A Practical Methodology for Engineers

6.1 Three-Layer Evaluation Architecture

• Layer 1: General benchmark screening — Use public benchmarks to quickly filter candidate models, but be aware of contamination and check version timeliness.
• Layer 2: Domain-specific evaluation sets — A private question bank sampled from real business tasks. This is the core basis for decision-making.
• Layer 3: Online canary validation — Small-traffic testing in real scenarios, using business metrics rather than offline scores for final judgment.

6.2 Engineering Essentials for Domain-Specific Evaluation Sets

• Task mining: Extract high-frequency tasks from tickets, logs, and manual operation recordings.
• Difficulty tiering: Divide tasks by step count, tool count, and required expertise; report scores separately for each tier.
• Regular question rotation and canary detection: Prevent the evaluation set from being indirectly contaminated over time.

6.3 Cloud-Based Engineering of the Evaluation Pipeline

Evaluation is not a one-off script execution but a continuously operated pipeline. It can be built using common cloud service categories: use object storage to host evaluation datasets and result artifacts, use container platforms to orchestrate large-scale parallel evaluation tasks, use logging services to retain model inputs and outputs for traceability, and use monitoring and alerting to set thresholds for abnormal score fluctuations. Cloud platforms such as Baidu AI Cloud provide these general capability categories, and teams can combine them as needed or build on open-source evaluation frameworks. The key is ensuring configurations are recorded, results are reproducible, and processes are auditable.

6.4 Reference Evaluation Pipeline Implementation

def evaluate(model, suite, harness_cfg, n_runs=3):
    results = []
    for run_id in range(n_runs):            # Multiple runs averaged to suppress random fluctuation
        for task in suite.tasks:
            traj = harness_cfg.run(model, task)
            score = task.grade(traj)        # Automated scoring + sampled human review
            results.append({
                "task": task.id,
                "score": score,
                "harness_hash": harness_cfg.hash(),   # Record configuration fingerprint
                "suite_version": suite.version,
            })
    return aggregate(results)               # Report by difficulty tier, not just total score

7. Conclusion

The technical value of benchmark scores lies in comparability and traceability, not in declaring a new era. Faced with grand narratives of “AGI has arrived,” the rational stance for developers is neither blind acceptance nor wholesale rejection, but rather returning to the seven dimensions — identity, construct, composition, metric, settings, contamination, and controversy — to unpack every score. For enterprises, the real moat is never about trusting someone else’s leaderboard, but about building your own business evaluation system. After all, whether a model is good enough depends on your tasks, your data, and your evaluation.