When Benchmark Scores Approach 100%: How to Assess Claims of General Intelligence

1. The Starting Question: Does a Saturated Benchmark Prove General Intelligence?

When a company releases a new model and claims that humanity has entered the era of artificial general intelligence (AGI), one striking piece of supporting evidence may be a score of 99.9% on a benchmark designed to assess abstract reasoning. The score may have been achieved with the help of a dedicated runtime harness.

The reasoning can sound straightforward: if the benchmark was designed to measure how close a system is to general intelligence, and a model scores almost perfectly, then the gap must be nearly closed. But anyone working on system design and deployment should be wary of treating a metric as a conclusion. A benchmark-breaking score may reflect a genuine leap in capability. It may also mean that targeted optimization has covered much of what the benchmark measures—or that the harness is doing a significant share of the work attributed to the model.

This article does not speculate about the motives behind such claims. Instead, it treats them as questions of evaluation methodology: What does the benchmark measure? How are its tasks structured and scored? What support does the runtime provide? Are there risks of data contamination or an outdated test set? What concerns has the research community raised? Clear answers are essential to understanding what a 99.9% score actually means.

2. Three Explanations for Benchmark Saturation: Capability, Adaptation, or Measurement Failure

When a score approaches 100%, a sound evaluation begins by considering at least three competing explanations.

First: a genuine improvement in capability. The model may have robust generalization across the benchmark’s task distribution, maintaining strong performance when tasks, contexts, or wording change. This is the interpretation a model provider would like readers to accept.

Second: adaptation through the runtime harness. Modern agent evaluations rarely test a bare model. They often combine a model with external scaffolding, including prompt templates, tool-calling protocols, validation and retry strategies, and task-decomposition workflows. If that scaffolding has been extensively tailored to a particular benchmark, it becomes difficult to tell how much of the score comes from the model and how much from the orchestration. The explicit mention of a “dedicated harness” is therefore an important qualification.

Third: measurement failure. A limited number of tasks, a fixed distribution of task types, or the reuse of older items in newer benchmark versions can create overlap between training and evaluation data. If that happens, a high score may reflect memorization or pattern matching rather than reasoning.

To determine which explanation best fits a 99.9% result, look for three things:

• Consistency across benchmarks: Does the model also lead on related benchmarks with different task formats?
• Ablation results: How much does performance change when the dedicated harness is removed?
• Held-out sets and rotating task pools: Are tasks refreshed regularly?

If any of these are missing, confidence in the conclusion should decrease.

3. Anatomy of an Agentic Long-Horizon Benchmark

The following example shows how to examine a typical agent benchmark. These evaluations target complex professional workflows in real software environments, with tasks spanning areas such as financial modeling, engineering design, and media production. They emphasize multi-step execution, tool use, and professional judgment.

3.1 Identity and Purpose

Field	Description
Benchmark type	Agentic long-horizon evaluation
Organizing body	An evaluation initiative led by an academic research institution
Release format	Preprint paper and public leaderboard
Purpose	End-to-end measurement of agent capabilities in realistic professional workflows

Academic institutions, rather than model vendors, often build these benchmarks, which can help support neutral task design. However, academic benchmarks may have smaller task sets, less frequent maintenance, and weaker contamination defenses than industrial-scale evaluations. Their freshness and limitations therefore need to be assessed.

3.2 Construct: What Does the Benchmark Measure?

The benchmark’s construct is an agent’s ability to complete complex professional workflows in real software environments. Three parts of that definition deserve closer attention:

• Real software environments: Tasks are not abstract text-based questions. They require actions in desktop applications, terminals, or specialized tools. The evaluation may therefore test visual understanding, interface navigation, command-line operation, and file-system interaction.
• Professional workflows: Industry experts design tasks involving areas such as financial modeling, CAD, and audio or video editing. Success requires domain understanding as well as general tool-use ability.
• Long-horizon execution: A task may involve dozens or even hundreds of steps. A mistake at any point can cause the overall task to fail, making the benchmark sensitive to error accumulation, context management, and plan revision.

The clarity of the construct determines how interpretable the score is. The broader the construct—such as “completing complex professional work”—the less a single score can explain. A high average may conceal strong performance in one task category and serious weaknesses in another. Those weaknesses may be central to whether a system is genuinely general.

3.3 Structure: Tiers and Difficulty Distribution

This type of benchmark may use multiple tiers, progressing from basic terminal tasks to its most difficult level. According to the reported figures, the pass rate for any model on the highest tier, labeled Frontier, is only about 2.6%. That detail matters: the benchmark as a whole has not saturated; saturation appears to be limited to lower and intermediate difficulty levels.

This creates a common evaluation pitfall: insufficient reporting by tier. If a benchmark has several difficulty levels but only an aggregate score is published, readers cannot tell where the model performed well. If the pass rate on the hardest tasks remains in the single digits, describing the overall result as nearly perfect can be misleading. Evaluations should report pass rates by tier, not just a single total.

Tasks run in real desktop environments, and results may be checked by automated scripts or expert reviewers. Automated validation is inexpensive and reproducible, but it can misrepresent partial completion. Expert review may be more accurate, but it is costly, harder to scale, and vulnerable to differences among raters.

3.4 Metrics: What Does “Pass Rate” Mean?

The benchmark uses pass rate as its main metric, scoring tasks according to completion. Several details of that scoring policy are easy to overlook:

1. Binary scoring or partial credit: Does a task earn a point only if it is completed in full, or does the score reflect the proportion of steps completed? Binary scoring is especially demanding for long-horizon tasks; partial credit can make capability appear stronger than it is.
2. Number of attempts: How many times may the model try the same task? Allowing repeated attempts can substantially raise the pass rate because long-horizon success is highly sensitive to the number of samples.
3. Attribution of failure: Did the task fail because of a model decision, or because of an unstable environment or a tool timeout? Counting environmental noise as model failure may depress the score. Conversely, automatically retrying through environmental problems may inflate it.

Ideally, a report should include both single-attempt pass rate and pass rate across multiple samples. This helps distinguish consistent performance from best-case performance.

3.5 Runtime Configuration: The Often-Overlooked Harness

This is one of the most important parts of the evaluation. The original report notes that the benchmark page does not specify the exact runtime-harness version used. It also states that a reported score of 59.3% was achieved at the “highest effort” setting, and that certain automatic review or confirmation strategies were not enabled for the comparison.

Each of these details matters:

• Unspecified harness version: The harness determines prompts, available tools, error handling, and retry logic. Different versions can change a model’s score by many percentage points. Without version information, the result is difficult to reproduce.
• “Highest effort” setting: Such a setting typically involves more reasoning time, more sampling, or more self-checking. It represents performance under resource-intensive conditions, not necessarily typical performance in a standard deployment. For engineering decisions, the latter is often the more useful reference.
• Automatic review disabled: Review strategies can improve final pass rates by allowing a model to check its work before submission. Disabling them may make the score more conservative, but it also makes the result difficult to compare directly with configurations that enable review.

A practical engineering principle is to record evaluation settings in a reproducible, structured format rather than relying on natural-language footnotes.

{
  "model": "<model identifier>",
  "harness_version": "<fixed runtime-harness version>",
  "effort_level": "high | medium | low",
  "max_retries_per_task": 1,
  "self_review_enabled": false,
  "tool_allowlist": ["terminal", "file_io", "gui_control"],
  "timeout_seconds_per_task": 1800,
  "scoring": "binary_pass | partial_credit",
  "reporting": ["overall", "per_tier", "per_domain"]
}

Cross-model comparisons have engineering value only when the evaluation report includes a sufficiently complete configuration. Otherwise, the comparison is between two different evaluation systems, not simply between two models.

3.6 Contamination Risks: A Hidden Weakness of Long-Horizon Benchmarks

Preventing contamination is particularly difficult for long-horizon benchmarks:

• Limited task sets: Complex expert-designed tasks are harder to generate at scale than multiple-choice questions. The task pool is naturally constrained, and public tasks may eventually appear in training data.
• Recognizable environments: Real software interfaces have stable visual features. A model may infer an action sequence from interface cues rather than understand the task’s underlying meaning.
• Leaked validation scripts: Public repositories may expose scoring logic, allowing models to optimize for specific validation checks rather than the actual task objective.

Possible mitigations include private held-out sets, regular task rotation, and parameterized variations of public tasks—for example, changing the data, goal, or intermediate constraints. When reviewing a high score, ask whether it was achieved on a public task pool or on a held-out set.

3.7 Disagreements with Other Leaderboards

The findings from any single benchmark should be checked against a broader matrix of results. Two common patterns of disagreement are worth noting:

• Different rankings on benchmarks that target similar constructs: A model may lead on a long-horizon workflow benchmark but trail on a short-horizon tool-use benchmark. This suggests that long-horizon execution and single-step tool use are distinct abilities and should not be collapsed into one score.
• A gap between benchmark scores and deployment experience: A model at the top of a leaderboard may still be impractical in real business settings because of latency, cost, or reliability. Benchmarks often do not measure these engineering properties, even though they can determine whether a deployment succeeds.

4. Model Selection for Enterprises: From Scores to an Evaluation Matrix

For teams integrating models into production systems, a 99.9% score on one benchmark is rarely enough to support a decision. A more reliable approach is to build a multidimensional evaluation matrix:

1. Construct coverage: List the tasks that actually occur in the business—such as retrieval, code changes, document generation, multi-turn tool orchestration, and long-context understanding. Choose at least one benchmark that matches the construct for each task category.
2. Configuration transparency: Give greater weight to results that disclose the harness, prompting approach, retry policy, and review settings.
3. Reporting by tier and domain: Ask for pass rates broken down by difficulty and task domain, not just an aggregate score.
4. Contamination audit: Check for private held-out sets, task-rotation mechanisms, and separation by time.
5. A small in-house gold set: Draw 100–300 real tasks from the organization’s own workflows and have expected outcomes labeled by people. This is the most direct way to evaluate performance on the organization’s task distribution rather than relying solely on public benchmarks.
6. Collect engineering metrics in parallel: Track latency percentiles, cost per task, failure recoverability, and tool-call reliability alongside capability scores.

A simple weighted framework can help compare candidates across these dimensions:

weights = {
    "task_success_rate": 0.30,   # Pass rate on the in-house gold set
    "tier_coverage": 0.15,       # Performance on the hardest tier
    "config_transparency": 0.10, # Reproducibility of evaluation settings
    "cost_per_task": 0.15,       # Cost per task
    "p95_latency": 0.10,         # Tail latency
    "recoverability": 0.20,      # Proportion of failures that can be recovered
}
# Normalize each dimension to 0–1, then calculate a weighted total
# to compare candidate systems.

5. Conclusion: Turn Claims into Testable Propositions

Can a score of 99.9% on one benchmark support the claim that humanity has entered the era of general intelligence?

From the perspective of evaluation science, the answer is no—or, at minimum, the evidence is insufficient. Three points explain why.

First, benchmark saturation is not the same as capability saturation. If a benchmark has multiple difficulty tiers and the hardest tier still has a low pass rate, a near-perfect aggregate score shows only that the easier tiers may have been mastered. It does not establish generality. Results broken down by tier are essential.

Second, the runtime harness is a hidden independent variable. A dedicated harness, a highest-effort setting, and the choice to enable or disable review all form part of the context behind a score. Comparing scores without that context is like comparing two different evaluation systems.

Third, no single benchmark can define generality. General intelligence is a multidimensional construct that includes abstract reasoning, transfer across domains, long-horizon planning, instruction following, and communicating uncertainty. Any one benchmark captures only part of that picture. Even if its stated goal is to measure the distance to general intelligence, it cannot by itself support the conclusion that the distance has reached zero.

For developers, a more useful response is to treat such claims as a public lesson in evaluation methodology. When presented with a high score, ask about the construct, the benchmark’s composition, the metric, the runtime configuration, and contamination risks. Making these questions part of an evaluation checklist is more valuable over time than remembering any one score. A system’s success depends not on its place on a leaderboard, but on whether it can reliably complete the tasks in your own workload under your own engineering constraints.