Has AGI Arrived? Separating Model Capability from System Capability

1. Two Opposite Conclusions in the Same Week

Around discussions about when AGI will be achieved, two very different claims often appear: one says it is “already achieved for many tasks,” while another offers a timeline of years ahead. They seem contradictory, but they measure different things. The first describes end-to-end task completion rates; the second describes the threshold for autonomous generalization.

AGI lacks a unified, verifiable, and reproducible definition. Therefore, any claim that it “has been achieved” or “has not been achieved” must specify its measurement basis. In practice, the industry often falls back on three observable proxy metrics:

• Autonomous completion rate for long-horizon tasks: Can the model plan, correct itself, and converge during work lasting hours?
• Replacement ratio for economic tasks: What proportion of real-world job tasks can it complete at acceptable quality?
• Cross-domain transfer capability: Does it still work on unfamiliar problem types and tools outside the training distribution?

As long as the measurement basis differs, conclusions can both be true. Therefore, the first thing to do when discussing AGI is not to compare conclusions, but to ask: What system, under what constraints, and with what metrics was this score measured?

2. A Key Phenomenon: Unchanged Weights, Scores From 30.16% to a Perfect Score

One detail in the original phenomenon is more important than any conclusion: on a highly difficult benchmark, the underlying driving model was exactly the same, but merely replacing the outer engineering solution caused the score to jump from 30.16% to a perfect score. Three different teams had very different solutions, but the base model was identical.

This shows that the gain did not come from model weights, but from the “shell,” commonly called a harness or scaffold in industry. There are three common sources of such gains:

1. Tools and environment interfaces. Give the model a real file system, code execution sandbox, retrieval interface, terminal, and browser, turning “guessing answers” into “looking up answers.” The same reasoning flaw may appear when answering in plain text, but may resolve itself when an execution environment is available.

2. Verification and feedback loops. Turn unverifiable single-pass output into a closed loop of “generate—execute—check—revise.” Unit tests, compilers, type checkers, formal verification, and cross-review by independent models all belong to this category.

3. Search and sampling budget. Use best-of-N sampling, parallel exploration of multiple solution paths, and process reward models that score and rerank intermediate steps. In essence, these methods trade inference-time compute for accuracy.

A minimal verification loop can be written as follows:

def solve_with_verifier(task, model, verifier, max_rounds=5):
    history, draft = [], None
    for _ in range(max_rounds):
        draft = model.generate(task, history)          # Generate a candidate solution
        ok, feedback = verifier.check(draft)           # Objective check: run tests/compile/type check
        if ok:
            return draft                               # Return as soon as it passes
        history.append({"role": "assistant", "content": draft})
        history.append({"role": "tool", "content": feedback})  # Feed failure information back
    return None                                        # Clearly fail instead of pretending success

The value of this code is not its complexity, but that it separates “model capability” and “systems engineering” into two independently replaceable parts.

3. Decomposing System Capability Into Measurable Layers

To make a conclusion defensible, ablation experiments must be performed, peeling away contributions layer by layer. Attribution by layer is recommended:

Layer	Content	Observation method
L0	Base model capability	Disable all tools and retries; single-shot answer
L1	Prompting and context organization	Adjust only context structure, task decomposition, and example selection
L2	Tools and permissions	Toggle tools one by one and record per-item gains
L3	Orchestration and search	Change sampling count, parallel branch count, and reranking strategy
L4	Domain adaptation	Inject private knowledge, domain verifiers, and terminology constraints

Only L0 numbers can support judgments about whether the “model is stronger”; L1 to L4 numbers should be recorded as whether the “system is stronger.” Confusing the two leads to the erroneous conclusion that the “base model has improved greatly.”

4. Engineering Requirements for Reproducible Evaluation

A one-time inflated score cannot support technical decisions. Reproducibility requires at least the following:

Freeze all variables. Model version snapshots, sampling parameters, random seeds, scaffold code commit hashes, tool versions, and container image digests must all be fixed. If any of these drift, historical scores lose comparability.

Isolate the execution environment. Use containers to fix dependencies, lock dependency versions, disable networking by default or allow only whitelisted access, and avoid implicit online retrieval contaminating results during evaluation.

Prevent data contamination. Maintain a private holdout set, split questions by time, construct question variants, and deduplicate repeated items. Public test sets can only be used for regression, not as the basis for capability claims.

Metrics should cover distributions, not just extremes. In addition to pass@1 and pass@k, also record end-to-end duration, cost per task, manual intervention rate, and failure-type distribution.

Be especially careful with the misleading nature of pass@k: when k is large enough, many questions will be guessed correctly by at least one sample, so high pass@k does not mean deliverable. Production runs use pass@1, and what truly determines user experience is the stability of the mean and variance, as well as the cost of failures in worst cases. An example reproduction checklist can be written as a configuration file:

run_id: eval-2026Q1-baseline
model: { snapshot: "frozen-sha256", temperature: 0.0, top_p: 1.0, seed: 42 }
scaffold: { repo_commit: "abc1234", tools: ["shell", "fileio", "retrieval"] }
search: { samples: 1, parallel_branches: 1, reranker: none }
environment: { image_digest: "sha256:...", network: none, cpu: 4, memory_gb: 8 }
dataset: { name: private-holdout, split: "2026-01", dedup: true }
metrics: [pass@1, pass@5, p50_latency, cost_per_task, manual_takeover_rate]

5. How Enterprises Can Use This for Selection and Deployment

First, build a golden set around business tasks. Extract hundreds of samples with standard answers or automatically judgeable outputs from real tickets, covering common paths and known difficult scenarios, and use it as the sole acceptance criterion.

Second, insist on ablation-based comparison. Candidate solutions must be compared under the same scaffold and the same budget. If a vendor solution depends on multiple retries and large parallel sampling, cost and latency should be converted according to its actual call volume before comparison.

Third, use shadow runs and canary validation. Run the new solution in parallel with the existing process for a period of time, compare agreement rates and manual rework rates, and then decide whether to switch traffic.

Fourth, for engineering infrastructure, general cloud capabilities can be combined: container platforms host isolated execution and elastic scaling, object storage stores datasets, artifacts, and evaluation reports, message queues decouple queuing and retries for long-running tasks, log services retain full-chain invocation records for review, and monitoring alerts watch for quality-metric regressions and cost anomalies. Such general-purpose capabilities can support the evaluation pipeline described above without relying on any proprietary interface.

6. Three Judgment Criteria for Decision-Makers

1. Demand the measurement basis, not the conclusion. Require the other party to disclose the evaluation set, constraints, sampling count, scaffold configuration, and failure samples. If any item is missing, the number is not informative.

2. Distinguish demonstrations from delivery. One successful run is a demonstration; a stable SLA is delivery. Ask about mean, variance, worst-case behavior, and fallback plans.

3. Include cost in capability. A score obtained with ten times the compute and multiple retries is often unusable in real business; cost per task and manual intervention rate should be first-class metrics.

7. Conclusion

The phenomenon of “unchanged weights, perfect scores” reminds us that much of the current capability growth comes from systems engineering rather than the model itself. This does not diminish the value of engineering—on the contrary, tools, verification, and search have become key links in turning model capability into usable products. But when publicly claiming capability levels, the two must be measured separately.

For developers, this means that worthwhile investments are verifiable loops and reproducible evaluations. For decision-makers, under the dual pressure of time and publicity, the only reliable anchor is still stable numbers on your own task set.