AGI Deployment: What Custom and Standardized Benchmark Scores Really Tell Us

A seemingly contradictory set of results deserves a closer look: models from the same generation score 99.9 in a custom evaluation framework but only 62.7 under a standardized evaluation. Meanwhile, computer-use performance rises from 65.7% to 72.6%, professional work automation improves from 18.1% to 41.4%, and terminal task performance reaches 57.9%.

These figures suggest that system-level capabilities are improving. They also show why differences in evaluation methodology can distort the conclusions drawn from benchmark results.

Assessing the practical value of artificial general intelligence (AGI) requires more than identifying the highest score. The important questions are what constraints applied, whether the result is reproducible, and whether the capability transfers to real business workflows.

1. Why Scores of 99.9 and 62.7 Can Coexist

A high score in a custom framework and a lower score under a standardized protocol are not necessarily contradictory. They often measure different things.

A custom evaluation measures the combined performance of the model, prompt templates, tool integrations, orchestration strategies, and scoring rules. A standardized evaluation focuses more on cross-scenario generalization under a common protocol, budget, and task distribution.

Deep adaptation to a particular model can systematically improve performance within a custom framework. A broader evaluation covering long-tail tasks, heterogeneous tools, and out-of-distribution scenarios can reasonably produce a lower score.

Why custom frameworks can produce very high scores

• Narrow task distributions: Task types, tool interfaces, and output formats may closely resemble those used during training or tuning, making the expected solution path easier to follow.
• Customized prompts and toolchains: Fixed system prompts, tool descriptions, retry policies, reflection mechanisms, and caching strategies can substantially improve end-to-end performance.
• Lenient scoring or ceiling effects: If tasks are insufficiently difficult, success criteria are permissive, or systems frequently reach the scoring ceiling, a score of 99.9 may indicate saturation within that evaluation—not reliability across environments.
• Human fallback or preconfigured environments: Preloaded intermediate results, broad permissions, or permitted human intervention can substantially increase measured success.
• Unclear boundaries between tuning and evaluation: Similarities between evaluation tasks and tuning data may reward familiarity rather than generalization.

These are possible explanations, not evidence that any particular evaluation uses these practices.

What standardized evaluations can expose

• More out-of-distribution tasks: Tasks spanning domains, tools, and formats demand stronger transfer and planning capabilities.
• Stricter tool protocols: Common interfaces, fixed budgets, timeouts, and permission boundaries reduce the advantage of model-specific adaptations.
• Long-tail failures: In a multistep workflow, one incorrect action can cause the entire task to fail. Errors can accumulate throughout the execution trace.
• Scoring closer to functional correctness: Evaluation considers whether the task was actually completed, whether the outcome is reproducible, and whether unintended side effects occurred—not merely whether the output looks plausible.

The 99.9 score is therefore better understood as a signal of performance achievable through engineering optimization in an adapted environment. The 62.7 score highlights generalization limitations under a common protocol.

Both results matter. Neither should replace the other.

2. How to Interpret the Key Metrics

Computer use: 65.7% to 72.6%

The increase is 6.9 percentage points, indicating progress in interface understanding, action planning, and tool use.

However, if the metric represents binary task success, a 72.6% success rate still means roughly one in four tasks fails. That is a substantial failure rate for production workflows, particularly those involving data modification, process submission, or actions across multiple systems.

Errors in these settings can be far more costly than errors in a demonstration environment.

Professional work automation: 18.1% to 41.4%

This represents an absolute gain of 23.3 percentage points and a substantial relative improvement. Yet a 41.4% success rate still suggests that most evaluated professional tasks cannot be completed fully automatically.

Professional workflows often combine high business value, lengthy processes, strict compliance requirements, and low tolerance for mistakes. Errors may create business risk rather than simply reduce efficiency.

This result is better interpreted as an expansion of opportunities for assisted automation, not evidence that professional work is ready for large-scale, unattended execution.

Terminal tasks: 57.9%

A 57.9% result suggests some ability to generate commands, recover from errors, and execute tasks in a controlled terminal environment.

Terminal access also carries destructive potential. A single incorrect command can have irreversible consequences. Without sandboxing, least-privilege access, command auditing, and human approval, this success rate does not justify autonomous operation in high-risk production environments.

What a single score leaves out

A point estimate has limited decision-making value if the accompanying materials omit:

• Sample size and confidence intervals
• Number of evaluation runs
• Cost and latency
• Human takeover rate
• Failure categories

A responsible evaluation should explain how many runs were performed, what budget and permissions were available, and which task distribution was used.

It should also distinguish planning failures, tool failures, environmental failures, and scoring ambiguity. Can the system recover after an error? What is the economic cost of each successfully completed task?

Without that context, comparing scores can be misleading.

3. Build a Unified Evaluation Framework

Enterprises should measure model capability and system capability separately, then evaluate them through a consistent, reproducible pipeline.

At a minimum, that framework should include seven layers.

1. Task coverage

Cover perception and understanding, tool use, multistep planning, long-running tasks, and system-level automation.

Use a combination of public datasets, private holdout sets, dynamically generated tasks, and time-based splits to reduce the risk of evaluation contamination.

2. Environment control

Pin environment images, dependency versions, tool permissions, timeout budgets, and initial state.

Every run should be reconstructable so that environment drift does not distort the results.

3. Common protocols and budgets

Standardize observation formats, action formats, tool descriptions, and result formats.

Apply consistent budget definitions, including limits on model or tool calls, latency, and cost.

4. Scoring

Combine exact matching, functional verification, rule-based scoring, and manual spot checks.

Cross-check results with multiple graders and record agreement between them.

5. Statistical reporting

Run repeated trials and report means and confidence intervals.

Include success rate at a fixed budget, P50 and P95 latency, cost per successful task, and human takeover rate.

6. Safety evaluation

Record the rate at which destructive actions are blocked, attempted unauthorized access, sensitive-data exposure risks, and audit completeness.

7. Failure classification

Separate failures into categories such as:

• Misunderstanding the task
• Planning errors
• Incorrect tool selection
• Invalid parameters
• Environmental errors
• Timeouts
• Scoring ambiguity

This makes the results useful for diagnosis and improvement rather than merely ranking systems.

Metrics that matter more than an aggregate score

Metric	What it reveals
Success rate at a fixed budget	Task completion under equivalent call, latency, and cost constraints
Cross-distribution performance drop	The gap between custom and standardized sets, or public and private sets
Human takeover rate	The proportion of tasks requiring human correction, confirmation, or rollback
Cost per successful task	Total cost, including failed attempts, retries, and human review
Long-running task stability	How success rates decline as the number of task steps increases
Safety blocking rate	The proportion of high-risk operations correctly blocked
Regression pass rate	Whether previously supported tasks remain reliable after a version update

These metrics help distinguish an impressive benchmark result from a system that can operate reliably within business constraints.

4. An Illustrative Evaluation Pipeline

The following pseudocode shows the minimum structure of a unified evaluation pipeline. The purpose is not to prescribe an implementation, but to evaluate tasks, environments, budgets, scoring, statistics, and failure categories consistently.

def evaluate(system, suite, runs=5):
    records = []
    for task in suite.tasks:
        for seed in range(runs):
            env = suite.prepare_env(task, seed)
            trace = run_agent(
                system,
                instruction=task.instruction,
                tools=env.tools,
                budget=task.budget,
                timeout=task.timeout,
                permissions=task.permissions
            )
            score = suite.grade(task, trace, env)
            records.append({
                "task_id": task.id,
                "slice": task.slice,
                "difficulty": task.difficulty,
                "score": score.value,
                "success": score.success,
                "latency": trace.latency,
                "cost": trace.cost,
                "human_takeover": trace.human_takeover,
                "failure_type": score.failure_type,
                "unsafe_action_blocked": trace.unsafe_action_blocked
            })
    return aggregate(records)
def aggregate(records):
    return {
        "mean_score": mean(records.score),
        "success_rate": mean(records.success),
        "ci95": bootstrap_ci(records.success),
        "p50_latency": percentile(records.latency, 50),
        "p95_latency": percentile(records.latency, 95),
        "cost_per_success": sum(records.cost) / max(sum(records.success), 1),
        "human_takeover_rate": mean(records.human_takeover),
        "failure_taxonomy": count_by(records.failure_type),
        "by_slice": group_by(records, ["slice", "difficulty"])
    }

With this approach, a custom-framework score of 99.9 and a standardized score of 62.7 can be reported side by side.

The former indicates what the system can achieve in a closely adapted environment. The latter provides a more constrained view of cross-scenario performance.

The most useful information for deployment decisions is the gap between them, the causes of that gap, and whether engineering improvements can narrow it.

5. From Evaluation to Enterprise Deployment

Start with task segmentation, not full automation

Divide workflows into three groups based on risk:

• Low-risk, high-frequency tasks: Start with assisted generation and automated validation.
• Medium-risk tasks: Use partial automation with human confirmation.
• High-risk tasks: Retain strong approval requirements and rollback procedures.

This provides a more controlled path to deployment than treating all tasks as candidates for unattended execution.

Use shadow testing and staged rollouts

Run the system alongside real traffic without allowing it to control the live workflow. Compare its results with human outcomes, and record failure traces and points where human intervention would be necessary.

During a staged rollout, establish rollback controls, budget limits, and circuit breakers.

Make execution observable

Collect logs, execution traces, scores, cost, latency, and failure categories through a consistent process.

For each failure, preserve a reproducible environment snapshot and the relevant inputs and outputs. This supports evidence-based diagnosis instead of subjective judgments about whether a model is “good” or “bad.”

Enforce least privilege and sandbox isolation

Computer-use and terminal agents require particular care.

Restrict file writes, network access, command execution, and system configuration permissions. Require additional confirmation and auditing for destructive operations.

A capability score does not remove the need for these controls.

Evaluate vendors under the same conditions

Require vendors to run against the same task set, environment, budget, and grader.

Ask for both custom-adapted results and standardized results. Published high scores should be accompanied by sample sizes, confidence intervals, costs, and failure classifications.

At scale, container platforms, object storage, logging services, and monitoring and alerting can support reproducible evaluation and deployment pipelines. Baidu AI Cloud is one possible infrastructure option for these capabilities, where available. Infrastructure selection should not replace evaluation of task success, generalization, and safety boundaries.

6. Conclusion: Deployment Value Depends on Reproducible Success Across Scenarios

A score of 99.9 may indicate that a system is approaching saturation within a custom framework. A score of 62.7 suggests that substantial room for improvement remains under standardized conditions.

Computer-use performance of 72.6%, professional work automation at 41.4%, and terminal task performance of 57.9% point in the same direction: capabilities are advancing, but reliable unattended operation remains a challenge for high-risk, long-running, and compliance-sensitive workflows.

The practical value of AGI should not be defined by its highest isolated score. It depends on the combined strength of reproducibility, generalization, economic viability, and safety boundaries.

Enterprises should prioritize unified evaluations, private holdout sets, repeated runs, cost-normalized comparisons, failure classification, and staged deployment with rollback.

Treat high custom scores as optimization signals and lower standardized scores as warnings about generalization. A high score becomes operationally meaningful only when the system repeatedly succeeds under real constraints, recovers from failures, operates at an acceptable cost, and keeps risk under control.