Evaluating Next-Generation General-Purpose AI Models: Capabilities, Agents, Benchmarks, and Governance

In recent years, general-purpose AI models have been iterating at a visibly accelerating pace. Each new generation quickly sparks debate: Has it entered the AGI stage? Can it replace a large amount of knowledge work? Can it operate a computer independently?

From a technical and engineering perspective, however, release slogans, demo videos, or a handful of benchmark results are not enough to support reliable conclusions. The more important question is not how the model name has changed, but whether the model can continuously complete complex tasks in open environments, whether it can control the cost of errors, and whether it can be embedded safely and stably into enterprise business processes.

A sound assessment of next-generation general-purpose AI models therefore needs to address four dimensions at the same time: capability boundaries, engineering architecture, evaluation methodology, and governance mechanisms.

1. The Core of Model Evolution: From Answering Questions to Completing Tasks

Traditional dialogue models mainly process input text and generate corresponding text. They can explain concepts, write documents, and generate code, but the final action is usually still performed by the user.

Newer models are gradually moving toward task-oriented agents. Such a model must not only understand user intent but also complete a closed loop:

1. Break down a vague goal into executable steps.
2. Identify the current environment and task state.
3. Select the appropriate software, interface, or mode of operation.
4. Execute actions and read the results.
5. Determine whether the results meet expectations.
6. When errors are found, roll back, correct, and continue.
7. Request human confirmation at high-risk points.

For example, “organize this month’s operating data and generate an analysis report” is not a single text-generation task. The model may need to read multiple data sources, check field meanings, handle missing values, calculate metrics, create charts, write conclusions, and submit the results to a specified location.

The difficulty of such tasks is no longer only language generation. It involves long-horizon planning, tool invocation, state management, and exception recovery. In other words, when evaluating next-generation models, the question should shift from “Does it sound like an expert?” to “Can it reliably finish the job?”

2. Why Screen Understanding and Computer Operation Have Become Important

Enterprise software environments are highly fragmented. Some newer systems provide structured APIs, but many internal systems still rely on graphical interface operations. Common situations include:

• Systems built long ago that lack standard APIs.
• Incomplete API documentation and calling methods that are difficult to maintain.
• Multiple systems built by different teams with inconsistent data structures.
• Business operations that depend on page forms, pop-ups, and manual approvals.
• Desktop software that cannot be controlled directly through network APIs.
• High system refactoring costs that make unified integration difficult in the short term.

In this context, the graphical user interface can be treated as a universal interaction layer. As long as a human can complete a task through the screen, a model with visual understanding and operation capabilities can theoretically attempt to perform it.

A typical computer-use agent usually includes the following modules:

Screen Perception

The model needs to identify buttons, input fields, menus, tables, icons, prompts, and pop-ups on a page and understand their spatial relationships.

This is not merely an image recognition problem. It also involves semantic understanding. For example, the same “Submit” button may mean saving a draft, initiating an approval, or performing an irreversible action on different pages. The model must use context to determine its actual meaning.

Action Planning

After recognizing the interface, the model needs to decide whether to click, type, scroll, or wait. For complex tasks, it must also plan the order of multiple steps and adjust dynamically as the page changes.

State Verification

A successful click does not mean the task has succeeded. The model needs to read feedback, determine whether the page has navigated, whether data has been saved, whether the operation has been blocked by a permission system, and whether the result meets business requirements.

Exception Recovery

Real software environments frequently experience network latency, misplaced elements, pop-up occlusion, session expiration, and insufficient permissions. A mature agent should neither terminate immediately after the first failure nor click repeatedly without conditions. It should re-observe the environment and choose a recovery strategy.

3. The Interface Is a Universal Entry Point, but Not an Ideal Interface

“Letting the model look at the screen and operate it directly” offers strong compatibility, but it does not mean that graphical interfaces can completely replace structured APIs.

Compared with standard APIs, interface automation has clear limitations:

• Page layout changes may cause operations to fail.
• Visual recognition results are uncertain.
• Operation speed is usually lower than direct API calls.
• Transaction consistency and idempotency are difficult to guarantee.
• Page feedback may be incomplete.
• Resource costs for large-scale concurrent execution are high.
• Auditing and fault localization are more complex.

Therefore, a more reasonable priority for enterprise system integration is usually:

1. Prefer stable, auditable structured APIs.
2. Use controlled interface automation when APIs are unavailable.
3. Introduce human confirmation for high-risk actions involving funds, permissions, deletion, and similar operations.
4. Independently verify execution results rather than trusting only the model’s judgment.

The real value of graphical interface capability is to lower the barrier to integrating legacy systems, not to dismiss the importance of API governance.

4. How to Assess Advances in Coding Ability

Code generation is only one part of coding ability. A model may generate syntactically correct code without being able to complete real software engineering tasks.

Evaluating coding ability should cover at least the following aspects:

Requirement Understanding

Can the model identify ambiguity, missing conditions, and edge cases in requirements instead of immediately generating a seemingly reasonable implementation?

Code Reading

Enterprise development often requires modifying existing projects. The model must understand module relationships, data flows, exception handling, and historical constraints rather than working only with isolated code snippets.

Environment Operation

Real tasks may include dependency installation, configuration changes, data migration, log analysis, and troubleshooting build failures. The model needs to combine coding ability with environment operation ability.

Testing and Verification

A reliable model should proactively construct test cases covering normal input, abnormal input, boundary values, and concurrency scenarios. Using “the program runs” as the only success criterion is far from sufficient.

Security Awareness

Model-generated code may contain overly broad permissions, insufficient input validation, sensitive information leakage, or unreleased resources. Enterprises still need static checks, dependency checks, tests, and human review before integration.

Therefore, evaluating a coding model cannot rely only on one-shot answer accuracy. It should also examine whether the model can complete a full software change in a controlled environment and pass automated tests and quality gates.

5. Why Benchmark Scores Do Not Equal Real Productivity

Benchmarks provide a relatively unified way to compare models, but they are always simplified models of reality. A high score does not necessarily mean that an enterprise will obtain the same proportional efficiency gain after deployment.

When interpreting test results, pay particular attention to the following questions:

Are the Test Tasks Close to Real Work?

Fixed pages, short workflows, and clearly defined goals are usually simpler than real business operations. Production tasks may involve missing information, conflicting rules, cross-system dependencies, and multi-person approvals.

Are Multiple Attempts Allowed?

If a model can retry indefinitely, its final success rate may be high, but the actual cost and time consumption may also increase significantly. Enterprises care more about first-attempt success rate, average number of retries, and failure recovery cost.

Is Error Impact Measured?

Clicking the wrong menu and initiating the wrong payment have completely different consequences. Evaluation should not only measure whether a task is completed but also calculate the cost of errors according to risk level.

Does It Include Human Intervention?

Some tasks may appear to be completed by the model, while the actual process may depend on human hints. Evaluation reports should clearly state the automatic completion ratio, the number of manual takeovers, and the intervention points.

Is There Environment Leakage?

If the model has been exposed to highly similar task templates, test scores may overestimate generalization ability. A more credible approach is to validate with newly created internal tasks and undisclosed business workflows.

Based on these factors, enterprises should build their own evaluation sets rather than directly using public scores as the basis for procurement or launch decisions.

6. Enterprises Should Build a Multidimensional Evaluation System

For models with computer operation and tool-calling capabilities, evaluation can use the following metrics:

• Task completion rate: the proportion of tasks fully completed within a specified time.
• First-attempt success rate: the success rate without retries or human intervention.
• Step accuracy: whether each operation meets expectations.
• Average completion time: total time from receiving a task to producing a result.
• Average execution cost: including model calls, computing resources, and human review costs.
• Exception recovery rate: whether the model can continue a task after page changes or tool errors.
• High-risk misoperation rate: whether unauthorized access, mistaken deletion, mistaken submission, or similar behavior occurs.
• Human takeover rate: the proportion of tasks that require human intervention during execution.
• Result reproducibility: whether the same input and environment can reliably produce consistent results.
• Audit completeness: whether observations, reasoning summaries, actions, results, and approval information can be recorded.

These metrics need to be weighted according to business risk. For content organization tasks, efficiency may be more important. For finance, healthcare, production control, and similar scenarios, safety and traceability should have higher priority.

7. Recommended Architecture for Agent Systems

Enterprises should not give models unlimited computer control. A safer approach is to add a policy control layer between the model and the business environment.

A general execution flow can be represented as:

Receive user goal
    ↓
Identify task type and risk level
    ↓
Generate candidate execution plan
    ↓
Policy engine checks permissions, scope, and sensitive actions
    ↓
Execute low-risk steps in isolated environment
    ↓
Read results and verify independently
    ↓
Request human approval for high-risk actions
    ↓
Complete task and generate audit record

The corresponding pseudocode is as follows:

function executeTask(goal, identity):
    context = collectContext(goal)
    plan = model.createPlan(context)
    for step in plan:
        risk = policy.evaluate(step, identity)
        if risk == "deny":
            record(step, "blocked")
            return failure("Action blocked by policy")
        if risk == "approval_required":
            approved = requestHumanApproval(step)
            if not approved:
                record(step, "rejected")
                return failure("Execution rejected by human reviewer")
        result = sandbox.execute(step)
        verified = verifier.check(step, result)
        record(step, result, verified)
        if not verified:
            recovery = model.createRecoveryPlan(step, result)
            if recovery is invalid:
                return failure("Unable to recover safely")
    return buildReport()

The most important principle here is this: the model proposes plans and candidate actions, the policy system decides whether actions are allowed, and the verification module confirms whether results are truly valid. The same model must not both execute actions and unilaterally declare that it has succeeded.

8. Permissions and Security Matter More Than Model Capability

When a model can only generate text, errors usually appear as inaccurate answers. When a model can operate a computer, errors become real actions. Security governance must therefore be moved earlier.

Least Privilege

A model should receive only the minimum permissions required to complete the current task. Different agents should use independent identities, and sharing high-privilege accounts should be prohibited.

High-Risk Action Classification

Actions such as deleting data, changing permissions, sending external messages, executing payments, publishing content, and modifying production configurations should have stricter approval processes.

Environment Isolation

Development, testing, and production environments must be separated. New models should first be validated in simulation environments or read-only environments and must not directly enter core production systems.

Sensitive Information Protection

The context fed into a model may contain personal information, business data, and access credentials. Systems need to implement redaction, encryption, access control, and retention-period management.

Prompt Injection Defense

Web pages, documents, or emails may contain malicious text designed to induce the model to change its task goal. Agents must distinguish user instructions, system policies, and external content. They must not treat all text seen on a page as trusted commands.

End-to-End Auditing

Every task execution should record identity, time, input, plan, actions, approvals, and results. Logs are used not only for security auditing but also for locating failure causes and continuously improving evaluation sets.

9. How to Judge Whether We Are Approaching AGI

“Artificial general intelligence” currently lacks a unified, operational industry definition. Progress on reasoning, coding, or computer-operation tests does not mean that a model already has complete general intelligence.

More meaningful dimensions for judgment include:

• Whether it can transfer knowledge across domains.
• Whether it can handle tasks it has never seen before.
• Whether it can perform long-duration, stable goal planning.
• Whether it can recognize the boundaries of its own knowledge and capabilities.
• Whether it can proactively ask for clarification when information is insufficient.
• Whether it can adjust strategies after the environment changes.
• Whether it can continue learning without damaging existing capabilities.
• Whether it can remain safe and controllable in complex real-world environments.

At the current stage, a more cautious statement is that models are evolving from general language assistants into general task execution systems. They may reach a high level on some work, but they may still show obvious shortcomings in common-sense judgment, long-term consistency, exception handling, and accountability.

10. Enterprise Adoption Should Start with Small-Scale, High-Frequency, Low-Risk Tasks

Enterprises should not push for full automation immediately just because a model demonstrates strong operating capabilities. A more reasonable path is phased adoption.

In the first phase, choose low-risk tasks such as document organization, spreadsheet entry, and test-environment inspection to validate basic completion rates and costs.

In the second phase, connect general capabilities such as internal knowledge retrieval, logging services, object storage, and monitoring alerts to form a limited tool-calling loop.

In the third phase, introduce cross-system tasks while retaining human approval at key points, and use process engines to constrain execution order.

In the fourth phase, consider expanding the scope of automated execution only when metrics are stable, auditing is complete, and permissions are clear.

Each phase should have clear exit conditions. For example, if the high-risk misoperation rate exceeds a threshold, the human takeover rate remains high, or the unit task cost exceeds the cost of manual handling, expansion should be paused and reassessed.

Conclusion

The truly important change in next-generation general-purpose AI models is not that they are better at chatting. It is that they are beginning to perceive environments, plan steps, invoke tools, and execute tasks. Screen understanding can help models overcome interface barriers in legacy systems, while coding and terminal operation capabilities bring them closer to becoming real software engineering assistants.

But as capabilities grow, risk can also escalate from “wrong answers” to “wrong actions.” When evaluating new models, enterprises should therefore rely less on single benchmark scores and grand slogans. They should focus on real task completion rates, exception recovery ability, execution cost, security boundaries, and auditing capability.

Only when models can work stably under clear permissions, strict policies, independent verification, and human oversight will they deliver not just impressive demos but sustainable, governable, and scalable productivity.