Evaluating End-to-End AI Agent Capabilities: Has the AGI Era Arrived?

1. The Real Shift: From Generating Answers to Delivering Results

Imagine a task where a user provides a simple circular sketch and asks the system to expand it into an aircraft porthole, create a 3D structure, add interactive logic, and ultimately generate a file ready for 3D printing.

Traditional generative models typically handle only isolated steps, such as explaining the design concept, generating a code snippet, or outputting 3D modeling instructions. Users still have to switch between multiple specialized software tools, copy model outputs, fix formatting errors, and continuously supplement instructions.

New-generation agent systems aim to change this collaboration model. Instead of providing step-by-step instructions, the user gives a goal. The system must then independently:

1. Understand the sketch and natural language requirements;
2. Decompose the goal into subtasks like design, modeling, programming, validation, and export;
3. Select and call appropriate external tools;
4. Operate software interfaces or structured APIs;
5. Check if intermediate results meet requirements;
6. Re-plan upon encountering errors;
7. Deliver the final product along with execution instructions to the user.

This means the criteria for evaluating models are changing. In the past, the focus was on “Is the answer accurate?” Now, we must ask, “Is the task actually completed?”, “Is the result usable?”, and “Was the execution process safe?”

Therefore, a more accurate description is not that large language models have suddenly acquired omnipotent intelligence, but rather that models combined with tools, runtime environments, memory systems, and permission controls have formed a more complete task execution system.

2. End-to-End Execution Does Not Mean Single-Step Generation

“End-to-end” is often misunderstood as the model generating the final result in a single inference pass. In reality, complex agent tasks typically run in a continuous loop:

Understand Goal → Make Plan → Call Tool → Get Feedback → Validate Result → Revise Plan → Continue Execution

For example, handling a used item listing task might require the system to read local images, identify the item type and cosmetic defects, estimate necessary description details, fill out the listing form, and confirm the price and visibility scope with the user before submitting.

Any step could fail:

• The image has obstructions, leading to incomplete defect recognition;
• The form fields differ from the expected structure;
• The user has not authorized sharing contact information;
• Network operations time out;
• The auto-generated description exaggerates the item’s condition;
• The user changes the price or delivery method mid-task.

Therefore, an agent cannot rely solely on generative capabilities; it also needs state management, exception handling, result validation, and permission governance. A practical execution framework can be abstracted as follows:

Task State = Initialized
Loop Execution:
    Read user goals and latest supplementary requirements
    Analyze current state
    Select the next executable action
    If action requires high-risk permissions:
        Request user confirmation
    Else:
        Call the corresponding tool
    Log tool inputs, outputs, and execution results
    Validate if results meet constraints
    If failed:
        Decide whether to retry, use an alternative approach, or request human intervention
    If goal achieved:
        Output results, evidence, and summary of changes
        End task

This workflow illustrates that an agent’s reliability is not solely determined by the foundational model. Task orchestration, tool interfaces, validation rules, and human confirmation mechanisms are equally important.

3. Two Key Capabilities: Asynchronous Execution and Mid-Task Steering

1. Asynchronous Tool Calling

In traditional serial workflows, after a system calls a time-consuming tool, it must wait for the result to return before proceeding to the next step. If information retrieval, file conversion, and code testing all take significant time, the overall task becomes highly inefficient.

Asynchronous execution allows the system to continue processing other independent tasks while waiting for a specific tool. For example:

• Generate usage instructions while converting a 3D file in the background;
• Check constraints in existing documents while waiting for a data query;
• Prepare backup plans for failures while executing a test task;
• Run multiple independent information retrieval tasks in parallel.

However, parallel does not mean unordered. The system must maintain clear dependency relationships. For instance, file export can only begin after structural validation passes, and transaction information can only be submitted after the user confirms the final price.

A generic representation of task dependencies is as follows:

Read Requirements
 ├─ Analyze Images
 ├─ Extract Business Constraints
 └─ Check User Permissions
       ↓
Generate Initial Plan
 ├─ Create Content
 ├─ Build Structured Files
 └─ Execute Automated Checks
       ↓
Human Confirmation
       ↓
Publish or Export

Engineering-wise, each task node must record its state, including pending, running, completed, failed, canceled, and awaiting confirmation. Without a reliable state machine, an increase in parallel calls actually raises the probability of tasks spiraling out of control.

2. Mid-Task Steering

Real-world tasks rarely follow the initial plan exactly. Users might add requirements halfway through:

• Change the presentation style from formal to lightweight;
• Hide contact information;
• Change the output format to a printable structure;
• Pause publishing to generate a preview first;
• Keep the completed design but rewrite the sales copy.

The system must determine which subtasks are affected by the new instructions, rather than simply clearing the context and starting over. This process can be called incremental re-planning.

Mid-task steering involves at least three questions:
First, which existing results are still valid?
Second, which running tasks need to be canceled?
Third, do the new requirements conflict with existing constraints?

If a user requests “publish immediately,” but organizational policy dictates that public releases require human confirmation, the system cannot bypass safety rules to comply with the latest instruction. Thus, agents must not only understand natural language but also handle instruction priority, task dependencies, and policy constraints.

4. Why “More Reliable” Might Come with “Harder to Monitor”

A common misconception is that the better a model performs, the easier its internal process is to understand. In reality, there is no necessary correlation between the two.

As model capabilities improve, they may learn to use shorter, more abstract internal representations to complete tasks. The reasoning explanations seen by an external observer do not necessarily equate to the computational process the model actually relies on. Even if a model can generate a seemingly reasonable analytical text, it does not prove that it did not omit constraints or adopt speculative strategies during execution.

Therefore, the model’s outputted “thinking explanation” cannot be taken as the sole audit evidence. Reliable monitoring should be built on observable behaviors, including:

• Which tools were called;
• Which files and data were accessed;
• Which fields were modified;
• What content was sent;
• Which steps went through user confirmation;
• Whether permission denials were triggered;
• Whether intermediate products passed independent validation;
• Whether the final result can be reproduced by external rules.

For example, a system claiming “no personal information was leaked” is insufficient. A more reliable approach is to perform independent sensitive information detection on the submitted content while checking tool call logs to confirm the system did not send data to unauthorized addresses.

This reflects an important principle:

Supervision of agents should shift from “believing how the model explains itself” to “verifying what the model actually did.”

5. What Conditions Must Be Met to Claim the AGI Era Has Arrived

AGI is generally understood as general intelligence capable of cross-domain learning, transfer, and performing a wide range of cognitive tasks. However, there is no single, unified, and universally accepted criteria in the industry. Therefore, it is impossible to prove AGI has been achieved based solely on a demo video, a few benchmark tests, or a handful of successful cases.

To judge whether we are approaching general intelligence, at least the following dimensions must be examined.

1. Task Breadth

Can the system span multiple domains like programming, visual understanding, writing, data analysis, software operation, and real-world planning, rather than just excelling at pre-designed tasks?

2. Transfer Capability

Faced with new environments not seen in training or testing, can the system quickly adapt based on minimal instructions? If a tool’s interface changes, can the model still identify the goal and adjust its operations?

3. Long-Term Stability

Completing a brilliant demo is different from working continuously for hours or days. The longer the task, the more likely errors accumulate. The system needs to maintain consistency in goals, constraints, file versions, and permission boundaries.

4. Autonomous Error Correction

Can the system proactively identify unreasonable results and correct them through testing or cross-validation? If every deviation relies on human reminders, it remains more of an efficient assistant than a highly autonomous general executor.

5. Real-World Reliability

Real business involves ambiguous requirements, insufficient permissions, missing data, system failures, and responsibility boundaries. The system must know when to continue, when to ask, when to refuse, and when to hand over to humans under uncertain conditions.

6. Safety and Governability

The greater the capability, the wider the impact of erroneous operations. If a system can modify files, publish content, purchase goods, or send messages, it must possess fine-grained authorization, operational audits, budget limits, and emergency stop capabilities.

By these standards, end-to-end agents represent significant progress toward more general intelligence, but “being able to execute more tasks” and “having achieved AGI” are not the same proposition. A more cautious conclusion is that human-computer interaction is entering the agent phase, but whether AGI has arrived still requires long-term, open, and reproducible evidence.

6. Enterprise Evaluation Cannot Look Only at Final Success Rate

For enterprises, a single success rate metric hardly reflects whether a system is suitable for production environments. A task might ultimately succeed but could have gone through massive retries, consumed excessive costs, or executed unauthorized operations.

A more comprehensive evaluation system should include at least the following metrics.

1. Task Completion Rate

Determine whether the system delivered a final result that meets acceptance criteria. Acceptance criteria must be defined in advance and cannot rely solely on model self-evaluation.

2. Subtask Accuracy Rate

By breaking down long tasks, individual checks on information extraction, file modification, tool calling, and result validation help locate the source of failures.

3. Zero Human-Intervention Completion Rate

Record the number of human corrections required for a task. A high success rate built on frequent human takeovers significantly diminishes its automation value.

4. Unauthorized Operation Rate

Check whether the system accessed unnecessary data, executed unapproved external actions, or bypassed confirmation steps. This metric is often more important than answer accuracy.

5. Recovery Capability

Artificially inject tool timeouts, file corruptions, API exceptions, and input conflicts to observe whether the system can safely retry, switch schemes, or halt execution.

6. Cost and Latency

Record the model invocation volume, tool usage count, runtime, and failure retry costs for each task. Parallel execution can reduce wait times but may also increase resource consumption, requiring comprehensive evaluation.

7. Auditability

Every key decision should be traceable to its input, policy, tool call, and output. Audit logs should also avoid recording unnecessary sensitive data.

7. Technical Architecture Required for Production-Grade Agents

A relatively complete enterprise agent system typically consists of six layers.

Layer 1: Interaction and Requirement Parsing

Responsible for receiving natural language, images, documents, and other inputs, and identifying goals, constraints, and delivery formats. For ambiguous requests, it should clarify first rather than autonomously filling in high-risk information.

Layer 2: Planning and Task Orchestration

Decompose goals into subtasks, establish dependencies, and dynamically adjust based on execution feedback. Plans must be modifiable; the initial scheme cannot be treated as an immutable process.

Layer 3: Tool Integration

Access file systems, databases, business applications, search capabilities, and compute environments through a unified adaptation layer. Each tool should declare its input format, output format, permission scope, timeout duration, and potential side effects.

Layer 4: State and Memory

Short-term state is used to save current task progress, while long-term memory stores authorized preferences and historical information. The two must be distinguished; all conversations cannot be permanently saved, and sensitive content cannot be written into long-term memory by default.

Layer 5: Validation and Security Policies

Check permissions before execution and validate results after execution. For high-risk actions like payments, public publishing, data deletion, and external messaging, mandatory human confirmation should be enforced.

Layer 6: Logging, Monitoring, and Governance

Record task trajectories, failure causes, resource consumption, and policy hits. Use monitoring alerts to detect abnormal loops, call spikes, or continuous failures. Enterprises can also build their runtime foundation using general-purpose cloud computing, storage, logging, and monitoring capabilities provided by Baidu AI Cloud, but specific selections should be based on business needs, compliance scopes, and existing architectures.

8. Permission Design Should Follow the Principle of Least Privilege

The biggest risk with agents is not “saying the wrong thing,” but “doing the wrong thing.” Once a system possesses software operation capabilities, permission design must become a core engineering issue.

The following control methods are recommended:

• Default to read-only; grant write permissions temporarily based on task needs;
• Set different permissions for previewing, saving drafts, and official publishing;
• Restrict accessible directories and data scopes;
• Require secondary confirmation for external sending, payments, and deletion operations;
• Set time, call count, and cost limits for each task;
• Support pausing, revoking, and human takeover at any time;
• Use deterministic rule validation for high-risk operations, rather than relying entirely on model judgment.

Furthermore, avoid granting agents overly broad long-term credentials. A safer approach is to use short-term, task-level authorization that expires immediately after the task ends.

9. Implementation Route: Start with Low-Risk, Verifiable Tasks

Enterprises should not pursue fully autonomous operation right from the start. A more reasonable route is to increase the degree of automation in phases.

Phase 1 is the Recommendation Mode. The system only generates plans and candidate content without executing external actions.

Phase 2 is the Assistance Mode. The system can read data, generate files, and fill in drafts, but final submission is completed by a human.

Phase 3 is Controlled Execution. The system automatically completes low-risk tasks within explicit permissions, while high-risk nodes require confirmation.

Phase 4 is Limited Autonomy. For processes with clear rules, verifiable results, and recoverable failures, the system is allowed to execute continuously, supervised by sampling audits and anomaly alerts.

Tasks suitable for early implementation generally share the following characteristics:

• Clear input and output formats;
• Low consequence of errors;
• Results that can be automatically validated;
• Repetitive and time-consuming manual processes;
• Limited required permission scope;
• Ability to roll back upon failure.

Conversely, scenarios involving major legal judgments, medical decisions, large financial transactions, or irreversible data changes should not rely solely on autonomous agents.

10. Conclusion: Beyond the AGI Debate, the Engineering Paradigm Shift Matters More

The core significance of end-to-end agents is not making chat interfaces smarter, but changing how software is used: humans are gradually shifting from operating software step-by-step to describing goals, setting boundaries, and accepting results.

This is indeed a significant shift. Asynchronous tool calling, mid-task steering, long-task management, and multimodal understanding enable systems to undertake more complex work. However, enhanced capabilities also bring new contradictions: a system can be more efficient and reliable while its internal decision-making process becomes harder to explain; it can reduce manual operations while expanding the impact scope of erroneous executions.

Therefore, “whether we have entered the AGI era” remains a question lacking unified evidence standards. For developers and enterprises, the more valuable focus is: can the system continuously complete tasks within explicit permissions, can results be verified with external evidence, can it safely stop during anomalies, and can it undergo a complete audit.

Future truly production-valuable agents will not just be the most capable models, but reliable systems composed of models, tools, permissions, validation, monitoring, and human oversight.