From Multimodal Agents to AGI: How to Evaluate Model Progress Rationally

When a vendor releases a new general-purpose model and declares that “humanity has entered the AGI era,” the most important thing to avoid is not caution, but treating a product demonstration as proof of general intelligence.

A meaningful evaluation should focus on whether the model can reliably understand complex goals, execute tasks across tools, handle changing conditions, verify its own results, and operate within defined safety boundaries over extended periods.

1. From Chat Models to Task-Oriented Agents

Traditional language models primarily support an interaction pattern of “ask a question, receive an answer.” They are effective at organizing knowledge, generating text, assisting with code, and understanding natural language. However, the user still performs most of the actual work.

Users typically need to research information, open applications, copy content, run commands, inspect results, and then provide new context back to the model.

The major shift in newer AI systems is that the interaction objective is changing from “generate some content” to “complete a task.” A user may only need to describe a goal, while the system handles steps such as:

1. Understanding the goal and its constraints;
2. Creating an execution plan;
3. Retrieving the necessary information;
4. Calling external tools or applications;
5. Reading the execution results;
6. Revising the plan based on feedback;
7. Producing the final file or result;
8. Reporting the process, risks, and incomplete items to the user.

For example, if a user asks the system to create a product presentation, the agent may need to identify reference images, extract the visual style, generate copy, edit a presentation file, check the layout, and revise the output based on subsequent feedback.

The difficulty does not lie in any single action. It lies in managing state and coordinating multiple actions over time.

Therefore, when evaluating a model, it is not enough to ask, “Can it produce a good answer?” The more important question is, “Can it complete the work in a real operating environment?”

2. Multimodal Understanding Is Only the Foundation

Multimodal capabilities commonly include the understanding and generation of text, images, audio, video, and structured data. However, “understanding an image” and “completing work based on that image” represent two different levels of capability.

A lower-level capability is identifying objects, text, and basic relationships in an image. A higher-level capability includes the ability to:

• Detect detailed defects in objects;
• Understand how those defects affect the task;
• Convert visual information into a structured description;
• Perform follow-up operations in professional software or business systems;
• Verify that the operations actually took effect;
• Adjust the plan when the result deviates from the target.

Consider a three-dimensional design task. The model may need to understand a concept image, create editable geometry, configure materials, adjust the viewpoint, and export a file that meets downstream manufacturing requirements.

If the system can only generate an attractive image but cannot provide a structurally correct engineering file, it is still closer to a content generator than a complete work-execution system.

This illustrates that the core value of a multimodal agent is not simply producing more attractive output. Its value lies in converting unstructured information into verifiable and deliverable work products.

3. Asynchronous Execution and Mid-Task Changes Matter in Real Workflows

Real-world work is rarely linear. Some tasks require waiting for data processing, file conversion, or a response from an external system. If a model can only wait synchronously, execution efficiency may suffer. If the user adds requirements in the middle of a task, the system must also preserve the existing context and determine which steps need to be repeated.

More mature task-oriented systems generally need two types of general-purpose mechanisms.

1. Asynchronous Task Management

When a tool is processing a file or querying data, an agent does not always need to stop completely. It may be able to:

• Continue preparing steps that do not depend on the current result;
• Execute multiple independent operations in parallel;
• Track the status of each task;
• Merge the returned results into the broader context;
• Handle timeouts, failures, and partial success.

This is already close to a small workflow engine. The system needs to manage task queues, dependencies, retry strategies, and persistent state rather than relying on a single model call.

2. Dynamic Instruction Insertion

Users may introduce new requirements while a task is in progress. For example, they may change the target audience, request a different file format, cancel an operation, or add an approval step.

The system should not simply discard the original plan. Nor should it mechanically append the new instruction to the end. Instead, it should reassess:

• Whether the new requirement conflicts with the original goal;
• Which completed steps remain valid;
• Which steps must be reversed or executed again;
• Whether the user needs to confirm the revised plan;
• Whether the task has already produced an irreversible effect.

This requires the agent to maintain an explicit representation of state and a mechanism for revising plans. Without these capabilities, so-called autonomous execution can easily become unpredictable automation.

4. Greater Reliability Does Not Necessarily Mean Easier Oversight

As model capabilities improve, an important question is whether the system remains observable and auditable.

A final output may appear reasonable without the entire process being reliable. The model may have used unnecessary tools, cited unverified information, or concealed a failure behind vague language when it could not complete the task.

This is especially difficult in long-running workflows. When the intermediate state is complex, inspecting only the final result makes it difficult to identify the source of a problem.

System design must therefore focus not only on reasoning capability, but also on observability mechanisms:

• Record every tool call, including its inputs and outputs;
• Preserve key decision points and state transitions;
• Identify the source and trust level of external data;
• Require human confirmation for high-risk operations;
• Classify failures, retries, and exceptional branches;
• Allow users to replay the task process;
• Compare the final result with the original goal item by item.

Oversight should not depend solely on the model explaining all of its internal reasoning. A more robust approach is to build an audit trail from structured logs, operation traces, permission controls, result validation, and independent evaluation.

5. How Far Are We from AGI?

There is no single engineering definition of AGI accepted by everyone. In general, a system described as broadly intelligent would need strong cross-domain adaptability, continual learning, complex task planning, reliable execution, and the ability to transfer capabilities to unfamiliar environments.

A model that completes several impressive demonstrations has not necessarily met these conditions. Evaluation should focus on the following questions.

1. Generalization

Does the task work only with specific templates, fixed interfaces, and familiar data? When input formats change, interfaces are adjusted, or additional constraints are introduced, can the system continue to operate?

2. Stability

When the same task is executed repeatedly, are the results broadly consistent? Occasional success and sustained success represent very different levels of engineering maturity.

3. Long-Horizon Planning

As the number of steps increases, does the model forget early constraints? Can it maintain alignment with the original objective after dozens of steps or more?

4. Error Recovery

When a tool returns an error, data is missing, permissions are insufficient, or the external environment changes, can the system identify the problem, revise its plan, and explain the situation to the user instead of continuing to produce plausible-looking results?

5. Verifiability

Does the final deliverable have clear acceptance criteria? For example, is the file complete, is the format correct, can the calculation be independently checked, and do the operations comply with permission requirements?

6. Safety Boundaries

When an operation involves money, privacy, legal implications, production environments, or irreversible changes, does the system pause and request confirmation?

If these questions have not been adequately tested, saying that “the AGI era has already begun” is better understood as a communications statement than as a rigorous technical conclusion.

6. Enterprise Adoption Should Prioritize Human-Agent Collaboration

For enterprise users, the most practical approach is not to immediately allow an agent to take over every workflow independently. A better starting point is to use low-risk, reversible tasks with clear acceptance criteria.

Suitable scenarios include:

• Organizing and classifying internal materials;
• Extracting information from multiple documents;
• Generating and assisting with test cases;
• Creating initial drafts of operational content;
• Explaining data reports and flagging anomalies;
• Assigning tickets and tracking workflow status;
• Checking code and completing documentation during development.

Organizations can also use tiered permissions:

• Read-only access: The agent can query resources and analyze data;
• Draft access: The agent can generate files but cannot publish them directly;
• Restricted execution access: The agent can perform reversible operations;
• High-risk access: Human approval is required;
• Production access: Granted only after sufficient testing and auditing.

At the same time, organizations should establish a closed loop of goal, plan, execution, verification, and review. The model improves execution efficiency, while people define boundaries, handle exceptions, and retain final decision-making responsibility.

7. A Practical Evaluation Framework

Before introducing a task-oriented agent, an enterprise can establish the following evaluation process.

Step 1: Select real tasks rather than demonstration examples.
The tasks should include normal workflows, abnormal inputs, and changes introduced during execution.

Step 2: Define acceptance metrics.
These may include completion rate, error rate, average number of human interventions, execution time, traceability, and the number of safety violations.

Step 3: Establish a baseline.
Compare the agent with a human workflow, traditional automation scripts, or a standard question-answering model. This prevents evaluation from relying on a single absolute score.

Step 4: Conduct stress tests.
Increase task length, input noise, permission restrictions, and external system failures. Observe whether the system degrades significantly.

Step 5: Analyze failure modes.
For each failure, determine whether the cause was a misunderstanding, planning error, tool failure, permission issue, or missing verification step.

Step 6: Use a phased rollout.
Begin with small-scale, low-risk workflows, and gradually expand the task scope and permission level.

Conclusion

The most important change in next-generation models is not simply more fluent text generation or more accurate visual understanding. It is the ability to connect information, tools, software, and business workflows.

AI systems are beginning to shift from “answer providers” to “task executors.” This is an important change in the way AI applications are designed and used.

However, improved task execution does not automatically mean AGI. A system that genuinely approaches general intelligence must combine broad generalization, long-term stability, reliable error recovery, transparent auditing, and controllable safety boundaries.

When confronted with any claim that “the AGI era has begun,” the most rational response is neither blind enthusiasm nor simple rejection. Instead, the claim should be translated into verifiable engineering metrics:

Can the system consistently complete complex tasks in real environments? Can it stop correctly when it fails? Can people understand and control its behavior?

Only when these answers withstand long-term, independent, and repeatable testing can a technical breakthrough be considered meaningful beyond a compelling demonstration.