From Screen Interaction to AI Agents: How Far Are We from AGI?

1. AI Is Moving from Generating Content to Completing Tasks

Until recently, the primary way to use a large language model was to provide text and receive text in return. Answering questions, drafting documents, and generating code were all forms of content generation. Newer general-purpose models are beginning to evolve into agents: they must understand user intent, observe software interfaces, plan actions, use external capabilities, and adjust their approach based on the results.

This shift can be understood in three layers:

1. Understanding: Interpreting natural language, images, tables, window layouts, and application state.
2. Decision-making: Breaking an ambiguous goal into actionable steps and managing dependencies between tasks.
3. Action: Clicking buttons, filling out forms, uploading files, running programs, checking results, and retrying or reversing actions when needed.

For example, if a user asks an agent to “organize this month’s sales data and create a report,” a conventional model might suggest how to do it. A computer-use agent would need to complete the whole workflow: download the data, clean its fields, identify anomalies, create charts, write the report, and save the results.

Model evaluation therefore cannot focus only on whether the model sounds like an expert. It must also assess whether the model can reliably complete tasks in real environments.

2. Why Screen Understanding Matters

Real-world software is highly fragmented. Some modern systems provide standard interfaces, but many internal business applications, desktop tools, and legacy systems lack complete APIs—or even usable documentation. Traditional automation often depends on fixed coordinates, control identifiers, or predefined workflows. If a page layout changes, the automation may fail.

Vision-driven agents offer another approach: treating the software interface as a general-purpose interaction layer.

In theory, if a person can complete a task through the screen, an agent can do so by:

• Identifying the current window and page state.
• Locating text fields, menus, buttons, and notifications.
• Selecting the next action based on the task goal.
• Observing how the interface changes after each action.
• Determining whether the task is complete or the plan needs adjustment.

The idea that “the interface is an API” does not make conventional APIs obsolete. Structured interfaces are generally faster, more stable, and easier to secure and audit. A sensible system should use a hybrid approach:

• Prefer reliable APIs when available.
• Use visual interaction when APIs are unavailable.
• Require human confirmation for high-risk actions.
• Validate important results in a structured way.

This approach can extend automation to more applications without relying on visual clicking for every task.

3. The Core Architecture of a Computer-Use Agent

A computer-use agent typically combines five components: perception, planning, execution, memory, and safety controls.

3.1 Perception

The perception component gathers information about the current environment, including screenshots, window structure, visible text, focus position, system notifications, and task history.

It must do more than identify what appears on a page. It also needs to understand the semantic relationships between elements. A blue button, for example, could mean “submit,” “save,” or “delete” depending on the page. The model must interpret its label, the page title, the surrounding context, and the user’s goal.

Complex tables can introduce additional challenges, including:

• Merged cells.
• Frozen sections.
• Horizontal scrolling.
• Paginated content.
• Obstructing pop-ups.
• Input validation messages.
• Icons and colors that convey business meaning.

3.2 Task Planning

User goals often leave out the steps needed to achieve them. The planning component must break a goal into subtasks, determine their order, and define what counts as completion.

For example, “enter the eligible customer information into the business system” could involve:

1. Reading the source data.
2. Checking required fields.
3. Removing duplicate records.
4. Signing in to the target system.
5. Entering each record.
6. Checking the system’s response.
7. Summarizing successful and failed records.
8. Producing an auditable execution report.

Planning should not happen only once at the start. Real environments may involve network interruptions, changed page layouts, insufficient permissions, or invalid data. The agent must adapt its plan in response to what it observes.

3.3 Action Execution

The execution component turns decisions into actions, such as:

• Moving the pointer and clicking.
• Entering or deleting text.
• Selecting an option from a drop-down menu.
• Scrolling a page.
• Switching windows.
• Uploading or downloading files.
• Calling authorized business APIs.
• Processing data in a controlled environment.

To reduce mistakes, each action should have preconditions and an expected outcome. The agent should not click through a sequence blindly. Before submitting a form, for example, it should verify that it is on the intended form, that required fields are complete, that validation has passed, and that submission will not cause an irreversible change.

3.4 State Memory

Long-running tasks may involve dozens or hundreds of steps. The current screen alone is not enough to maintain consistency. An agent needs to track:

• The user’s original goal.
• Steps already completed.
• The item currently being processed.
• Failed attempts.
• Important intermediate results.
• Remaining work.
• Permission and risk constraints.

Memory should not simply preserve the entire conversation. A more effective approach is to structure the task state using items such as task IDs, step status, input summaries, output validation values, and error types. This reduces unnecessary context.

3.5 Safety Controls

The more capable a computer-use agent becomes, the greater the potential risk. Independent safeguards must operate outside the model, including permission controls, action allowlists, sensitive-data masking, execution audits, rate limits, and human approvals.

A natural-language instruction should not automatically give a model the ability to delete data, change permissions, or perform other high-risk actions. Permissions must be granted by external systems, not determined by the model itself.

4. How the Agent Execution Loop Works

A typical execution process can be represented with the following pseudocode:

Input: User goal, permission scope, risk policy
Initialize: Task state, action log, maximum step count
Repeat:
    Capture the current interface and system state
    Extract key elements and error messages
    Check whether the task is complete
    If the task is complete:
        Validate the result
        Generate an execution report
        Stop
    Generate candidate actions using the goal and action history
    Check candidate actions against permissions and risk policies
    If an action is high risk:
        Request human confirmation
        Stop if confirmation is not granted
    Execute the action
    Record the action, time, input, and result
    Check whether the interface changed as expected
    If several consecutive attempts fail:
        Replan
        If recovery is still unsuccessful, hand off to a person

This process highlights an important point: a computer-use agent is not simply a vision model that sees a button and clicks it. It is a closed-loop system involving observation, reasoning, execution, verification, and recovery.

5. Why a Single Benchmark Score Cannot Establish “Global Leadership”

When a model is released, attention often focuses on its highest score on a particular benchmark. But computer-use capabilities involve many sources of uncertainty, and a single score cannot fully represent production readiness.

Organizations should pay attention to the following metrics.

5.1 End-to-End Task Success Rate

Recognizing a button does not mean that the business task is complete. The more meaningful metric is how often the entire workflow—from receiving a goal to delivering a result—is completed correctly.

End-to-end success rates should also distinguish between task types, such as:

• Entering information on a single page.
• Transferring data across multiple pages.
• Coordinating work across applications.
• Completing long workflows with exception branches.
• Handling critical business tasks involving approvals and permissions.

5.2 Action Accuracy

Action accuracy includes more than clicking the right location. It also covers whether the agent:

• Selected the right item.
• Acted at the right time.
• Entered the correct data.
• Avoided duplicate submissions.
• Detected when the page state had changed.

In interface interaction, clicking something and clicking the right thing are very different.

5.3 Completion Time and Number of Steps

An agent may be able to complete a task but still be less efficient than a person if it repeatedly makes mistakes, takes unnecessary screenshots, or performs too many ineffective actions.

Evaluation should therefore track:

• Average completion time.
• Average number of actions.
• Proportion of ineffective actions.
• Number of retries.
• Number of human handoffs.

5.4 Stability and Repeatability

Production systems need more than a successful demonstration. They must remain reliable across different data, at different times, and through minor interface changes.

Tests should deliberately introduce disruptions, including:

• Changes in page loading speed.
• Randomly appearing pop-ups.
• Adjusted control positions.
• Unexpected data formats.
• Expired sessions.
• Brief network interruptions.
• Insufficient-permission messages.

An agent demonstrates stronger environmental adaptability only if it can recover correctly under these conditions.

5.5 Cost and Resource Use

Agents may need multiple rounds of visual analysis and reasoning. Evaluation should account for the cost per task, compute use, screenshot transfer volume, and context requirements for long-running tasks.

Potential optimizations include:

• Reanalyzing the interface only when it changes.
• Caching structural information for stable pages.
• Using lightweight strategies for simple tasks.
• Reserving deeper planning for complex tasks.
• Preferring structured APIs where possible.
• Turning repeated workflows into reusable templates.

5.6 Rate of Risk Events

For business applications, safety matters as much as success rate. Organizations should track incidents such as unauthorized access, accidental deletion, sensitive-data exposure, unapproved external sharing, and irreversible changes.

An agent with a high success rate may still be unsuitable for core production work if it occasionally makes serious mistakes.

6. Better Coding Ability Does Not Make a Model Ready to Own Production Systems

With access to terminals and code-generation capabilities, a model can assist with environment checks, log analysis, data processing, test writing, and troubleshooting. But software engineering involves more than generating code. It also requires understanding requirements, respecting architectural constraints, ensuring compatibility and maintainability, managing security, and taking responsibility for deployment.

Organizations using AI coding capabilities should establish at least the following safeguards:

1. Isolated execution environments: Run generated content in a controlled environment before it can access production data.
2. Least privilege: Grant access only to the directories, resources, and operations needed for the current task.
3. Change review: Require developers to review important code and configuration changes.
4. Automated validation: Run syntax checks, unit tests, integration tests, and security scans.
5. Rollback mechanisms: Track all changes and define clear recovery procedures.
6. Dependency controls: Prevent the model from adding unverified or unevaluated external dependencies.
7. Sensitive-data protection: Keep credentials, secrets, and user data out of the model’s context.

A more appropriate role for models is to handle repetitive, verifiable work, while developers remain responsible for setting goals, making architectural decisions, reviewing risks, and approving final delivery.

7. What Still Separates Computer-Use Agents from Artificial General Intelligence?

The ability to operate a computer is an important advance, but it is not enough to declare that artificial general intelligence (AGI) has arrived. Systems closer to general intelligence would need to demonstrate consistent performance in several areas.

7.1 Transfer Across Domains

A model should be able to apply lessons learned in one domain to unfamiliar domains, rather than achieving high scores only on familiar interfaces or fixed tasks.

7.2 Long-Term Autonomy

Complex tasks may take hours or even days. A system must maintain its goals over time without forgetting constraints, drifting away from the task, or repeating work.

7.3 Causal Understanding

A relationship between interface elements does not necessarily establish a causal relationship. Models need to understand why an action produces a particular result, rather than guessing the next step from historical patterns.

7.4 Uncertainty Management

A reliable system should recognize when it is uncertain. When evidence is insufficient, it should ask a question, request confirmation, or stop. Blind confidence is one of the most dangerous flaws an autonomous agent can have.

7.5 Adaptation to New Environments

A system should be able to handle layouts, workflows, and data formats it has never encountered, rather than relying on specific page templates.

7.6 Alignment and Accountability

Greater capability makes clear behavioral boundaries more important. A system must do more than follow user instructions: it must recognize requests involving unauthorized access, fraud, privacy violations, or destructive actions, and comply with organizational safety policies and legal requirements.

For these reasons, the arrival of the AGI era is better treated as a technical proposition that requires ongoing validation—not a conclusion that can be drawn from a single release or benchmark score.

8. A Practical Adoption Path for Computer-Use Agents

Organizations should not hand critical business operations over to an agent from the outset. A phased rollout is safer.

Phase 1: Observe, Do Not Act

Have the model analyze the interface and suggest actions while employees carry them out. This phase helps validate the model’s understanding of business processes and identify common errors.

Phase 2: Automate Low-Risk Tasks

Choose reversible, reviewable tasks such as sorting materials, retrieving information, filling out drafts, and entering test data. The system can act automatically, but should keep complete logs and allow results to be checked.

Phase 3: Use Human-Agent Collaboration

Let the model handle most of the workflow, but pause for human confirmation at critical points such as submitting, deleting, sharing externally, making payments, or changing permissions.

Phase 4: Allow Limited Autonomy

Once task success, safety, and exception recovery meet internal requirements, allow the agent to operate automatically within clearly defined boundaries, with ongoing sample-based audits.

Good candidates for early adoption typically have the following characteristics:

• They occur frequently.
• Their rules are relatively clear.
• Inputs and results are easy to validate.
• Errors have limited impact.
• Changes can be rolled back quickly.
• Manual execution is costly.

Tasks involving large financial transactions, core permissions, legal commitments, or irreversible data changes are not suitable early pilots.

9. Conclusion: From “Can Chat” to “Can Get Things Done”

The real value of newer general-purpose models lies not in producing more fluent responses, but in developing the ability to understand and act within digital environments. Screen understanding can extend automation to legacy systems without standard interfaces. Task planning turns natural-language goals into actionable steps, while execution and feedback loops help models evolve from content assistants into task-oriented agents.

But greater capability does not automatically mean production reliability, and public demonstrations or individual benchmark results cannot establish that AGI has arrived. For developers and organizations, the more important measures are end-to-end success, exception recovery, permission boundaries, auditability, and cost per task.

In the future, users may no longer need to navigate menus one layer at a time. They may describe a goal and have an agent coordinate interfaces, APIs, and business workflows to achieve it. Until that vision is realized, engineering teams must continue to prioritize verifiability, control, reversibility, and accountability. Agents will move from impressive demonstrations to trusted production systems only when they can complete tasks while managing risks reliably.