The Capabilities and Future of Computer-Operating AI Agents

1. From “Can Operate a Computer” to “Can Complete a Task”

AI agents that can read screens, understand interfaces, click, type, drag, and process files have attracted significant attention. On the surface, these capabilities reproduce the way humans use computers. More fundamentally, they address a long-standing automation challenge: how can an agent complete an end-to-end task when a system lacks stable APIs, business processes are spread across multiple tools, and critical data is accessible only through graphical interfaces?

“Operating a computer” should therefore not be reduced to mouse and keyboard automation. It is better understood as the ability to execute tasks across different environments. A mature computer-use agent typically needs the following capabilities:

1. Visual understanding: Identifying page layouts, form fields, menu hierarchies, status messages, charts, dialogs, and other interface elements.
2. Semantic reasoning: Understanding user goals such as “organize this week’s abnormal orders,” “generate an operational analysis report,” or “synchronize approval results with the internal system.”
3. Action planning: Breaking a natural-language objective into multiple operations and dynamically adjusting the path based on intermediate results.
4. State and context memory: Maintaining task context throughout long workflows while avoiding duplicate actions, missed steps, or overwriting completed results.
5. Exception recovery: Responding to expired sessions, redesigned pages, timeouts, insufficient permissions, and similar conditions by pausing, rolling back, asking for help, or using an alternative path.
6. Result verification: Confirming that the intended outcome was actually achieved—for example, that a file was created, data was written, or a workflow was submitted successfully.

The business value of an agent is not determined by how many clicks it completes. The key question is whether it can deliver results in a controlled, verifiable, and auditable way.

2. Why Small Accuracy Gains Matter So Much in Long Tasks

Many evaluations measure single-step accuracy, page-recognition accuracy, or success rates for specific tasks. In real enterprise environments, however, a task is rarely completed in one step. It may require dozens or even hundreds of sequential actions. Under these conditions, small differences in step-level capability can be amplified significantly.

Assume that a task requires 40 consecutive critical actions and that each action succeeds independently with probability p. The approximate probability of completing the entire task is:

Task success rate = p ^ 40

When the step-level success rate is 95%:

0.95 ^ 40 ≈ 12.9%

When it improves to 98%:

0.98 ^ 40 ≈ 44.6%

When it reaches 99%:

0.99 ^ 40 ≈ 66.9%

In long-horizon agent tasks, an improvement of only a few percentage points in step-level accuracy can therefore multiply the overall task success rate. More importantly, failure is not simply a matter of “the task was not completed.” An error in an intermediate step may trigger human intervention, data rollback, state verification, and re-execution. The resulting cost can be far greater than the cost of the original action.

From an engineering perspective, efficiency improvements in long-running tasks generally come from three areas:

• Reducing recognition and execution errors at each step;
• Minimizing unproductive exploration and repeated actions after an error;
• Improving self-verification, recovery, and replanning.

The core challenge for computer-operating agents is therefore not just understanding what is on the screen. It is completing complex workflows reliably in uncertain environments.

3. Why GUI Automation Still Matters

Some argue that graphical interfaces are merely a transitional form of software interaction designed for humans. In the future, systems should expose structured capabilities directly to agents instead of requiring agents to imitate human clicks. This view is reasonable, but it does not mean that GUI automation has no current value.

Many organizations still depend on legacy software, internal management systems, desktop applications, and manual processes. These systems commonly have several characteristics.

1. Incomplete System Interfaces

Many business systems expose APIs for only part of their functionality. Core operations may still require a web page or desktop client. Searching, reviewing, downloading, confirming, and archiving may be distributed across different screens without a unified orchestration mechanism.

2. Fragmented Cross-System Workflows

A typical business operation may involve multiple systems:

Read data → Log in to an administrative portal → Search records → Download attachments → Fill out a form → Submit for approval → Send a notification → Update a tracking ledger

Even when each system provides some internal automation, employees may still need to connect the systems manually. Computer-operating agents can perform this “last-mile” integration work.

3. High Modernization Costs for Legacy Systems

Converting every existing application into a standardized, API-enabled, agent-ready service requires substantial investment in development, testing, security assessment, and migration. For workflows that are infrequent but tedious, a controlled GUI agent may offer a better return on investment.

4. Human Workflows Are Naturally Interface-Centered

Enterprise rules, forms, approval stages, permission boundaries, and audit requirements are often embedded in existing interfaces. If an agent can understand and use those interfaces, an organization can gain automation benefits without immediately rebuilding all of its business infrastructure.

For these reasons, GUI automation is likely to remain an important adaptation layer between agents and the existing digital world for a considerable period of time.

4. Why “Using a Computer Like a Human” Is Not the Final Form

Although GUI automation has clear value, making agents depend entirely on visual recognition and interface clicks has inherent limitations.

First, a GUI is an interaction layer designed for humans, not an execution protocol designed for machines. Humans use visual layouts, buttons, menus, and messages to understand a system. Machines are generally better served by directly reading structured state, invoking explicit actions, and receiving verifiable results.

For example, creating a record through a GUI may require:

Open the page → Find the entry point → Click Create → Fill in fields → Select a dropdown option → Submit → Wait for a confirmation → Check the result

A structured capability designed for agents could abstract the same task as:

create_record({
  "name": "Example object",
  "priority": "high",
  "owner": "user_a"
})

Both approaches can complete the task, but the second is generally more stable, efficient, controllable, and auditable.

Interfaces are also fragile. A visual redesign, a changed button position, an additional dialog, or a minor text change can disrupt automation based on screen locations. Even when an agent has some ability to generalize, complex scenarios can still lead to incorrect judgments.

GUI interactions can also hide or lose structured semantics. A status such as “processing,” “completed,” or “reversible” may be easy for a human to understand. Without backend state definitions, field constraints, and workflow rules, however, an agent may have to infer the meaning from the screen. This increases decision uncertainty.

The more reasonable long-term conclusion is not that GUI automation will disappear immediately. Instead:

GUI automation will remain important for compatibility with legacy systems, while structured services, event-driven architectures, and machine-readable protocols will gradually become the default design approach for new systems.

5. From Human-First Interfaces to Agent-Callable Systems

If enterprises want to fully unlock the productivity of agents, they should not focus only on making models better at clicking. They also need to reconsider how software systems are designed. Future software architectures will need to serve two types of users:

• Human users, who understand and operate business processes through web, mobile, or desktop interfaces;
• Agent users, who complete tasks through structured context, controlled actions, and explicit feedback.

This requires a number of design principles.

1. Expose Core Business Capabilities as Services

Core actions such as querying, creating, approving, exporting, archiving, and sending notifications should be exposed as stable services rather than existing only behind buttons on a specific page.

Service-based design does not mean that every function must be openly available. High-risk actions can still require tiered permissions, human confirmation, transaction limits, or multi-step approval.

2. Return Machine-Verifiable Results

After an agent invokes a capability, it should not receive only an ambiguous message such as “Operation successful.” It should receive a clear state, for example:

{
  "success": true,
  "record_id": "generated_id",
  "status": "pending_review",
  "next_action": "wait_for_approval"
}

This allows the agent to continue planning based on actual system state instead of guessing from screen messages.

3. Include Permission Boundaries in Task Design

When an agent can operate across systems, permissions should not be tied only to whether an account can log in. They should also be tied to the scope of the task, the data involved, and the actions allowed.

For example, an agent may be restricted to:

• Reading data only within a specified business domain;
• Creating drafts but not submitting them directly;
• Executing individual operations but not bulk deletions;
• Waiting for human confirmation before handling funds, private data, or external communications.

These restrictions should be enforced by system mechanisms rather than relying solely on prompt instructions.

4. Establish an End-to-End Audit Trail

When an agent performs a task, the system should record the original objective, planning steps, invoked actions, key interface states, result feedback, and exception information. This helps identify the causes of failure and supports enterprise compliance, accountability, and process governance.

6. Choosing Between GUI Agents and Structured Agents

In practice, organizations do not need to choose exclusively between interface operation and API-based execution. A more practical strategy is to use a hybrid architecture and select the execution method according to the characteristics of each task.

Scenarios Where GUI Agents Are Often Preferable

• The system is old and does not currently provide usable APIs;
• The task is not frequent, but manual execution is costly;
• The workflow spans multiple heterogeneous systems;
• The interface process is relatively stable and results can be verified;
• The goal is to assist employees rather than provide fully unattended execution.

Examples include regularly collecting data from several administrative portals, downloading and categorizing files, assisting with repetitive form completion, and performing low-risk information checks.

Scenarios Where Structured APIs Are Preferable

• The operation is frequent or requires high concurrency;
• It involves funds, orders, permissions, privacy-sensitive data, or other high-risk assets;
• Business rules are complex and require strict validation;
• The system is newly built or undergoing modernization;
• High availability, low latency, and large-scale observability are required.

Examples include batch processing business records, triggering rule engines, generating standardized reports, updating master data, and scheduling large-scale jobs.

Recommended Hybrid Execution Model

Task understanding and planning
        ↓
Call structured business capabilities first
        ↓
If the target system lacks an API
        ↓
Use controlled GUI operations as a fallback
        ↓
Verify results and record audit data
        ↓
Route exceptions to human confirmation or a degraded workflow

This model makes use of existing systems while avoiding the need to build every critical process on fragile page interactions.

7. Engineering Issues That Matter When Building Computer-Operating Agents

The gap between a demonstrable agent and a production-ready agent often has less to do with whether the model is sufficiently intelligent and more to do with whether the engineering system is complete.

1. Task Decomposition and Checkpoint Design

Agents should not be asked to execute excessively long, open-ended instructions in a single run. Tasks should be divided into verifiable stages, with checkpoints at critical points.

For example:

Stage 1: Confirm the target object
Stage 2: Retrieve the required data
Stage 3: Generate the content to be submitted
Stage 4: Obtain human confirmation for critical fields
Stage 5: Execute the submission
Stage 6: Verify the final state

Each stage should have clearly defined inputs, outputs, and failure-handling procedures. This can significantly reduce the risk of losing control during long-running tasks.

2. Idempotency and Rollback

Network failures, page delays, or incorrect state recognition may cause an agent to repeat an action. For operations such as creating, submitting, sending, or deleting, systems should provide idempotency controls to prevent duplicate execution and serious side effects.

For example, before performing a write operation, the system can check whether the target already exists. After execution, it can record a unique task identifier. If a failure occurs, it should preserve the intermediate state instead of retrying blindly.

3. Human Confirmation Is Not a Failure

In high-risk scenarios, proactively requesting confirmation is a sign of maturity rather than a weakness. A well-designed system should determine which actions can be automated and which require human approval.

For example:

Low risk: Querying, organizing, categorizing, and generating drafts
Medium risk: Modifying ordinary records and initiating internal workflows
High risk: External communication, data deletion, financial operations, and permission changes

Different risk levels should correspond to different authorization policies. This is essential for balancing efficiency and safety.

4. Observability and Evaluation

In addition to tracking overall task success rates, teams should continuously monitor:

• Average task duration;
• Human intervention rate;
• Number of retries;
• Successful exception-recovery rate;
• Interception rate for high-risk actions;
• Failure rate caused by interface changes;
• Stability across different task types.

Only an evaluation system based on real business workflows can show whether an agent is creating genuine value.

8. Conclusion: Reliable Collaboration Matters More Than Clicking Ability

Computer-operating agents mark an important step toward intelligent systems that can directly enter real digital work environments. They can connect legacy applications that lack standardized interfaces and significantly lower the barrier to automating cross-system workflows. For enterprises today, this is a highly practical technology path.

From a longer-term perspective, however, the GUI should not be the endpoint of agent capabilities. The central challenge is not to make agents imitate human computer use indefinitely. It is to build software infrastructure that is more compatible with agents: business capabilities should be callable, state information should be understandable, permission boundaries should be enforceable, execution results should be verifiable, and the entire process should be auditable.

In the short term, strong GUI operation will remain an important lever for deploying agents. Over time, however, the upper limit of productivity will be determined by whether the software world can evolve from a collection of interfaces designed for humans into a task network in which humans and agents can work together.