Computer-Use Agents: Principles, Performance, and a Secure Enterprise Deployment Guide

Large language models are moving beyond text generation to operating computers. A new generation of computer-use agents can view screens, identify buttons and input fields, and perform actions such as clicking, typing, and scrolling without relying on dedicated application interfaces.

This article explains how computer-use agents work, how to measure their performance, and how to design their architecture, security controls, and implementation approach for enterprise deployment.

1. How Does Computer Use Work?

A typical computer-use loop consists of four steps:

1. Capture the screen state. The agent usually receives a screenshot and may also receive an accessibility tree or DOM structure when available.
2. Interpret the interface and task. The model analyzes the current interface and task objective, then determines the next action.
3. Execute the action. An executor translates the model’s output into system-level mouse and keyboard events.
4. Verify the result. The system captures the screen again, checks whether the expected state has been reached, and starts the next iteration.

The coordinates produced by the model are usually based on the image dimensions it receives. The physical display, however, may use a 4K resolution or system-level scaling. The executor must therefore map the model’s coordinates to physical pixels according to the screenshot’s scaling ratio.

If this conversion is omitted, click locations will be offset. This is one of the most common implementation errors in computer-use systems.

Multimodal input also has a cost. Higher-resolution screenshots consume more visual tokens and increase inference latency. Four common optimization strategies are:

1. Lower the screenshot resolution according to the requirements of the task.
2. Send new screenshots only when the interface changes, using incremental screenshots where possible.
3. Clean webpage DOM data by removing advertisements, hidden elements, and irrelevant styles, while retaining only interactive nodes.
4. Keep only the most recent key screenshots during long tasks. Older screenshots can be replaced with text summaries.

2. How Should Visual Grounding and Structured Information Be Combined?

A vision-only approach is effective at identifying the position and meaning of buttons, but it can misinterpret interfaces with very small text or visually similar controls.

A DOM- or accessibility-tree-based approach is usually faster and more stable, but it cannot handle every interface. It may fail with Canvas-based applications, remote desktops, or non-standard controls.

In practice, a hybrid strategy is often more reliable:

• Use the accessibility tree or DOM to retrieve labeled elements, including their names, roles, and states.
• Fall back to visual grounding when the target cannot be found through structured information.
• For critical actions, compare the interface state before and after execution.

For example, after submitting a form, the system should check for a success message or a page transition. It should confirm that the action actually took effect rather than assuming success after the click event was sent.

3. How Should Performance Be Measured?

Whether one approach is faster depends on measurable results. Claims of “multiple times faster” in promotional materials often lack sufficient context, so they should not be treated as directly comparable evidence.

At a minimum, an evaluation should track three categories of metrics:

• Task success rate: The percentage of tasks completed and verified successfully within a fixed task set.
• End-to-end latency: The total time from instruction submission to completion, including model inference, screenshot capture, network transmission, and retries.
• Per-task cost: The number of tokens consumed, computing resources used, and manual corrections required for each task.

OSWorld is a commonly used public benchmark. It evaluates an agent’s ability to complete open-ended computer tasks in real desktop operating systems and applications.

In the original OSWorld paper, early models achieved success rates in the low-teens range, while humans completed approximately 70% of the tasks. Leading models have improved substantially over the past two years, but the exact figures change quickly with model versions and evaluation settings.

When citing benchmark results, always specify:

• The model version
• The evaluation date
• The maximum number of steps
• Whether retries were allowed

Without these conditions, benchmark numbers cannot be compared directly. The latest figures should be checked against the official leaderboard.

The following table compares three automation approaches across five dimensions. These are qualitative assessments; actual values should be measured in the target business environment.

Comparison dimension	Traditional RPA	Direct API calls	Computer-use agents
Adaptability to interface changes	Low; depends on fixed element locations	Not applicable; reliable when the API is stable	Relatively high, but depends on visual judgment
Development and maintenance cost	Scripts require frequent maintenance after UI changes	One-time integration cost when the API is available	More debugging during initial deployment, potentially less maintenance later
Execution predictability	High	High	Lower; verification and retry mechanisms are required
Per-task latency	Relatively short	Usually the shortest	Depends on screenshot frequency and the number of inference steps
Suitable scenarios	Stable, repetitive processes	Systems with available APIs	Processes without APIs or with frequently changing interfaces

A practical approach is to select a representative set of business tasks, record the three categories of metrics above, and then decide whether to expand deployment. Public benchmark numbers should not be applied directly to a specific business workflow.

4. How Should an Enterprise Architecture Be Designed?

A computer-use agent must wait for networks and interfaces to load while also performing model inference. Its workload is therefore both I/O-intensive and compute-intensive.

A cloud-native architecture commonly includes the following components.

Isolated Sandboxes

Assign each task an isolated sandbox, such as a browser running in a container or a lightweight virtual machine.

Tasks should not share sessions or files. When a task fails, its sandbox can be destroyed and recreated directly.

Event-Driven Scaling

In Kubernetes, the default Horizontal Pod Autoscaler generally scales based on CPU or memory metrics. For agent workloads, however, the main bottleneck is often queue buildup. Scaling based on queue length is therefore usually more appropriate.

KEDA can read the lag of a Kafka consumer group directly. The following is an example of a ScaledObject based on Kafka lag:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: computer-use-worker-scaler
  namespace: agent-system
spec:
  scaleTargetRef:
    name: computer-use-worker
  minReplicaCount: 1          # Keep one replica available to reduce cold-start latency
  maxReplicaCount: 20         # Limited by cluster resources and model API quotas
  cooldownPeriod: 300         # Wait 300 seconds before scaling down to reduce flapping
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka.agent-system.svc:9092
        consumerGroup: computer-use-workers
        topic: automation-tasks
        lagThreshold: "10"    # Scale out when average backlog exceeds 10 messages per replica
        offsetResetPolicy: latest

The value of lagThreshold should be estimated from average task duration. For example, if a task takes approximately two minutes and the target waiting time is no more than ten minutes, the backlog threshold can be derived accordingly.

Queue-Based Load Smoothing

User instructions should first be written to Kafka or RabbitMQ. Backend workers can then consume and execute tasks asynchronously, while results are returned through callbacks or a status-query API.

Even when task volume spikes, the queue prevents the system from becoming overloaded immediately.

5. What Security Risks Must Be Addressed?

An agent that can operate internal CRM systems and access sensitive customer data requires stronger controls than ordinary script automation.

The following six measures should be implemented together.

1. Human-in-the-Loop Approval for High-Risk Actions

For payments, transfers, bulk deletion, permission changes, external email delivery, and similar operations, the agent should only generate a proposed action plan. The operation must wait for confirmation from an authorized person through an approval system.

Approval must follow a fail-closed policy:

• If approval times out, treat it as rejected.
• If the approval system is unavailable, treat it as rejected.
• If the callback is lost, treat it as rejected.
• If the callback returns an abnormal result, treat it as rejected.

The task should terminate and record the reason. It must never be allowed to proceed by default.

The approver, approval time, decision, and timeout status should all be written to the audit log.

2. Defenses Against Visual Prompt Injection

Webpages, images, and PDFs may contain instructions designed to manipulate the model. Examples include white text on a white background, extremely small text, or pop-ups disguised as system messages.

Key defenses include:

• Treat all page content as data to be processed, not as instructions.
• Clearly separate system instructions from page content in the model input.
• For sensitive actions, make decisions based only on the user’s original task. Do not allow page text to change the objective.
• If the page contains instructions such as “execute immediately” or “ignore previous requirements,” stop the operation and request human confirmation.

3. Operation Scope Allowlists

At the scheduling layer, restrict the domains and internal IP ranges that the agent can access. Sites that have not been explicitly declared should be blocked.

4. Credential Management

Passwords must never enter the model context. A secure proxy should inject login credentials at the network layer, while the agent passes only the intended operation.

5. End-to-End Auditing

Store screenshots, action sequences, and key decisions so that operations can be reconstructed later.

Screenshots may contain customer or other sensitive data. They must be encrypted at rest and assigned an appropriate retention period.

6. Timeout and Failure Controls

Set maximum step counts and total execution time limits. Retry counts should also be bounded.

When failures continue, stop the task and transfer it to a human operator. This prevents the agent from repeatedly acting in an incorrect state.

6. How Should the Task Scheduler Be Implemented?

The following is a pseudocode prototype that illustrates the control flow. agent_sdk and cloud_infra are fictional modules and must be replaced with the SDK and message-queue client used in the actual environment.

# Note: This is a pseudocode prototype.
# agent_sdk and cloud_infra are fictional module names and must be
# replaced with the actual SDK and message-queue client.
import logging
import time
from agent_sdk import ComputerUseClient              # Fictional SDK
from cloud_infra import MessageQueue, ObjectStorage  # Fictional wrappers
logger = logging.getLogger("agent_scheduler")
# Keywords are only the first filter and cannot