Computer Use: Mechanisms, Effectiveness, and Cost Evaluation

1. What Is Computer Use, and How Does It Differ from Tool Calling?

Developers often ask: What is computer use? How does it work? How should costs be calculated? What security considerations apply? This article provides an engineering breakdown of these questions.

Over the past two years, the most common way for large models to interact with external systems has been tool calling, also known as function calling. Developers define interfaces, parameters, and return values in advance. The model outputs the parameters for a call, and the business system executes it and returns the result. This approach is clear, stable, and auditable, but it has a natural limitation: the model can only do what developers have already encapsulated into an interface.

If the target software has no API, or only provides legacy desktop clients, internal systems, or graphical configuration consoles, function calling reaches its boundary. The same is true for operations such as moving data between applications, entering information into forms, or exporting data in batches. In these cases, computer use offers another approach: treating the computer desktop itself as an external environment that the model can operate.

From a technical perspective, computer use is not a completely new paradigm. It is a generalization of tool calling:

• The tool granularity expands from “functions” to “devices or environments”.
• The observation space expands from “JSON return values” to “screenshots, control trees, and page structures”.
• The action space expands from “parameter enumeration” to keyboard and mouse events such as clicking, typing, scrolling, and waiting.

From an engineering perspective, it is important to recognize that computer use inherits the orchestration properties of tool calling, but it also inherits the uncertainty of GUI automation. It is suitable as part of an automation capability, but it should not be understood as “letting the model replace humans in all operations”.

2. How Can a Model Use a Computer? Breaking Down the Closed Loop

Computer use can usually be understood as a closed loop of perception, decision-making, execution, and verification. The model does not directly modify the system. Instead, it continuously observes the screen, decides on an action, executes it, and then observes the result.

2.1 Perception Layer: Structure Matters, Not Just Screenshots

The model usually receives a screen snapshot, but pixels alone are not enough. In practice, a combination of environment representations is more common:

• Screenshots: used to determine the overall page state, dialogs, and button positions.
• Control tree or element tree: used to improve click localization accuracy.
• Cursor coordinates or accessibility information: used to help the model understand interactive controls.
• OCR text: used as a fallback when a control tree is unavailable.

This means the environment state cannot rely only on the model “looking at the screen”. The system must also provide stable and traceable structured information. Common demonstration environments use resolutions such as 1280×720 and 1920×1080. If the resolution, scaling ratio, or window layout changes, coordinate mappings from previous actions may fail entirely. Therefore, screenshot resolution, DPI, and scaling ratio should be fixed before execution and recorded in logs.

2.2 Decision Layer: The Model as a Policy Function

In each loop iteration, the model receives the current screenshot, task description, action history, and optionally an element tree. It then outputs the next action. The task description is usually in natural language, for example:

“Open the spreadsheet application, create a new file, enter three rows of data, and save it to a specified directory.”

The model must handle task decomposition by itself, including:

1. Opening the target application.
2. Finding the “new file” menu.
3. Entering the content.
4. Selecting the save path.
5. Confirming the save operation.
6. Verifying that the file exists.

Decision quality depends on the system prompt, the clarity of the task description, the quality of the action history summary, and the model’s ability to understand the current interface.

2.3 Execution Layer: Keep the Action Space Unified, Auditable, and Bounded

It is recommended to encapsulate computer use as a unified interface-operation tool, rather than mixing capabilities such as screenshots, clicks, text input, and script execution into multiple separate tools. Interface actions can include:

• Screenshot
• Mouse move
• Left click
• Double click
• Drag and drop
• Keyboard input
• Key combination
• Scroll
• Wait
• Find element
• Get page text

If command execution, file read/write, or object storage operations are required, it is better to define them as separate tools and clearly specify responsibility boundaries in the system prompt. This avoids multiple entry points for the same capability, which can cause the model to select the wrong tool or repeat actions.

A simplified tool definition example:

{
  "name": "computer",
  "description": "Perform keyboard and mouse operations in an isolated virtual desktop, retrieve screen state, and return screenshots and element information",
  "input_schema": {
    "type": "object",
    "properties": {
      "action": {
        "type": "string",
        "enum": [
          "screenshot",
          "mouse_move",
          "left_click",
          "double_click",
          "type_text",
          "key",
          "scroll",
          "wait",
          "find_element",
          "get_text"
        ]
      },
      "coordinate": {
        "type": "array",
        "items": {
          "type": "integer"
        }
      },
      "text": {
        "type": "string"
      },
      "timeout_seconds": {
        "type": "integer"
      }
    },
    "required": ["action"]
  }
}

The key point is: the interface action tool should only handle interface operations. It should not mix scripts, files, network access, or permission management into the same tool enumeration.

2.4 Verification Layer: Loop Termination, Timeouts, Retries, and Cost Governance

In engineering implementation, the core is a loop with a maximum step count, timeouts, retries, and context governance. A skeleton is shown below:

MAX_STEPS = 40
STEP_TIMEOUT = 30          # Timeout for a single model call or action execution, in seconds
MAX_RETRY = 2              # Maximum retries for a single tool action
MAX_IMAGE_EDGE = 1280      # Upper bound for downsampling the screenshot's long edge, used to control token costs
KEEP_RECENT_IMAGES = 3     # Keep only the 3 most recent raw screenshots; earlier screenshots are converted into text summaries
while step < MAX_STEPS:
    screenshot = capture_screen(scale_to=MAX_IMAGE_EDGE)
    element_tree = get_element_tree(
        source="accessibility_tree_or_dom_or_ui_automation",
        fallback="ocr_if_no_structured_tree",
    )
    resp = call_model_with_timeout(
        messages=messages,
        tools=[computer_tool],
        tool_choice={"type": "auto"},
        timeout=STEP_TIMEOUT,
    )
    for block in resp.content:
        if block.type == "tool_use":
            result = execute_tool_with_retry(
                tool_name=block.name,
                payload=block.input,
                retries=MAX_RETRY,
                timeout=STEP_TIMEOUT,
            )
            messages.append(tool_result(block.id, result))
    if task_completed(resp):
        break
    step += 1

This skeleton highlights several runtime prerequisites:

• MAX_STEPS: prevents the model from entering an infinite loop.
• STEP_TIMEOUT: prevents model calls or interface actions from hanging for a long time.
• MAX_RETRY: provides limited retries for recoverable failures.
• Screenshot downsampling: avoids continuous consumption of large numbers of tokens by high-resolution images.
• Control tree source: prioritize the system accessibility tree, browser DOM, or desktop UI Automation. Use OCR as a fallback when they are unavailable.
• History context governance: keep only the most recent screenshots, compress earlier actions into summaries, and avoid secondary cost growth.

In addition, the implementation must handle action failures, page loading delays, popup interception, permission dialogs, infinite-loop detection, and repeated action detection. A real implementation is usually much more complex than this skeleton.

3. How to Evaluate Effectiveness? Which Scenarios Are Suitable, and Which Require Caution?

The effectiveness of computer use cannot be judged only by demonstration videos. It requires evaluation criteria. Otherwise, it is easy to mistake “a task succeeded once” for “this type of task is reliably usable”.

3.1 Evaluation Criteria: What Should Be Recorded?

At a minimum, the following fields should be recorded:

• Task sample name and description
• Target software name and version
• Operating system and desktop environment
• Model version and system prompt version
• Screenshot resolution, scaling ratio, and whether a control tree was used
• Tool permission scope
• Average number of steps
• Maximum number of steps
• Success rate
• Failure types
• Number of manual interventions
• Token consumption per task

The success rate can be defined as: number of tasks that pass automated verification / total number of task samples. If tasks require manual acceptance, it is also necessary to distinguish between “successful model execution” and “successful business acceptance”.

For example, if the single-step success rate is assumed to be 95%, the theoretical probability of 20 consecutive correct steps is approximately 0.95^20, or about 35.9%. This is only a mathematical example used to illustrate error accumulation, not a measured result. Real projects must replace this assumption with actual task samples.

3.2 More Suitable Scenarios

Computer use is more suitable for tasks with clear objectives, stable interfaces, short steps, and verifiable feedback. Examples include:

• Opening an application and entering a fixed form.
• Exporting data in batches from a web admin console.
• Moving files between applications.
• Executing a fixed set of configuration clicks.
• Performing UI inspection and status checks.
• Providing lightweight automation for legacy software without an API.

These tasks share common characteristics: the path is relatively unique, feedback is immediate, and failures can be rolled back.

3.3 Scenarios Requiring Caution

The following scenarios carry higher risk:

1. Long-chain tasks
   The more steps there are, the more obvious error accumulation becomes. A small error in a single step can significantly reduce the overall success rate.

2. Information-dense interfaces
   In spreadsheets, IDEs, and multi-panel tools, buttons are closely arranged, and coordinate clicks are more likely to be inaccurate.

3. Asynchronous and dynamic content
   Lazy loading, animations, progress bars, and background tasks may cause screenshots to capture intermediate states.

4. Non-visual feedback
   Scenarios such as CAPTCHAs, native system file dialogs, and drag-and-drop uploads may have higher failure rates.

5. Implicit business rules
   If a system does not have explicit validation logic, the model may perform operations that are technically legal but business-wise incorrect.

3.4 Common Failure Modes and Troubleshooting Order

It is recommended to troubleshoot in the following order:

1. Is the screenshot stale? Is the page still loading?
2. Is the control tree missing or mismatched with the version?
3. Have coordinates shifted because the resolution changed?
4. Is the wait time insufficient?
5. Was the action blocked by permissions or popups?
6. Did the model repeat the same action?
7. Is the task termination condition configured correctly?

Compared with structured scripts, computer use has an advantage: replayability. Each step’s screenshot, action, and model output can be recorded. But the risk is also here: the model may execute instructions that appear on the screen. Therefore, security governance is required.

4. Where Do the Costs Come From? How to Estimate the Cost per Task?

The cost of computer use is usually higher than intuition suggests. The reason is not the price of a single model call, but the structure of the calls themselves.

4.1 Costs Mainly Come from Four Parts

First, image input overhead.
Each loop iteration usually requires sending a screenshot. Image input is converted into tokens and has relatively high information redundancy compared with text. Higher screenshot resolution means higher cost.

Second, loop-step amplification.
Cost is roughly proportional to the number of steps. More complex tasks require more steps. If the model needs to try, backtrack, or wait, the number of steps may exceed expectations.

Third, repeated computation of historical context.
If every iteration carries all historical screenshots, costs will rise quickly. It is necessary to keep only the most recent K screenshots and compress earlier steps into text summaries.

Fourth, rework and failure costs.
Failed tasks have already consumed tokens and may also trigger manual fallback. Therefore, the actual cost per task should be amortized by the success rate.

4.2 Example Estimation Parameters

A preliminary estimate can be made using the following parameters:

• Screenshot resolution: 1280×720
• Screenshot input per step: approximately 800–1500 tokens, depending on compression and scaling
• Keep the most recent 3 raw screenshots
• Convert earlier screenshots into text summaries
• Average number of steps: 20
• Task success rate: 70%

The cost per task can be estimated as follows:

Cost per task ≈
(base system prompt
 + Σ per-step(screenshot tokens + UI text + model output)
 + context summary overhead)
 / task success rate

Do not focus only on the model unit price. Focus on the total cost per task.

4.3 Optimization Directions

• Reduce screenshot frequency: do not take a new screenshot if the interface has not changed.
• Downsample resolution: use a resolution that is sufficient, and avoid unnecessary 4K screenshots.
• Use control trees to reduce dependence on images.
• Prefer APIs or CLIs, and use GUI operations only as a fallback.
• Cache decision templates for similar tasks.
• Set a hard maximum step limit.
• Quickly stop failed tasks to avoid continuing to burn budget.

4.4 Comparison with RPA and Browser Automation

Computer use is not the only path. When making a selection, multiple approaches should be compared:

Approach	Development Cost	Maintenance Cost	Suitable Scenarios	Main Failure Risks
API/CLI	Medium	Low	Systems with open interfaces	Interface changes, insufficient permissions
RPA/browser automation	Medium	Medium	Fixed processes with clear rules	Page changes, control recognition failures
Computer use	Medium-high	Medium-high	No API, complex graphical interfaces, variable tasks	Visual errors, dynamic pages, permission risks

For legacy software automation, computer use is a feasible general-purpose automation path worth evaluating. Whether to adopt it should be decided based on task stability, budget, compliance requirements, and the ability to provide manual fallback.

5. What to Pay Attention to During Deployment? Sandboxing, Permissions, and Governance Checklist

Because computer use executes real keyboard and mouse events, engineering implementation must uphold a bottom line: default untrusted, least privilege, manual confirmation at key nodes, and full process logging.

5.1 Security Baseline

• Strong isolated sandbox: All operations are performed in a containerized virtual desktop. By default, host files, clipboard, network, and peripherals are not visible.
• Do not place credentials in the environment: Secrets, CAPTCHAs, and private data should not appear on the screen. Authentication should be injected through a separate channel whenever possible.
• Least privilege: File operations should be limited to specified directories. High-risk actions should be prohibited, such as deleting system directories, modifying firewall settings, or executing unknown downloaded scripts.
• Manual confirmation at key nodes: Operations involving payment, sending, deletion, or batch changes must be paused for confirmation.
• Full audit logging: Screenshots, actions, model outputs, execution results, and failure reasons should be written to disk item by item.
• Prompt injection protection: Content such as web pages, documents, and emails must be treated as data, not as instructions.

5.2 Task Classification Checklist

Task Level	Example	Recommended Strategy
Read-only query	Check status, export reports, inspect information	Can be executed automatically, with audit records
Low-risk write operation	Create draft, modify non-production data	Execute automatically, with a step limit
High-risk write operation	Submit order, modify permissions, batch delete	Manual confirmation required
Cross-system change	Operations in finance, customer, or production systems	Manual confirmation + replay audit + two-person review

5.3 Architecture Recommendation

A hybrid architecture is recommended:

1. Prioritize structured channels such as APIs, CLIs, and SDKs.
2. For systems without interfaces, evaluate RPA or browser automation.
3. Consider computer use only when tasks are highly dynamic, interfaces vary greatly, and stable interfaces are lacking.
4. All paths should share a unified task description layer, making it easier to switch dynamically based on software capabilities.

When building a self-hosted platform, you can evaluate general cloud platform capabilities such as model services, container platforms, object storage, logging, and monitoring, and design the solution according to business isolation and compliance requirements without being tied to a single implementation.

6. Conclusion: Evaluate the Task First, Then the Capability, and Finally the Permissions

Computer use solves a real problem: a large number of legacy software systems do not have APIs, but still need to be automated. It provides a general-purpose automation capability for these scenarios, closer to the way humans operate.

However, it is currently more suitable as an “automation assistant during capability exploration”, rather than an unattended production executor. The real concern is not “whether the model can operate a computer”, but whether we have given it permissions beyond what the task requires.

A one-sentence selection rule:

Prefer APIs when APIs are available. Prefer scripting when scripting can solve the problem. Only evaluate computer use when the graphical interface is complex, stable interfaces are unavailable, and tasks need dynamic adjustment, and pair it with sandboxing, auditing, manual confirmation, and cost governance.