Computer Use Evaluation and Engineering Practices for Large Language Models

1. Background and Problem: LLMs Move from Generating Content to Executing Tasks

Over the past few years, the core value of large language models has mainly appeared in scenarios such as text generation, coding assistance, knowledge question answering, and content summarization. These tasks share a common characteristic: the model primarily deals with the linguistic relationship between input and output, with limited direct access to real business systems.

As task complexity increases, enterprise users are no longer only asking whether a model can answer a question. They increasingly care about whether a model can complete a specific piece of work.

Computer use capability generally refers to a model’s ability to perceive the current state of a graphical interface or desktop environment, understand user goals, and invoke appropriate tools to perform actions such as clicking, typing, scrolling, opening files, reading information, and submitting forms. This capability pushes large language models further from conversational assistants toward execution-oriented agents.

From a technical perspective, computer use is not simply about generating natural language. It involves building a closed loop of perception, planning, execution, feedback, and correction. The model needs to understand interface elements, determine which step it is currently at, decide the next action, and adjust based on execution results. This expands how large language models interact with software systems and provides a new automation approach for scenarios involving multiple systems, multiple applications, or limited API coverage.

However, computer use should not be equated with letting a model replace humans in operating a computer. It is more like a general-purpose interactive execution agent. To enter real production environments, it must be combined with engineering controls such as task orchestration, permission control, runtime isolation, log auditing, and cost governance.

2. Technical Mechanism: The Essence of Computer Use Is a Tool-Calling Loop

Computer use capability can usually be abstracted into five core stages: environment perception, task decomposition, action generation, execution feedback, and result verification.

2.1 Environment Perception

Environment perception is the foundation that allows a model to understand “what is currently on the screen.” Common methods include:

• Capturing interface screenshots and combining them with visual understanding to identify buttons, input fields, text, popups, and other elements;
• Reading structured data such as window structures, control trees, and accessibility information;
• Extracting semantic information such as page text, table content, file names, and paths;
• Recording historical operations to form short-term context and help the model understand the current task progress.

Compared with simply reading API return values, environment perception is closer to the way humans use software. The model does not rely solely on predefined standardized interfaces. Instead, it observes interface changes to infer system state. As a result, it is better suited to tasks where systems lack open interfaces, page structures are complex, or operational processes are highly customized.

2.2 Task Decomposition

Complex tasks often cannot be completed with a single action. For example, “organize qualifying data in a system and export a report” may involve logging in, filtering, paginating, copying, pasting, saving, and verifying results. Computer use capability usually requires the model to break the goal down into executable steps.

The key to task decomposition is defining clear goal boundaries and termination conditions. If the model does not know when the task is complete, it may get stuck in repeated clicking, repeated navigation, or ineffective waiting. In practice, tasks often need structured descriptions, including target objects, filtering conditions, success criteria, and exception-handling strategies.

2.3 Action Generation

After understanding the task and the environment, the model needs to generate concrete actions. Common action types include:

• Clicking a control;
• Entering text into an input field;
• Selecting an item from a dropdown menu;
• Scrolling a page or window;
• Opening a file or switching applications;
• Reading screen content;
• Waiting for a page to finish loading;
• Submitting a form or saving results.

These actions are usually completed through tool-calling mechanisms. Compared with invoking a single API function, computer use capability emphasizes the combination of multiple actions. The model needs to know when to click a button, when to wait first, when to read feedback, and when to stop.

2.4 Execution Feedback

Execution feedback is what distinguishes computer use from ordinary generation tasks. After each action is executed, the system needs to return the result to the model. For example:

Task input: user goal
Environment state: current interface description, window structure, interactable controls, text content
Model output: action sequence (click, type, scroll, wait, read)
Execution result: success, failure, exception information, latest interface state

If a click succeeds but the page does not change, the model may need to determine whether the cause is network latency, insufficient permissions, or an incorrect element locator. If the entered content is rejected by form validation, the model needs to read the error message and correct the input. If a popup appears, the model needs to identify its type and close or confirm it accordingly.

2.5 Result Verification

Task completion does not mean the model can exit immediately. For critical tasks, result verification mechanisms are required. For example, the system may need to check whether the target file was generated, whether the number of records is correct, whether a success message appears on the page, or whether the exported content meets format requirements.

Result verification can be performed in multiple ways:

• Rule-based text verification, such as checking whether key fields exist;
• Structured data verification, such as comparing record counts, amount ranges, or sequence continuity;
• Screenshot-based or interface-state verification, such as confirming that a success page has appeared;
• Manual review for high-risk tasks.

Only by incorporating result verification into the workflow can computer use capability move from “seemingly able to operate” to “safe to use.”

3. Effectiveness Evaluation: Look Beyond Demonstrations to Closed-Loop Quality

The effectiveness of computer use capability should not be judged only by whether a few demonstrations complete successfully. Real business environments are more complex. Interface changes, network latency, permission restrictions, unexpected popups, multilingual interfaces, and dynamic loading can all affect task stability. When evaluating this capability, it is recommended to consider the following dimensions.

3.1 Task Completion Rate

Task completion rate is the most basic metric. It measures whether the model can complete a goal under specified conditions. When evaluating it, distinguish between:

• Single-attempt completion rate: success on the first execution;
• Multi-turn correction completion rate: success after several rounds of error correction;
• Human-assisted completion rate: completion requires human intervention.

If the number of correction rounds is too high, even a successful result may indicate insufficient stability or poorly designed tasks.

3.2 Step Efficiency

Step efficiency measures whether the path the model takes to complete a task is concise. A mature execution agent should avoid:

• Repeatedly clicking the same button;
• Meaningless scrolling;
• Repeatedly opening the same menu;
• Waiting for a long time on an unchanged page;
• Skipping necessary verification steps.

Step efficiency affects not only execution speed but also cost and stability. More steps usually mean more failure points and higher reasoning consumption.

3.3 Exception Recovery

In real software environments, exceptions are normal. A strong system should have recovery capabilities, such as:

• Reloading the page when loading fails;
• Closing popups and continuing the original workflow;
• Correcting input when validation fails;
• Pausing and requesting human assistance when login status becomes invalid;
• Switching to an alternative path when a file path does not exist.

Exception recovery reflects how well the model adapts to the environment and determines whether a system can move from experimental scenarios to production.

3.4 Explainability

Explainability refers to whether the system can clearly explain why each operation was performed. For enterprise applications, fully black-box execution is not advisable. Each key action should ideally have a log record, including:

• The current task goal;
• The page state;
• The reason the model generated the action;
• The execution result;
• The follow-up strategy.

Explainability not only helps with troubleshooting but also supports auditing and retrospective analysis.

3.5 Applicability Boundaries

Computer use capability is not suitable for all tasks. For tasks with a high degree of standardization, complete interfaces, and stable processes, traditional API automation or process automation may be more stable and cheaper. The advantages of computer use are mainly found in:

• Systems without open interfaces;
• Scenarios where interfaces change frequently but task goals are clear;
• Long-tail business processes;
• Human-machine collaborative tasks that span multiple applications;
• Temporary or exploratory automation needs.

Therefore, when evaluating its effectiveness, task type must be considered rather than simply asking whether the model can click a button.

4. Cost Evaluation: Beyond Explicit Consumption There Are Hidden Costs

The cost of computer use capability is more complex than that of traditional text-based conversational tasks because it usually involves multiple rounds of reasoning, interface perception, tool execution, environment maintenance, and risk control. Cost evaluation should consider both explicit and implicit costs.

4.1 Model Inference Cost

Computer use tasks often require multiple rounds of interaction. Each round may include:

• A description of the current interface;
• Historical operation records;
• The task goal;
• Exception information;
• Candidate next actions.

If every round inputs a full screenshot or a large amount of page text, inference consumption increases significantly. The more complex the task and the more rounds involved, the higher the cost.

To reduce costs, consider:

• Preferring structured text and reducing unnecessary image input;
• Summarizing page content and keeping only task-relevant areas;
• Caching repeated interfaces;
• Delegating deterministic steps to a rule engine;
• Using stronger models for complex reasoning tasks and lower-cost models for simple steps.

4.2 Runtime Environment Cost

Computer use capability usually requires an independent runtime environment. For security and controllability, the environment may include isolated desktops, containers, file storage, input injection, log collection, and monitoring modules. These components bring ongoing compute, storage, and network overhead.

If tasks are executed infrequently, keeping the environment always running can waste resources. Costs can be reduced through on-demand startup, task queues, and elastic scaling. For cloud deployments, it is possible to combine common capabilities such as container platforms, object storage, message queues, logging services, and monitoring alerts to improve resource utilization and operational efficiency.

4.3 Development and Maintenance Cost

Computer use capability is not as simple as connecting to a model API. Enterprises also need to invest in:

• Designing task templates;
• Configuring permission policies;
• Writing exception-handling rules;
• Establishing log auditing;
• Defining human fallback mechanisms;
• Continuously optimizing prompts and action strategies;
• Tracking interface changes and adjusting model behavior.

Once this capability enters production, it requires long-term operation. Without governance mechanisms, the system may easily suffer from incorrect operations, workflow interruptions, or rapidly increasing maintenance costs.

4.4 Risk Cost

Risk cost is the most easily overlooked part. Incorrect actions performed by a computer use system may cause data overwrites, erroneous submissions, permission violations, or leakage of sensitive information. For irreversible operations, the risk cost may be far higher than the labor cost saved.

Therefore, cost evaluation should include security auditing, permission control, rollback mechanisms, manual review, and circuit-breaker strategies in the overall budget. A seemingly cheap automation solution can ultimately become more expensive if it lacks risk control.

5. Security Governance: Computer Use Capability Must Assume Distrust by Default

Computer use capability inherently involves operating real systems, so security design must be built in from the beginning. It is not safe to assume that actions generated by the model are always correct, nor that interface feedback is always trustworthy. Security mechanisms should be established from the following perspectives.

5.1 Runtime Isolation

Task execution environments should be isolated from core production systems. Approaches include independent containers, virtual desktops, restricted accounts, and dedicated networks to limit what the model can access.

5.2 Least Privilege

The system should grant only the minimum permissions required to complete a task. For example:

• Read-only tasks should not be granted write permissions;
• Query tasks should not be allowed to delete data;
• Export tasks should not access unrelated directories;
• Test-environment tasks should not connect to production data.

Permission control should be as fine-grained as possible to prevent the model from bypassing restrictions through path traversal, directory switching, or command concatenation.

5.3 Secondary Confirmation for Sensitive Operations

High-risk actions should require manual confirmation or approval. Examples include submitting payments, deleting files, changing permissions, releasing versions, and overwriting data. These actions should not be completed by the model alone.

5.4 Full Log Auditing

Logs should record the task source, operation target, execution result, a summary of model inputs and outputs, and key screenshots. Sensitive data should be masked while preserving the necessary audit trail.

5.5 Circuit Breakers and Rollback

When continuous failures, abnormal clicks, timeout blocking, permission errors, or dangerous operations are detected, the system should automatically pause and enter a human-handling process. For operations that can be rolled back, recovery mechanisms should be designed to prevent errors from spreading.

6. Engineering Deployment: Separate the Control Plane from the Execution Plane

To deploy computer use capability in practice, it is recommended to use an architecture that separates the control plane from the execution plane.

6.1 Control Plane

The control plane is responsible for task orchestration, model invocation, policy decisions, and result verification. It includes:

• Task entry management;
• Task state machines;
• Model routing;
• Action whitelists;
• Exception policies;
• Audit records;
• Metrics monitoring.

The control plane determines how the system plans and acts as the brain of the overall workflow.

6.2 Execution Plane

The execution plane is responsible for operations in real environments. It includes:

• Starting isolated desktops or runtime environments;
• Receiving action instructions;
• Executing input and clicks;
• Collecting interface state;
• Uploading logs and screenshots;
• Returning execution results.

The execution plane determines how the system operates and should focus on isolation, stability, and observability.

6.3 Observability System

The observability system should include:

• Task success rate;
• Average execution duration;
• Average number of turns;
• Distribution of exception types;
• Human intervention rate;
• Cost consumption;
• Number of high-risk actions triggered.

Only with complete metrics can a system continuously assess whether its use should be expanded.

7. Best Practices: Start with Low-Risk Tasks and Expand Gradually

Computer use capability is best adopted through a gradual rollout strategy. It is not recommended to use it directly for complex core business processes from the beginning.

7.1 Prioritize Low-Risk Scenarios

In the early stage, it is preferable to choose scenarios such as:

• Internal back-office operations;
• Test-environment data organization;
• Report downloading and aggregation;
• Form filling;
• Cross-system information synchronization;
• Repetitive query tasks.

These scenarios are usually rollbackable, verifiable, and have controllable impact.

7.2 Standardize Task Inputs

Do not only provide the user with a vague goal. Design structured task templates, for example:

Goal: export a certain type of data
Input conditions: time range, filter fields, export format
Success criteria: file generated and record count greater than zero
Prohibited actions: deleting data, changing permissions, submitting approvals
Exception handling: switch to human after more than three failures

The clearer the task, the more stable the model behavior and the easier it is to control costs.

7.3 Set Up Human Fallbacks

At key points, human takeover capabilities should be preserved. Human fallback is not a failure; it is a common and reliable design in complex systems.

7.4 Continuously Optimize Strategies

Execution logs should be used to continuously optimize strategies, including:

• Which steps fail most often;
• Which interface elements are difficult to recognize reliably;
• Which exceptions should be intercepted by rules;
• Which tasks are suitable for models and which are better handled by traditional automation;
• Which inputs can be compressed to reduce inference costs.

8. Applicability Boundaries: Suitable for Long-Tail Interactions, Not Unconstrained Human Replacement

The value of computer use capability lies in lowering the barrier to cross-system operations. It allows models to complete some human-machine interaction tasks even when standard interfaces are unavailable. It is especially suitable for long-tail processes with clear steps, verifiable results, and controllable risks.

However, it is not suitable for unconstrained replacement of human labor, especially in the following scenarios, where caution is required:

• Financial transactions and irreversible operations;
• Permission changes and account management;
• Releases to core production systems;
• Batch operations involving privacy-sensitive data;
• CAPTCHA, strong identity verification, or legally compliant processes;
• Complex web pages that change frequently and lack stable rules.

For these scenarios, a safer approach is to combine human review, permission control, process automation, and rule engines rather than relying entirely on free model operation.

9. Conclusion: The Key to Computer Use Is Not Looking Human, but Being Controllable, Auditable, and Optimizable

Computer use capability represents an important direction for the development of intelligent agents. It pushes large language models from text generation to interaction with real systems and creates new possibilities for cross-application automation, long-tail task processing, and optimization of complex business processes.

From an engineering perspective, however, the true limit of its value is not whether a model can demonstrate completion of a task. It is whether the system can run stably in real environments, effectively control costs, ensure security, and continue iterating.

Therefore, when evaluating and adopting computer use capability, enterprises should focus on four areas:

1. Whether the task has a clear goal and verifiable results;
2. Whether the execution environment is isolated and permissions are minimized;
3. Whether costs are measurable and inference consumption can be optimized;
4. Whether exceptions are recoverable and operations are auditable.

Only when these capabilities are incorporated into a complete engineering system can computer use capability move from experimental demonstrations to reliable productivity.