Computer-Use Agents for Legacy System Automation: Capabilities, Task Loop, and Deployment

I. Background: Long-Standing Pain Points in Legacy System Automation

Enterprise IT environments contain many long-lived business systems that can handle login, form entry, file uploads, and similar operations but do not expose stable, documented interfaces. These legacy back-office systems are difficult to automate for concrete reasons:

• No API: The system exposes only a graphical interface, so every interaction must go through the UI.
• Long change cycles: Directly modifying a legacy system requires cross-team coordination and regression testing, often measured in quarters.
• Fragile traditional RPA: Approaches that record fixed steps or rely on coordinates or selectors break as soon as the page structure or layout changes.

Advances in large model capabilities—especially visual understanding, long-context processing, and complex reasoning—have opened a new technical path for “letting a model operate software like a person.” So-called computer-use capabilities essentially let a model understand the current interface from screenshots or page structure, infer the next action (click, type, scroll, upload, and so on), and repeat this loop until the task is complete.

II. Four Components of a Closed Task Loop

Being able to “click” is not enough to complete real business work. A complete task chain must close four gaps in sequence; missing any one of them creates a failure point where the task stops.

1. Acting: Interface Operation Capability

The model must be able to generate and execute UI-level actions: move the cursor, click buttons, fill forms, select dropdown options, upload attachments, and so on. This requires the model to understand interface semantics from screenshots or DOM structure rather than relying on hard-coded coordinates.

Example action sequence (pseudocode):

{
  "task": "Enter customer data into legacy system",
  "actions": [
    {"type": "click", "target": "Login button"},
    {"type": "type", "field": "Username", "value": "agent_user"},
    {"type": "upload", "field": "Attachment", "file": "report.pdf"},
    {"type": "click", "target": "Submit"}
  ]
}

2. Grounding: Page Structure Awareness

Pixel-coordinate localization alone is highly unreliable—even a small page adjustment can cause an element to be clicked in the wrong place. A more robust approach combines the accessibility tree or DOM structure to read the element hierarchy of a web page or application, allowing the model to locate targets by semantics (for example, the “Submit for Review” button) rather than position. In this way, even if the page layout changes, the task can still be executed as long as the semantics remain unchanged.

3. Knowledge: Embedding Team Rules into Tasks

An agent does not know how numbering fields should be filled, which fields are mandatory, or what the review rules are. Such knowledge is usually scattered across team documents, operating manuals, and internal specifications. Through a pluggable skill/knowledge package mechanism—a common industry approach is to encapsulate instructions, script snippets, and documentation into reusable modules—relevant rules can be dynamically injected during task execution, so the model follows the rules rather than guessing.

4. Files: Input and Output Flow

Real business work always involves files: customer PDFs, Excel reports, scanned documents, and so on. An agent must be able to read input files, parse their contents, extract fields, and write output files back along the same chain. Supporting file-management capabilities, such as general-purpose file upload/download/hosting services, make “receive materials → process → write back results” a complete chain instead of ending with a text recommendation in the middle.

III. Why the Combined Value Exceeds Individual Capabilities

Each of these four capabilities corresponds to a failure point:

Missing component	Typical failure mode
UI operations only	Does not know numbering rules or review logic, so it fills out forms incorrectly
Knowledge documents only	Cannot obtain customer files and has no starting point
Files available but coordinate-based localization	Clicks in the wrong place when the page changes, causing task failure
Page structure reading only	Cannot execute actual actions and can only give recommendations

Only when combined can an agent start from receiving materials and proceed through reading, understanding, operating, and writing back results until it produces a deliverable. This also explains why “computer operation + browser perception + knowledge injection + file flow” should be evaluated as an integrated solution rather than as four independent features.

IV. Deployment Advice: Start with Low-Risk Processes

For developers and enterprises considering this approach, the following principles are recommended:

1. Start with least privilege: Give the agent an account with the minimum necessary permissions, exposing only the systems and fields required to complete its tasks.
2. Design for rollback: Prefer operations that can be undone or redone, such as filling drafts or saving temporarily, and avoid directly executing irreversible actions such as deletion or external payments.
3. Make results verifiable: Record screenshots, input values, and logs for every operation, and route outputs into human review. A confidence threshold can be set so the model escalates to a human when uncertain.
4. Expand scope gradually: First validate on high-frequency, low-risk processes such as queries, data entry, and organization. After accumulating a rule base and failure cases, expand to critical business processes.

Simple execution loop example:

while not task_done:
    screenshot = capture_screen()
    page_tree = get_accessibility_tree()
    context = load_knowledge(task_type)      # Inject team rules
    action = model.decide(screenshot, page_tree, context, files)
    if action.confidence < THRESHOLD:
        escalate_to_human()                  # Escalate to a human
        break
    execute(action)
    audit_log.record(action, screenshot)     # Record for verification

V. Conclusion

“Letting a model operate a computer” does not replace existing integration approaches. Instead, it offers a new option for the long-standing gap of legacy systems without APIs. Its value comes from closing the task chain: UI operations solve “acting,” structured perception solves “grounding,” knowledge and skill injection solve “rules,” and file capabilities solve “flow.” Once the technology is feasible, engineering discipline determines success—least privilege, rollback safety, and verifiable results. Only by putting these three principles into practice can agents move from demos to production and reliably handle real workloads such as legacy system automation.