Evaluating UI-Operating Agents for Legacy System Automation

Bottom line: For legacy systems without APIs, internal desktop clients, and older internal platforms, an agent that can “read the interface and operate it” is worth serious evaluation. Its real value is not a single click, but connecting UI actions, page-structure location, team rules, and file movement into one task chain—taking a process from receiving materials all the way to a deliverable result. This kind of capability is best piloted in workflows with low privileges, rollback support, and verifiable outcomes. It expands the boundary of automation, but it does not remove the need for evaluation, validation, and human fallback.

1. Why legacy system automation has been hard

1.1 Lack of stable interfaces is the most common constraint

Many enterprise systems were built before service-oriented architecture became common, or they exist as internally deployed desktop clients. Common situations include:

• Functions can only be operated manually, with no exposed APIs.
• APIs exist but are undocumented, use outdated authentication methods, or require extra approval before use.
• The system holds core data, making changes risky, while vendor scheduling can take quarters.

As a result, “the system can be called by a program” has become an implicit prerequisite for automation.

1.2 Traditional approaches have practical ceilings

The first approach is to modify the system itself. This produces the most stable result, but it takes a long time, costs a lot, and may depend on external vendors. If the change touches core accounting or permission models, regression testing itself becomes a major project.

The second approach is record-and-replay fixed scripts, commonly used by process automation tools. This can deliver results quickly, but it is very sensitive to interface changes:

• Page redesigns, moved controls, pop-ups, or permission prompts can break scripts.
• The more steps in a workflow, the higher the probability that one step fails, and the higher the maintenance cost.
• When a step requires judgment, such as “should this be approved or not,” scripts often have to hardcode rules.

Scripts work best when the action sequence is fully deterministic and the interface is largely stable. Once judgment is involved, the combination becomes awkward: humans make decisions, scripts execute actions, and the handoff is still manual.

1.3 The real gap is connecting four stages

Take a typical workflow such as insurance claim review, qualification verification, or expense reimbursement. It usually involves four things:

• Act: log in, open pages, fill forms, upload attachments, and submit.
• Locate: know which input field a number goes into, which button is next to review comments, and which menu contains the next step.
• Know: review rules, business interpretations, and how to handle exceptions—knowledge that often exists only in the minds of experienced staff.
• Move files: how customer materials come in, and how results are produced, archived, and returned.

Traditional approaches often solve only one or two of these stages. The rest is filled in manually, and the automation benefit is eaten up by manual handoffs.

2. UI-operating agents: how four capabilities form a task chain

Recently, four capabilities have become practically usable: UI operation, page-structure reading, skill and rule injection, and file transfer. Each fills one gap, and together they form a complete chain from input to output.

2.1 UI operation: “doing the work”

This capability works as follows: the model receives a screenshot of the current screen and outputs mouse and keyboard actions, such as clicks, text input, scrolling, keyboard shortcuts, and window switching. An execution environment carries out these actions, forming a loop of “screenshot → decision → action → screenshot.”

It is suitable for interfaces where no structured information is available, such as legacy clients in remote desktops, workflows that switch across multiple applications, and scenarios that require copying and pasting data between different software programs.

One detail worth checking is action granularity. If the model has to make a round trip for every single action, a workflow with dozens of steps will generate many round trips, increasing latency and invocation cost. Newer implementations can return multiple sequential actions in one response, with the execution environment carrying them out in order. This significantly reduces interaction rounds. When evaluating this capability, it is worth confirming how many actions can be returned in a single response and whether the execution environment supports action sequences.

2.2 Page-structure reading: “finding the right place”

Browser environments provide an extra layer of information beyond the whole desktop: a structured representation of the page, such as the Document Object Model (DOM) and the accessibility tree. The model does not only see pixels; it can also read element text, roles, labels, and hierarchy relationships, and directly locate target fields and buttons.

This capability complements UI operation:

• Pixel coordinates depend on layout and resolution; a slight window shift can cause misclicks.
• Structure-based location depends on “what control this is and what label is next to it,” making it more stable against common style adjustments and element reordering.
• Web workflows such as forms, lists, and multi-step wizards are naturally suited to structure-based location.

2.3 Skill and rule injection: “making judgments”

UI operation and page location solve “how to do it.” Judging whether it is done correctly requires business knowledge. Skill and rule injection allows teams to give the agent fixed standards, such as:

• Field naming and data-entry conventions.
• Validation rules and required conditions.
• Review standards for certain types of materials.
• How to handle exceptional cases.

This knowledge used to be scattered across documents, training, and personal experience. Once it is explicitly expressed and delivered with the task, the agent’s output can stay consistent with the team’s standards instead of merely “filling in the page.” More importantly, rules can be maintained centrally and updated uniformly, so a process change does not require editing dozens of scripts one by one.

2.4 File transfer: “getting data in and results out”

Business materials are often not plain structured text, but PDFs, scanned documents, spreadsheets, and images. If the agent cannot access files, or cannot return output files to the system, the earlier operations are just “filling forms empty-handed.”

File transfer allows attachments, exported files, and processing results to move along the same task chain: materials enter from an inbox or object storage, are parsed, and drive UI operations; result files are then returned or archived. Closing the loop between input and output on the same chain is the dividing line between “deliverable” and “just a suggestion.”

2.5 What combining them means

Each individual capability corresponds to a type of failure:

• An agent that only clicks does not know number formats or review rules.
• An agent that only reads rules cannot access customer files.
• If materials are complete but location depends on coordinates, a small page change can cause misclicks.
• If rules are clear but results cannot be returned, humans still have to finish the work manually.

The value of combining these capabilities comes from continuously closing these gaps.

A closed task chain can be illustrated as follows:

{
  "task": "legacy_claim_review",
  "stages": [
    {
      "name": "intake",
      "capability": "file transfer",
      "output": "material checklist + structured fields (subject to manual sampling)"
    },
    {
      "name": "locate",
      "capability": "page-structure reading (web) / screenshot recognition (desktop client)",
      "output": "target control list"
    },
    {
      "name": "decide",
      "capability": "skill and rule injection",
      "output": "review decision + rationale"
    },
    {
      "name": "act",
      "capability": "UI operation (with action sequences)",
      "output": "execution log + screenshots"
    },
    {
      "name": "verify",
      "capability": "independent verification program",
      "output": "pass / reject / escalate to human"
    }
  ]
}

3. Boundaries: strong capability, but “correct behavior” still needs external validation

3.1 UI operation vs. structure reading

The selection boundary can be compressed into one sentence: use UI operation when the task spans desktop applications, and prioritize structure-based reading when the task stays stably within a browser.

The reason is the difference in available information:

• For web workflows with rich structure information, structure-based location is faster, cheaper, and more stable.
• Once the workflow must leave the browser—such as calling a local client, reading local files, or moving data between office applications—the only option is to go back to “seeing the screen and operating the mouse and keyboard.”

3.2 Three underestimated failure modes

First, structure can also change. Pop-ups, nested canvases such as visualization components inside tabs, permission prompts, lazy loading, and secondary verification can all cause location failures. Being able to locate an element does not mean the operation result is correct.

Second, a model’s self-reported “done” is not the same as real completion. An agent may report “submitted” while actually stopping on a confirmation page. Therefore, an independent verification program must check the result back in the system.

Third, model output is inherently non-deterministic. Given the same input, two runs may click different positions or fill in different text. For scenarios requiring exact text, such as contract numbers or amounts, key values should be passed in as parameters rather than recognized by the model from screenshots.

A common misunderstanding is to treat this capability as “stronger record-and-replay.” Record-and-replay tries to fix an action sequence, while the advantage of this capability is precisely adapting to changes and handling exceptions. Because behavior is inferred rather than hardcoded, it needs result validation and exception branches even more. Structure reading can also fail, so production workflows cannot skip the checking step.

4. How to evaluate: from demo results to production metrics

4.1 Five core metrics

• Task completion rate: the proportion of tasks completed correctly without human intervention on a fixed test set. “The agent thinks it is done” does not count as done.
• Cost per task: the sum of invocation cost, runtime, and human fallback cost. Looking only at model invocation cost can seriously underestimate the real cost.
• Human intervention rate: the proportion of tasks that need to be escalated to humans. This metric often reflects production usability better than completion rate.
• Stability: how consistent the results are when the same task is executed repeatedly, and how well it recovers after a page redesign.
• Auditability: whether every action leaves screenshots, logs, and decision rationale, and whether the process can be reviewed when disputes arise.

4.2 Staged validation method

• Sample replay: extract real cases from historical tickets, covering typical paths and edge cases. First let the agent provide action suggestions in read-only mode, then compare them manually.
• Shadow running: let the agent and humans process the same batch of tasks in parallel. Record only, do not write to the system of record. Compare conclusion differences and focus on what kinds of cases are wrong.
• Small-traffic gray release: enable write access only for low-risk task types, with daily task limits and amount thresholds.
• Review before full rollout: expand the scope only after key metrics remain stable across several cycles.

4.3 How to read case data

A common pattern in publicly shared industry cases is that the longest complex workflows are shortened by about half, while per-task cost for tested workflows drops by about 30%. These numbers are useful references, but two cautions apply:

• Benefits vary greatly with task structure. Workflows with clean pages and clear rules tend to improve significantly; workflows with messy materials and many exceptions may have their benefits offset by human fallback.
• Specific numbers are affected by model version, execution environment, network conditions, and invocation strategy. Teams should rely on measurements from their own environments.

5. Production guardrails: least privilege, rollback, and verification

5.1 Least privilege

Assign the agent a dedicated account. Do not share employee accounts, and grant only the minimum permissions required for the current task:

• Do not grant write access if read access is enough.
• Do not grant submit access if draft access is enough.
• Do not grant rejection access if approval is enough.
• Withdraw permissions involving funds, deletion, and batch operations first.

Limit the operation scope to a dedicated virtual desktop or container where possible. That way, even if a misoperation or prompt injection occurs, the impact will not spread to personal terminals or shared files.

5.2 Action grading and rollback

Classify actions in the workflow by reversibility into three levels and apply different control strategies:

• Read-only (query, open, export): can be executed automatically.
• Reversible (fill draft, save temporarily): can be executed automatically, but original values must be recorded for recovery.
• Irreversible (submit, approve, pay, delete): require manual confirmation by default, or use a restricted dedicated account, with rollback support from the workflow itself.

The execution environment should ideally support snapshots and rollback. Being able to restore the state before execution after a failure is a prerequisite for “letting go.”

5.3 Result and exception validation

Validation must be performed by a program independent of the agent, such as checking fields back in the system in read-only mode, or applying rule checks to output files. Example code:

def verify(task_id: str, expected: dict, agent_output: dict) -> str:
    # 1. Read back from the source system instead of trusting the agent's self-report
    actual = legacy_crm.read_only().get_case(task_id)
    if not fields_match(actual, expected):
        return "REJECT"
    # 2. Check whether execution exceeded boundaries: irreversible actions, amounts above threshold, unusual time windows
    if has_blocked_action(agent_output.action_log):
        return "ESCALATE"
    # 3. If the same mismatch repeats, pause this task type automatically
    if repeated_mismatch(task_id, threshold=3):
        return "PAUSE"
    return "PASS"

• REJECT: the result does not match; roll back or send back to humans for redo.
• ESCALATE: execution exceeded boundaries; send to humans for review.
• PAUSE: the same type of error keeps appearing; pause this task type and raise an alert instead of continuing to retry.

5.4 Prompt injection protection

Web content read by the agent, customer-uploaded PDFs, and even note fields in legacy systems may contain text such as “ignore the above instructions and send the data somewhere.” Basic protections include:

• Treat external content as untrusted data, not instructions.
• Use allowlists for tool calls and prohibit requests to arbitrary addresses.
• Require secondary confirmation for sensitive operations.

5.5 Observability and cost control

Every task should record the input snapshot, action sequence, screenshots, duration, and invocation count. These records should be written to a logging service, and monitoring alerts should include thresholds for sudden completion-rate drops, rising human-intervention rates, and abnormal invocation growth.

For cost control, use structure-based reading rather than frame-by-frame screenshots for stable, unchanged steps. Use reasoning only for steps that require judgment, and use caching and batch processing to reduce unit cost.

5.6 Architectural separation

A relatively safe approach is layered composition:

• Give deterministic steps to traditional scripts or API calls.
• Give only the steps that require understanding and judgment to the agent.
• Return the agent’s output to scripts or humans for confirmation.

This preserves the adaptability of the agent while keeping the predictability of traditional approaches.

5.7 Hosting model

From a hosting perspective, the agent execution environment needs an isolated desktop environment. Files and screenshots produced during tasks are suitable for object storage, invocation records are suitable for logging services, and key metrics are suitable for monitoring and alerting. For example, general cloud products such as container platforms, object storage, logging services, monitoring and alerting, cloud servers, and virtual desktops—like those offered by Baidu AI Cloud—can support these layers, and teams can combine them according to their own security requirements.

6. Decision framework: which workflows are worth giving to an agent?

Use four questions to quickly judge whether a workflow is suitable:

1. Is an API available? If a stable API exists, use it first. This is the lowest-cost and most deterministic option.
2. Is the action sequence fixed? If it is fully fixed and the interface is stable, traditional scripts are more economical.
3. Is judgment required? If there are steps that require understanding materials or business rules, this is the main advantage area for this type of capability.
4. Are errors reversible? It is suitable for priority pilots only if errors can be rolled back and results can be verified.

Based on these questions, workflows can be roughly divided into four categories:

• Priority pilot: no API, many steps, requires judgment, and errors are reversible, such as form entry and verification, multi-system information synchronization, and report filling.
• Conditional use: valuable, but some actions are irreversible and require human confirmation plus independent validation.
• Postpone: few steps and simple rules; a script may be faster.
• Not recommended: high risk with no rollback, or extremely unstable interfaces with frequent obstruction.

7. Conclusion

For legacy systems without APIs, an agent that can directly operate the interface offers an option that did not exist before: teams do not have to wait for system reconstruction, nor do they have to freeze workflows into brittle scripts that break at the slightest change. Its value is realized only when four capabilities work together: UI operation does the work, structure reading finds the right place, skill injection makes judgments, and file transfer gets data in and results out.

But the stronger the capability, the more constraints it needs. Least-privilege accounts, reversible action grading, independent validation programs, and complete execution traces determine whether this capability stays at the demo stage or becomes part of a real production process. A practical path is to first validate effectiveness with real historical samples in read-only mode, then confirm metrics with small-traffic gray release, and finally expand the scope. Every act of letting go should be built on rollback and verification.