Evaluating the Computer Operation Capabilities and Costs of Multimodal Agents

1. Evaluation Focus: Beyond Just “Can the Model Answer”

Multimodal agents can simultaneously process text, images, and interface states, completing operations through mouse and keyboard inputs. When evaluating such systems, relying solely on QA accuracy or single-turn reasoning capabilities is insufficient. Enterprises are more concerned with whether the agent can understand screen information, decompose goals into executable steps, recover from operational errors, and how much time and cost are required to complete an entire task.

Therefore, the evaluation target should expand from a single model to the complete execution chain. Input understanding, task planning, UI localization, action execution, result verification, and exception recovery—failure in any of these links can cause the entire task to fail. For tasks such as batch document organization, data entry, and page content verification, the end-to-end success rate is often more valuable than a single benchmark score.

2. Visual Understanding and Computer Operation: Judging Capabilities in Context

Visual capabilities determine whether an agent can extract valid information from pages, charts, images, or documents; computer operation capabilities determine whether it can translate this information into actual actions. The two are interdependent: if a page element is misidentified, subsequent clicks may land in the wrong place; if task planning is unclear, even if the page is understood correctly, unnecessary or dangerous operations might be executed.

Evaluations should cover different types of interfaces and workflows, such as extracting fields from tables, filling out web forms based on rules, searching for information across pages, and structuring data based on charts or diagrams. Task design should closely resemble real-world work rather than just testing isolated visual QA. It is also necessary to record whether tasks are fully completed, whether key fields are correct, whether unnecessary operations were triggered, and whether the system can identify and stop upon errors.

Real desktop environments are inherently uncertain: pages may load slowly, button positions may change, pop-ups may obscure content, and forms may display validation prompts. Therefore, static screenshot testing in a lab cannot replace real-environment testing. A more reliable approach is to gradually increase environmental complexity—first testing fixed pages, then dynamic pages and long workflows, and finally conducting small-scale pilots in controlled business environments.

3. Speed and Reasoning Intensity: Comparing End-to-End Efficiency

The speed at which an agent completes a task does not equal the model’s text generation speed. A single task may involve multiple rounds of observation, reasoning, action, and confirmation. A fast single-step response might still result in a long total duration due to excessive API calls. Conversely, reducing steps might save time but increase the risk of missed details and operational errors.

During evaluation, record end-to-end duration, operation steps, wait times, retry counts, and success rates simultaneously. Distinguish between model inference time and external page response time. For high-risk tasks, more comprehensive reasoning and result verification are preferable; for batch tasks with clear rules and recoverable errors, testing a more lightweight execution configuration is better. The key is not blindly pursuing the fastest speed, but finding a highly efficient overall solution that meets correctness requirements.

It is recommended to compare at least three types of metrics: first, the average duration of successful tasks; second, the actual duration including failures and retries; third, the number of valid tasks completed per unit of time. Only counting successful samples hides the rework costs of failures and easily overestimates the system’s production efficiency.

4. Cost Accounting: Focusing on Total Expenditure per Valid Result

Model API costs are only a fraction of automation costs. A complete accounting should also consider input/output processing, repeated calls, execution environments, log storage, manual review, and error rework. Especially in long workflows, if the system needs to frequently capture screens, re-understand context, or repeat actions, a low price per call does not necessarily mean a low overall cost.

A more useful comparison metric is the “cost per qualified result.” For example, you can track the total API costs, manual review time, and rework volume required to complete one hundred tasks, and compare them against the original manual process. For low-risk, stable, and high-volume scenarios, automation can yield significant economies of scale; for work requiring expert judgment, involving high error costs, or frequent process changes, manual review and maintenance costs may offset some benefits.

Enterprises should also calculate costs by task tier. Simple data manipulation, formatting, and repetitive entry have different complexity levels than tasks requiring cross-document reasoning or handling exceptions. Mixing them into a single average can lead to misleading cost conclusions.

5. Reliability and Hallucinations: Based on Verifiable Results

Agents might misread interfaces, omit conditions, or provide seemingly plausible judgments when evidence is insufficient. Mitigating these risks cannot rely solely on the model’s self-reported confidence; verification mechanisms must be integrated into the workflow design. For instance, check required fields and key inputs before submitting a form; require human confirmation before executing high-impact operations like deletions, payments, or publishing; and generate traceable operation logs for the results.

Testing should distinguish between “action completed” and “goal achieved.” Clicking a submit button does not mean the data was saved correctly; generating a file does not mean its content complies with business rules. You can define clear acceptance criteria for each task type—such as field consistency, file format, target page state, or key content coverage—verified through a combination of programmatic checks and manual spot checks.

For easily recoverable errors, design limited retry and rollback strategies; for situations where the state cannot be confirmed, halt automatic execution and escalate to human handlers. Defining failure boundaries is generally safer and more audit-friendly than letting the system retry indefinitely.

6. Deployment Recommendations: Start with Low-Risk, Measurable Workflows

Before deployment, map out process inputs, outputs, permissions, and exception branches. Confirm which steps can be fully automated and which must retain manual approval. Prioritize pilots with clear rules, high repetition, and verifiable results. Establish a set of test samples covering normal, boundary, and exception cases.

During the pilot phase, apply the principle of least privilege: only open the pages and operation scopes necessary to complete the task. Mask or isolate access to sensitive data. Save key actions, timestamps, and result statuses to facilitate issue tracking. After going live, continuously monitor success rates, rework rates, manual intervention ratios, and unit task costs. When UIs or business rules change, re-validate the workflow rather than assuming prior performance will persist.

7. Conclusion: Suitable for Execution-Oriented Work, Not All Work

The primary value of multimodal agents lies in combining visual understanding with desktop operations to handle large volumes of repetitive, verifiable digital tasks. Determining their worth should involve a comprehensive assessment of end-to-end success rates, completion times, total costs, error impact, and manual review requirements—not just isolated capabilities or benchmark scores.

For developers and enterprises, the most effective evaluation approach is to build small-scale, reproducible test workflows around real tasks, and then gradually expand the scope. Computer operation agents are ready to move from demo to actual production only when automation results are stable, failures are controllable, costs are measurable, and responsibility boundaries are clear.