Computer-Using Agents: How to Evaluate Their Intelligence Beyond Leaderboards

    This article examines how to assess the intelligence of computer-using agents that operate web pages, desktop applications, and business systems. It explains the four capability layers, clarifies what public benchmark leadership does and does not prove, proposes a six-dimensional evaluation framework, and discusses production architecture, selection metrics, and scenario-specific validation. The conclusion is that leadership on computer-use tasks can be evaluated, but claims of “highest intelligence” require private benchmarks, engineering metrics, and safety governance.

    1. From Answering to Acting: Why Computer-Use Capability Is a Dividing Line

    Evaluating whether a general agent is “smarter” used to rely on text Q&A, code generation, or knowledge retrieval. Once a model starts operating web pages, desktop applications, and business systems directly, however, the object of evaluation changes. The model is no longer only generating content; it must complete a loop of perception, planning, execution, and verification in a real environment. Computer-use capability is often seen as a dividing line because it tests multimodal understanding, long-horizon task planning, tool invocation, state tracking, and error recovery at the same time.

    In terms of capability composition, a computer-using agent includes at least four layers:

    1. Interface perception: Identify buttons, icons, input fields, menus, tables, popups, and page states, and understand the correspondence between visual elements and semantic functions.
    2. Task planning: Break goals such as “organize a calendar,” “fill out an online form,” “update customer relationship records,” or “do web research and write a summary” into executable steps.
    3. Action execution: Within the allowed permissions, click, type, scroll, switch applications, invoke tools, or generate intermediate artifacts.
    4. Result verification: Check whether a form was submitted successfully, whether data was written correctly, whether a chart was generated, and whether a page passed front-end quality self-checks, and replan when failures occur.

    The biggest difference from pure text capability is that the environment is dynamic. Pages may load slowly, buttons may be obscured, forms may fail validation, and applications may pop up confirmation dialogs. The agent must maintain task state, identify anomalies, and avoid duplicate submissions or mistaken operations. Therefore, leading computer-use capability usually implies a strong foundation in visual grounding, cross-application long-horizon tasks, and tool orchestration, but it is still only one slice of intelligence.

    2. What Public Benchmark Leadership Does and Does Not Show

    If public materials show that a type of model leads in agentic composite tasks, desktop system interaction, or real-interface element grounding, the technical meaning deserves unpacking. Composite task benchmarks focus more on multi-step task completion; desktop interaction benchmarks focus more on cross-application operation and state management; real-interface element grounding benchmarks, when conducted without external tool assistance, emphasize the model’s ability to identify buttons, icons, input fields, and menus using vision alone.

    These results show at least three things: first, vision-language-action alignment is strong, and natural-language goals can be mapped to interface elements; second, step planning and execution stability in long-horizon tasks are relatively good; third, the chain of tool invocation and result verification is relatively complete. But claiming “highest intelligence” still requires caution. Public benchmarks have limitations in task distribution, benchmark contamination risk, differences in success criteria, differences in tool permissions, and differences in cost and latency. A model leading on operation tasks does not mean it leads comprehensively in open-world reasoning, scientific discovery, long-term memory, understanding of social norms, cross-domain transfer, and safety and compliance.

    A more rigorous statement is: on computer-use-related tasks, the model is in the leading tier; whether it represents “highest intelligence” depends on how intelligence is defined, which task set is used for measurement, under what cost and safety constraints the comparison is made, and whether independent reproduction and private-scenario validation have been performed.

    3. A Six-Dimensional Framework for Evaluating “Intelligence”

    A single leaderboard is suitable for initial screening, not for final conclusions. It is advisable to build an evaluation framework from the following six dimensions.

    1. Task Breadth and Depth
    Breadth covers scenarios such as form filling, customer record maintenance, calendar organization, web research, email and document summarization, scientific data analysis, chart generation, website creation, and front-end quality self-checks; depth covers multi-constraint, multi-hop, multi-tool long-horizon tasks. Only when both breadth and depth hold can the capability be shown not to be overfit to a few demo scenarios.

    2. Long-Horizon Planning and State Management
    Real tasks often contain dozens of steps. Evaluation should focus on whether the agent can remember intermediate results, identify dependencies, replan after failure, and avoid duplicate submissions, missing fields, or accidental data deletion. Average number of steps, number of replans, and task interruption rate are important metrics.

    3. Vision-Language-Action Alignment
    Interface element recognition requires not only knowing “where the button is” but also knowing “what this button means in the current task.” Evaluation should cover complex cases such as high-resolution pages, dense tables, dynamic popups, missing accessibility labels, and ambiguous icon semantics.

    4. Tool Invocation and Orchestration
    Computer-using agents often need to combine retrieval, spreadsheet processing, file generation, code execution, and business system operations. Evaluation should focus on whether tool selection is correct, whether parameters are valid, whether invocation order is reasonable, and whether the agent can degrade gracefully when a tool returns an exception.

    5. Reliability and Error Recovery
    Production environments value “stable completion” more than “occasional impressive results.” It is advisable to measure first-attempt success rate, final success rate, human takeover rate, error recovery rate, and replayability rate. An agent that can proactively detect anomalies and request human confirmation is often more reliable than one that retries blindly.

    6. Safety, Compliance, and Cost Efficiency
    Once an agent has operation permissions, safety boundaries must be designed in advance: least privilege, secondary confirmation for sensitive operations, sandbox isolation, audit logs, data masking, and operation replay are all essential. At the same time, end-to-end latency, per-task cost, and resource consumption should be evaluated to avoid a situation where “it can be completed but at an unacceptable cost.”

    4. Engineering Deployment: Bringing Computer-Using Agents into Production

    A production-grade agent should not be designed as a black box that directly controls real systems. A layered architecture is better: the perception layer reads interface state; the planning layer generates task steps; the execution layer operates in a sandbox or controlled environment; the verification layer checks results; and the audit layer records actions, screenshots, inputs and outputs, and decision basis. Below is generic pseudocode:

    1. def run_agent(task, env, policy, guard):
    2. state = env.observe()
    3. plan = policy.plan(task, state)
    4. for step in plan:
    5. action = policy.act(step, state)
    6. guard.check(task, action) # Permission, sensitive field, and risky action checks
    7. result = env.execute(action) # Execute in a controlled environment
    8. state = env.observe()
    9. if not policy.verify(task, state):
    10. plan = policy.replan(task, state, result)
    11. return policy.summarize(task, state)

    The following principles are recommended for deployment:

    • Least privilege: Grant only the account permissions required to complete the task; sensitive operations must require human confirmation.
    • Sandbox isolation: Replay tasks in an isolated environment before entering the production system.
    • Full-chain auditing: Record operation steps, page screenshots, input parameters, and verification results to support troubleshooting and replay.
    • Human takeover: Set takeover points for high-risk, low-confidence, or repeatedly failed tasks.
    • Metrics monitoring: Continuously collect success rate, takeover rate, latency, cost, and anomalous events.

    In a general cloud environment, for example, general services such as container platforms, logging, monitoring and alerting, and object storage—such as those offered by Baidu AI Cloud—can be combined to support isolated execution, audit trails, and metric monitoring. Such infrastructure usage should serve security and observability, not replace independent validation of the model’s capabilities.

    5. Selection and Validation: Look Beyond Leaderboards

    Actual selection should establish a private evaluation set covering high-frequency pages, anomalous popups, permission boundaries, and long-horizon tasks in the organization’s own business. Key metrics to collect include final task success rate, first-attempt success rate, recovery success rate, human takeover rate, average execution steps, end-to-end latency, per-task cost, safety violation rate, and replayability rate. Validation should also include ablation tests: remove visual input, remove tool invocation, limit retries, inject network latency and popup interference, and observe performance degradation.

    6. Conclusion: Leadership Can Be Evaluated, but “Highest” Requires Scenario-Specific Evidence

    Overall, agents with strong computer-use capability may indeed be in the leading tier on operation tasks, and multi-task leadership in public benchmarks also indicates competitiveness in visual grounding, long-horizon planning, and tool execution. But “highest intelligence” is a global judgment and cannot be drawn from a single capability domain or a few leaderboards. A more professional conclusion is: in scenarios such as web and desktop operation, form processing, data organization, document summarization, and front-end self-testing, these models deserve priority validation; in open-world reasoning, long-term memory, cross-domain transfer, safety and compliance, and cost efficiency, scenario-specific selection still requires private benchmarks and production monitoring. Only by combining leaderboards, engineering metrics, and safety governance can local task leadership avoid being mistaken for overall highest intelligence.