Do Desktop Agents Really Work? A Reusable Evaluation Methodology

1. From “Chatting” to “Delivering”: The Evaluation Challenge of a Paradigm Shift

Over the past two years, the first-generation interaction paradigm centered on conversational Q&A has largely run its course. The industry’s collective shift is clear: a model’s value is no longer measured by “how well it answers,” but by “what it delivers”—whether that’s a meeting-ready report, an accurately scoped statistical table, or a successfully executed data processing workflow. Even programming scenarios are converging in this direction; developers no longer want mere code snippet suggestions, but autonomous entities that can understand tasks, orchestrate tools, and produce usable artifacts.

Consequently, desktop agents have rapidly emerged, widely supporting multi-device synergy and remote collaboration, with a highly consistent marketing pitch: “It actually does the work for you.” However, “doing work” is a highly subjective judgment. For the same tool, one person might save half a day, while another might end up reworking late into the night. To move selection decisions beyond personal gut feelings, we need an objective, reproducible, and quantifiable evaluation method. The purpose of this article is to abstract the experience gained from testing multiple products into an evaluation framework that any team can directly reuse.

2. Four Iron Rules of Evaluation Design

1. Same task, same materials, same delivery standards. The only variable source should be the object under test: prompts must be verbatim identical, materials completely the same, and acceptance criteria strictly defined beforehand. No post-test adjustments to the scale are allowed.
2. Single execution opportunity. No mid-task prompt supplementation, no human intervention, no “try again.” Iteratively approaching a goal through multiple rounds is an Agent’s strength, but in real work, users lack the patience to guide it. The first-pass success rate represents the true baseline of daily usability.
3. Three-dimensional separation of quality, efficiency, and cost. Quality is the “1,” and efficiency and cost are the “0s” that follow: if the artifact is unusable, being faster and cheaper is meaningless.
4. Postpone cost calculation to the end. Single-task cost is simultaneously affected by subscription models, API billing, and retry rounds. Calculating it midway can easily bias the judgment of quality and efficiency; it should be uniformly converted and compared at the very end.

Why emphasize “single execution”? Because Agent-class products excel at gradually approaching goals within human-machine feedback loops, but what an evaluation needs to simulate is the real-world scenario where “the user issues an instruction only once.” By turning off all intervention opportunities, what is exposed is the product’s engineering maturity.

3. Task Design: Using “Composite Tasks” as a Touchstone

Single-capability tests (translating a paragraph, summarizing an article) cannot distinguish an Agent from a chat assistant. An effective touchstone must cover the complete work chain: Web Search -> Data Understanding -> Code/Tool Execution -> Multi-Artifact Generation. A practically tested task template is as follows:

Search the web for new energy vehicle sales in China in June 2026.
Produce a market research report (.docx, under 1500 words)
and a sales statistics spreadsheet (.xlsx, must include charts).
Requirements: Concise, explicitly state data scope and sources, no irrelevant analysis.

Design intent breakdown:

• Word count limit: Filters out watered-down outputs that “pad word counts to fake workload,” forcing the model to compress information.
• Sales statistics: Tests sensitivity to statistical scopes, which is the most common pitfall (expanded in Section 5).
• Chart requirements: Tests the complete chain of sandbox code execution and artifact rendering.
• Dual-file delivery: Tests multi-artifact orchestration and file organization capabilities.
• Explicit scope and sources: Front-loads acceptance criteria into the prompt to observe whether the Agent proactively resolves ambiguities.

4. Environment Control: Fairness Details Neglected by Many

• Time window alignment: Web search tasks are affected by source updates. All subjects should execute within the same time window, otherwise data timeliness differences will pollute quality conclusions.
• Network and account consistency: Use the same network exit, same account tier, and version to avoid misinterpreting rate limits or quotas as capability differences.
• Localized materials: Save materials to disk in advance with unified paths to eliminate interference from download fluctuations.

These details may seem trivial, but they are prerequisites for “reproducible conclusions”—the rigor of the evaluation methodology itself determines the credibility of the evaluation results.

5. Quality Assurance: Three High-Frequency Failure Points

Failure Point 1: Statistical Scope Ambiguity. Taking vehicle sales as an example, the industry has at least two common scopes: wholesale (total shipments from manufacturers to dealers, including domestic and export) and retail (volume sold by dealers to end consumers). When the prompt only says “sales,” different Agents may interpret it as wholesale, retail, or a mix. If the artifact does not explicitly declare the scope, the final numbers may differ significantly—this is not a calculation error, but an unresolved ambiguity at the requirement level. A qualified delivery must contain three elements: Scope declaration (wholesale/retail), boundary declaration (whether exports are included), and source with data cutoff time. Missing any during acceptance results in a deduction.

Failure Point 2: Granularity of Detailed Data. Aggregate numbers are easy to get right, but detailed data (by manufacturer, by model) is the real trial-and-error zone. Common failure modes include mixing data from different sources, conflating monthly values with cumulative totals, and misattributing popular model rankings. It is recommended to set up a “spot check” item in the acceptance checklist: randomly select 3 to 5 sub-items and trace them back to original sources for verification. If any item is inaccurate, a quality gate is triggered.

Failure Point 3: Chart Rendering and Cross-Environment Compatibility. “File generated” does not mean “deliverable.” A typical failure is the silent loss of charts in spreadsheet artifacts: most Agents generate charts via scripts in a sandbox and embed them into the spreadsheet file. Scripts are written for the sandbox environment by default, but when executed in the user’s local desktop environment, they may silently fail due to differences in font paths, dependency library versions, or graphics rendering configurations—the process throws no errors, the file opens, but the charts are gone. The countermeasure is to build a habit of “item-by-item checking”: don’t just verify if the file exists, but confirm page by page that charts, tables, and formatting are intact.

6. Efficiency Evaluation: Redefining “Fast”

Completion time differences for composite tasks will diverge in the first round, but one must be wary of a trap: finishing first does not mean winning. The rework cost of a fast but inaccurately scoped artifact far exceeds that of a product that takes ten minutes longer but gets it right the first time. Therefore, it is recommended to replace “first output time” with “first deliverable time” (the time from task assignment to producing a rework-free artifact) as the efficiency metric. Intermediate processes (writing commands, calling tools, back-and-forth planning) are worth recording but should not be scored separately—as long as it falls within the first deliverable time, it is a reasonable execution cost.

7. Cost Accounting: A Three-Layer Cost Model

During the final unified calculation, split the cost into three layers:

1. Explicit costs: Subscription fees or token consumption conversions for a single task.
2. Retry costs: Extra rounds and compute consumed due to rework caused by substandard quality.
3. Manual correction costs: The invested manual time converted to costs for fixing detailed data, supplementing charts, and adjusting formats.

The third layer is the easiest to ignore, yet it best explains why seemingly cheap solutions lose their edge in real-world usage. This is the economic rationale for ranking quality weight first: if the quality gate is not passed, comparing cost and efficiency is meaningless.

8. Directly Reusable Acceptance Checklist and Scoring Template

task: Web search + document + spreadsheet composite task
constraints:
  execution: one_shot        # Single execution
  prompt_identical: true     # Verbatim identical prompt
  material_identical: true   # Identical materials
  time_window_aligned: true  # Same time window
acceptance:
  report:
    - word_count_within_limit  # Word count meets standard
    - data_scope_declared      # Scope explicitly marked
    - source_cited             # Source traceable
    - structure_clear          # Complete structure, no padding
  spreadsheet:
    - cells_spot_check         # Detailed data spot check
    - charts_rendered          # Charts present item by item
    - format_consistent        # Consistent layout
metrics:
  quality_weight: 0.6      # Acceptance item pass rate
  efficiency_weight: 0.25 # First deliverable time
  cost_weight: 0.15       # Three-layer cost conversion

Scoring logic illustration:

def score(result):
    quality = acceptance_pass_rate(result)   # Acceptance checklist pass rate
    if quality < 0.8:                        # Quality gate
        return {"verdict": "fail", "reason": "quality_gate"}
    efficiency = normalize(result.ttc)       # First deliverable time
    cost = total_cost(result)                # Explicit + retry + manual
    return 0.6 * quality + 0.25 * (1 - efficiency) + 0.15 * (1 - cost)

Implementation note: The quality gate threshold (0.8 in the example) should be set according to business fault tolerance—for data-driven decision scenarios, it is recommended to raise it above 0.9; for draft generation scenarios, it can be appropriately relaxed.

9. Conclusion: Establish a Baseline Before Discussing Selection

The general capabilities of foundation models determine an Agent’s upper limit. Currently, mainstream general large model services (such as the model capabilities provided by Baidu AI Cloud) have overall reached a usable water level in basic tasks like retrieval, code execution, and document generation. What truly widens the gap is the engineering layer: whether task understanding can proactively resolve scope ambiguities, whether the execution chain can render stably in heterogeneous environments, and whether artifacts can meet “directly deliverable” standards. For individuals and teams, rather than agonizing over which one to choose, it is better to first solidify your own benchmark task sets, acceptance checklists, and scoring weights—this is closer to your real work than any external evaluation. When the next wave of new products arrives, run the benchmark once, and the answer will naturally emerge.