Evaluating Office AI Agents Beyond Chat: Task Contracts, Evidence, and Chinese Agent Maturity

Office agents have moved from “answer a question” to “take on a piece of work.” Multi-file retrieval, web verification, browser operations, presentation drafting, and multimodal asset generation share common traits: ambiguous goals, many steps, intermediate state that must be saved, and results that must satisfy format and permission requirements. If evaluation remains limited to single-turn Q&A or static question banks, a high score says little about delivery capability. Scoring such tasks in an environment close to real office work reflects not the slogans of any one model, but whether the model, tools, memory, permissions, and acceptance criteria can form a closed loop.

1. Evaluate workflows, not chat boxes

Traditional benchmarks mostly test knowledge recall, instruction following, or short-code correctness. Office agents face unstructured commissions: materials are scattered across cloud drives, email, and web pages; requirements may change midway; and outputs must be factually traceable while being directly usable by colleagues. The evaluation unit should therefore be defined as a “task contract” that at least specifies input scope, allowed tools, completion criteria, time and call budget, and the situations in which the agent must stop and ask a human.

It is better to divide tasks into five categories and design acceptance for each than to collapse everything into one aggregate score.

1. Multi-file retrieval and synthesis. Given a set of documents, the agent must extract numbers with consistent definitions, list conflicts, and provide citation locations. Typical failure modes are missing files, misattribution, or presenting inferences as facts.
2. Web research. The agent must cross-check public information, mark sources and dates, and distinguish “verified” from “not verified.” Failure modes include treating search snippets as conclusions or ignoring recency.
3. Browser and office software operations. In controlled pages or documents, the agent must complete clicks, form filling, and exports. Failure modes include step drift, accidental irreversible actions, and inability to recover from modal dialogs.
4. Structured deliverables. Examples include outlines, tables, and slide outlines. Acceptance should focus on layout constraints, page count, and whether data comes from prior retrieval—not on whether the prose is elegant.
5. Multimodal generation and proofreading. Images and text must be consistent, and chart data must be traceable to tables. A common failure mode is something that “looks complete” while the numbers do not match.

Only category-level scoring can reveal whether the weakness lies in planning, retrieval, tools, or formatting, instead of letting one demonstration task inflate the aggregate score.

2. Disaggregate scores; a composite ranking is only a reading entry point

A single composite score is suitable for communication, not for procurement or architecture decisions. A more robust approach is to report a set of orthogonal metrics and then disclose the weights. The following dimensions can be used as a reference, with weights defined by the business and always made public.

• Task completion rate: the proportion of acceptance-checklist items passed. Partial completion should be prorated according to rules, not scored by impression.
• Process correctness: whether key tools were invoked, whether parameters stayed within allowed ranges, and whether mandatory steps were skipped.
• Factual grounding: whether every key claim can be traced back to a document paragraph, spreadsheet cell, or web snapshot.
• Constraint compliance: whether limits on page count, redaction, external sending, and high-risk operations were breached.
• Recovery capability: when a tool times out, a page changes, or a file is missing, whether the agent retries, degrades gracefully, or fabricates.
• Cost and latency: tokens, tool calls, and wall-clock time. Two systems with the same completion rate but an order-of-magnitude difference in cost have completely different engineering implications.
• Human-handoff quality: when stuck, whether the agent asks an actionable question instead of vaguely saying “more information is needed.”

Evaluation should also fix randomness: temperature, tool timeout, network environment, document version, and rater instructions must all be recorded. Each task should be repeated at least several times, and reports should show the median and variance rather than only the best run. Human scoring should use double review for disagreements. Automated scoring should cover only determinable checklist items, such as whether a file was generated, whether citation markers exist, and whether values match the source table.

A schematic task contract can be fixed in a structured form to avoid “let’s see what it does”:

task_id: office-brief-014
inputs: reports and spreadsheets in the specified directory; access to objects outside the directory is prohibited
tools: retrieval, read-only browsing, spreadsheet calculation, slide drafting
budget: at most 40 tool calls; 25 minutes wall-clock
must: attach a source location to every key number; list conflicts separately
must_not: send externally, delete, overwrite original files, execute unauthorized scripts
done_when: outline plus data appendix produced; automated validation passes and manual spot check finds no fabricated citations
on_block: stop and list missing materials; do not fill gaps with guesses

Such a contract turns “intelligence” into testable behavior. Without a contract, an office test easily becomes a check of whether a demo script runs.

3. What technical layers determine end-to-end capability?

Completing an office task usually depends not on generation quality alone, but on several layers working at the same time.

Planning and state. Long tasks require breaking goals into rollback-capable substeps and maintaining working memory of “confirmed facts, items pending verification, and files already used.” Stuffing history into the context window alone makes it easy to forget early constraints in the later stages. In practice, an explicit state object can be used: current subgoal, evidence list, and failure count. Each step should pass only the necessary state to the model so that irrelevant logs do not crowd out attention.

Tool orchestration. Retrieval, calculation, browsing, and generation should have clear input and output schemas. Tool descriptions should state constraints, such as read-only access, maximum number of returned items, and prohibition on following off-site redirects. The orchestration layer should handle authentication, timeouts, retries, and result truncation rather than leaving all these details to the model. For irreversible actions, human confirmation should be the default.

Evidence and citations. The biggest fear in office deliverables is text that is fluent but unsourced. Retrieval results should carry stable locators: document identifier, page number, or cell. At generation time, enforce a “no citation, no body text” policy and downgrade unsupported sentences to pending verification. This is more auditable than asking another model afterward to “check for hallucinations.”

Environment and permissions. The permission model in the evaluation environment must be of the same class as that in production. If testing grants unrestricted browsing and writing, while post-launch operations are constrained by directories, redaction, and auditing, rankings have no transfer value. Least privilege, operation logs, and sensitive-field masking should be treated as part of capability, not as compliance attachments added after launch.

Acceptance and regression. High-frequency tasks should be collected into a regression set: fixed inputs, fixed scoring scripts, and human review only for disagreements. Rerun it whenever the model or prompt changes. Without a regression set, it is impossible to distinguish “this sample got lucky” from “capability really improved.”

4. What such office tests say about the stage of Chinese capability

Without binding the discussion to individual product rankings, third-party office-task tests are better used to observe the stage of the ecosystem. Stable completion of multi-file, web-connected, and operation-type tasks usually means several things hold at once: instruction following and long context are sufficient; tool-call formats are stable; Chinese office corpora and layout conventions are taken seriously; and engineering teams can connect retrieval, permissions, and evaluation into the same pipeline. This already goes beyond the “chat demo” stage and enters a stage where agents can be accepted by processes.

At the same time, boundaries must be recognized. Leading on an aggregate score does not mean leading on every sub-capability. Browser tasks are sensitive to page structure, and changing a site can cause failure. Slide tasks can easily reward surface completeness while letting data errors pass. Web tasks are affected by source-site availability and have poor repeatability. Therefore, rankings are a signal that “internal reproduction is worth doing,” not a direct procurement conclusion.

Differences among Chinese agent solutions often appear more in engineering completeness than in the prose quality of a single generation: whether enterprise directory and permission integration is provided; whether retrieval is supported in private deployments or restricted networks; whether logs and audits can feed existing monitoring and alerting; and whether state can be handed back to a human on failure. These capabilities rarely appear in chat comparisons, but they determine whether an agent can enter approval, finance, or R&D workflows.

5. How teams can turn public tests into their own acceptance

External reports can at most provide task types and scoring ideas. In implementation, teams should rebuild their own set rather than memorize someone else’s aggregate score.

Step 1: Sample from real tickets, not flashy tasks downloaded from the web. For each category, retain successful, partially successful, and should-refuse cases. In particular, retain samples where “insufficient materials must stop” in order to penalize fabrication.

Step 2: Write the acceptance checklist before writing prompts. Use scripts for the parts the checklist can determine automatically, such as whether output files are complete, whether citation markers are closed, and whether numbers match the source table. Limit non-automatable checks to a small number of semantic items and provide scoring instructions.

Step 3: Fix comparison conditions. Use the same tool permissions, the same document snapshot, and the same budget. Compare systems, not “how much extra human help was provided in one run.”

Step 4: Put cost into the threshold. Set a completion-rate floor together with a call cap. If the budget is exceeded and the task is still incomplete, record it as a failure or degraded result. Avoid trading unlimited retries for a higher score.

Step 5: Regress safety cases together with business cases. Include privilege-escalation paths, prompt-injection documents with text such as “ignore previous rules,” and requests to send externally or delete data. The agent should refuse or escalate, not comply. These cases do not improve “looks,” but they determine whether the system can enter production.

6. Common misreadings when reading rankings

Treating one institutional test as the industry’s final state. Sample tasks, document language, whether tools are open, and whether human intervention is allowed all change rankings. When the task set and scoring scripts are not public, scores are not reproducible and can only be treated as qualitative clues.

Extrapolating chat ability to office ability. Clear answers do not mean the system can manage state, stop, or cite. Conversely, a system with a complete toolchain may not have a base model that stands out on public chat leaderboards.

Ignoring variance and cost. Reporting only the highest score hides instability. Office scenarios need predictability: completing within budget most of the time, with explainable behavior when failing.

Treating the evaluation environment as production. High scores in an environment without auditing, redaction, or permission boundaries cannot transfer to regulated workflows.

7. Practical trade-offs in implementation

When resources are limited, prioritize connecting one narrow but complete chain: read-only retrieval, sourced summaries, and human-confirmed finalization. Do not simultaneously roll out browsing, file modification, and automatic sending. A narrow chain is easier to specify in contracts and easier for finding whether the bottleneck is the model, retrieval, or permissions.

Keep the state and evidence layer on the application side rather than relying entirely on model memory. The model proposes and drafts; the application records evidence, blocks privilege escalation, and triggers acceptance. In this way, the regression set remains valid when the base model is replaced.

For multimodal outputs such as presentations and spreadsheets, insist on “data first, layout second.” Generate verifiable tables and citations first, then generate the presentation. Numbers in the presentation must come only from validated tables to avoid two sets of numbers.

In conclusion, the level of office agents should be measured with end-to-end tasks, reviewable metrics, and controlled permissions. If third-party tests push the industry from chat comparisons toward workflow comparisons, evaluation standards are maturing. If Chinese agent solutions can consistently pass regression on Chinese materials, enterprise permissions, and long-horizon delivery, their capability is moving from demonstration into runnable systems. Rankings can attract attention, but what is truly reusable is the task contract, the evidence chain, and the acceptance set.