How to Evaluate End-to-End Office Agent Capability

Evaluate “finishing a job,” not “answering a question”

A common mistake when adopting office agents is to reuse chatbot evaluation habits: give a passage, ask a question, and check whether the answer stays on topic. Those metrics can reflect language understanding and summarization, but they do not show whether the system can finish work under constraints. Real office work is closer to a delegated task. Materials are spread across files, facts need external checks, the process may require a browser or an internal system, and the deliverable is often a slide deck, minutes, a spreadsheet, or a reviewable research brief—not a chat reply.

Capability evaluation should therefore treat the agent as an execution system with tools, state, and acceptance criteria. The core question is not “can it talk?” It is “after a task is assigned, can it collect evidence within a limited number of steps, make tradeoffs, produce an acceptable result, and leave a diagnosable trace when it fails?”

Fix the task boundary before talking about scores

An aggregate score without boundaries is hard to interpret. Before evaluation starts, write down five kinds of constraints.

• Input scope: which files, knowledge bases, and external sources may be used; do not invent data that was not provided.
• Tool permissions: whether retrieval, a browser, document generation, spreadsheet calculation, and similar tools are available, plus timeouts and call limits.
• Definition of done: deliverable format, required sections, citation rules, and whether a person may correct the run mid-task.
• Definition of failure: skipped steps, citations that cannot be traced, unauthorized actions, infinite loops, or timeouts should count as incomplete or only partially complete.
• Environment consistency: the same file snapshot, network conditions, and prompt template, so environmental noise is not mistaken for model ability.

Rankings are meaningful only after these conditions are aligned. If one product can use the web, another cannot operate a browser, and a third allows a person to fill gaps, their scores should not be compared directly.

Five capabilities worth testing in office work

Multi-file retrieval and evidence consolidation

Office materials rarely live on a single page. Prepare a cross-format corpus: meeting notes, spreadsheets, email excerpts, and text from scans. The task should require a conclusion that only several files together can support, plus sources. Watch for missed key files, treating a similar but outdated version as current, and citations that cannot be located by file name and passage. An output that gives a conclusion without evidence locations should be penalized.

Web research and fact checking

When the task explicitly requires external verification, check whether search queries cover the constraints in the question, whether primary material is distinguished from secondhand accounts, and how conflicting information is handled. A passing result flags uncertain items instead of smoothing gaps with fluent prose. The evaluation set can deliberately include outdated figures or contradictory public statements, to see whether the system stops and explains the conflict rather than picking a number that “looks reasonable.”

Browser and business-system operation

Browser tasks measure operational reliability, not writing quality. Typical steps include opening a specified page, filtering by conditions, extracting fields, and filling and confirming a form. Record the click path, whether the agent enters unrelated pages, and whether it stops honestly at a captcha or a permission wall. For internal systems, prefer an isolated environment and masked data. Success means extracted fields are correct and no unauthorized action occurs—not that the agent “appears to have clicked around.”

Structured delivery: reports and presentations

Slides and long documents need their own scores. Fluency of the body text is not enough. Split the score into structure, facts, and editability: whether page order serves a decision, whether titles can be read on their own, whether charts match the tables, and whether key numbers can be traced. If the result cannot be restyled or exported in a standard format, its value in collaborative work drops sharply.

Multimodal content generation

When the task includes figures, tables, or screenshot explanations, check that text in the image matches the body, that axes and units are correct, and that screenshots correspond to real operations. A common multimodal failure is “the figure looks good, but the numbers were invented.” That error is more serious than awkward wording, because colleagues downstream may cite it directly.

Use process metrics to complement outcome metrics

Looking only at the final file misses a large amount of risk. Record the process as well.

• Planning: whether the agent lists subtasks and completion conditions before writing a long document.
• Tool calls: counts, failed retries, and repeated useless calls for retrieval, calculation, and page actions.
• State retention: whether a later step forgets a constraint already confirmed, such as a date range, currency, or statistical definition.
• Exception handling: when a file is missing, a page changes, or permission is insufficient, whether the agent stops and explains or continues to fabricate.
• Traceability: whether conclusions, citations, and operation logs can be aligned.

These metrics do not depend on any one model’s internal interface. Task run logs are enough. Keep an input summary, tool name, latency, error code, and a hash of the final artifact so runs can be regressed.

A sample case can stay product-neutral:

Task: Using three given sets of minutes and one expense sheet, check the
public industry-average definition on the web and produce a decision brief
of no more than 12 pages.
Done when:
1. Every key number cites a source (internal file, or external page title plus access date)
2. The expense-sheet total and the figures in the brief differ by 0
3. Items that cannot be verified are listed separately as "to be confirmed"
Fail when: citations are missing, the statistical definition is changed without saying so,
or the run times out without a deliverable

Score in layers so one total does not hide weaknesses

Split results into completion, correctness, efficiency, and safety. Break each layer into sub-items and use weights that reflect business preference. A team with strict compliance needs should raise the penalty for unauthorized actions and unsupported conclusions, rather than letting writing quality average those failures away.

Completion asks whether the required format was delivered. Correctness covers facts, calculations, and citations. Efficiency covers steps, latency, and repeated calls. Safety covers whether fields that should not be emitted were leaked, and whether write actions outside the prompt were executed. Partial completion may receive a score, but it must be reported separately from fully automatic completion. Otherwise, human gap-filling is counted as product capability.

Do not treat one impressive demo as stable ability. Run the same task multiple times and look at variance. Success sometimes and a missed file at other times means the workflow cannot be relied on. The regression set should gain cases as the business changes, rather than remaining a fixed set of public quiz questions.

Engineering practice for building the evaluation set

Mask the corpus: remove real customer names, account identifiers, and internal links, and keep structure with placeholders. Freeze file versions so the “gold answer” does not drift when a shared drive is updated. Write a reference answer or scoring rubric for each case, and have someone who did not write the item review it, to reduce author bias.

Layer difficulty: single-file summary, cross-file checking, external retrieval, browser operation, and generation of an editable deliverable. Layering shows whether the bottleneck is retrieval, planning, or generation. Long tasks can use checkpoints—for example, an evidence table before the brief—so the failing stage can be located.

Human raters need calibration. Two people score the same output against the rubric. Items with large disagreement go back to the rubric rather than being forced into agreement by feel. Prefer scripted comparison for calculations, to reduce subjective points. Language quality can remain in the rubric, but its weight should not outweigh facts and process.

Put evaluation into the team workflow

Use results for selection and release thresholds, not for external promotion. Before release, set a minimum completion rate, a citation-coverage rate, and a safety line of zero unauthorized actions. After release, feed failure samples from real tickets back into the evaluation set. Follow least privilege: by default, read only a specified directory; writes and external sending need an extra confirmation. When logs and deliverables go into object storage or a document library, set a retention period. Alerts should watch abnormal retries, timeouts, and leakage of sensitive fields, not call volume alone.

Running an isolated browser on a cloud container platform, dispatching evaluation jobs through a message queue, and aggregating traces in a logging service are ordinary engineering methods. What matters is a reproducible environment, masked data, and traceable conclusions. A single cross-product total should not be read as overall technical level. More informative is whether, under the same constraints, end-to-end completion rate, evidence quality, and failure modes improve in a stable way.

Any gap or lead between domestic and overseas options should show up in reproducible experiments, not in slogans. Treat the office agent as software that must be accepted: define the task, freeze the environment, record the process, score in layers, and regress continuously. Only then can a team judge whether it can actually carry a job from start to finish.