LLM Agent Evaluation in Office and Finance Scenarios: Practical Observations

    This article examines how LLM agent evaluation has shifted from static Q&A benchmarks to task-based testing in realistic environments. Drawing on recent public evaluations in office automation, financial workflows, and coding, it analyzes agent performance, benchmark design choices, and reliability gaps such as overconfident answering. It also outlines practical steps for enterprises building task-specific evaluation sets and guardrails for production deployment.

    1. Background: From “Can It Chat?” to “Can It Do the Work?”

    Over the past two years, the focus of large language model (LLM) evaluation has shifted noticeably. Early benchmarks mainly tested language understanding, knowledge question answering, and reasoning, with accuracy as the primary metric. As the agent paradigm has gained traction, the more relevant question has become: can a model call tools, plan steps, and reliably complete an end-to-end task in a real work environment?

    This shift has brought profound changes to evaluation methodology. Static question-bank benchmarks struggle to reflect real productivity. They are increasingly replaced by dynamic evaluations built around real business environments: the model must operate in a simulated or real workspace, complete document processing, data organization, workflow approval, information retrieval, and other tasks through multi-turn interaction, and finally be scored on task completion quality.

    Based on recent public evaluation observations, this article focuses on two representative scenarios—office work and finance—to analyze the current practical level of model agent capabilities, the design logic of evaluation systems, and reliability issues that are easy to overlook.

    2. Office Scenarios: What Multi-Task Comprehensive Evaluation Reveals

    2.1 Evaluation Design and Overall Performance

    In office automation evaluations, a common approach is to pre-build workspaces covering multiple office task types, such as spreadsheet processing, document writing, schedule management, and information summarization. Each workspace contains several complete tasks that require multi-step collaboration. During evaluation, the model acts as an agent operating autonomously within the workspace, and the system assigns a quantitative score based on how well the task objectives are achieved.

    Recent public results show that leading models can already exceed 80 points (on a 100-point scale) in comprehensive office agent scenarios, with task pass rates above 90%. In terms of sub-scenarios, different models have different strengths: some perform particularly well in structured data processing, while others excel in long-document understanding and information extraction tasks. Models developed in China already lead in most office sub-scenarios, suggesting that this scenario is closely tied to Chinese-language context and localized office practices.

    2.2 Why Office Scenarios Are a Good Test of Agent Capability

    Office tasks may appear simple, but they are an ideal environment for testing agent capabilities for three reasons:

    1. Rich toolchains with clear boundaries. Interfaces for spreadsheets, documents, calendars, and instant messaging are clearly defined. The model must correctly select tools, assemble parameters, and process returned results. An error at any step can cause task failure.
    2. Limited tolerance for error. Unlike open-ended writing, office tasks often have a clear correct state—data must be calculated correctly, fields must be filled correctly, and workflows must be completed. This makes evaluation results objective and quantifiable.
    3. Multi-step dependencies expose planning weaknesses. A task such as “summarize this week’s meeting notes and generate a weekly report” involves retrieval, filtering, summarization, and formatting. It directly tests the model’s task decomposition and long-horizon planning abilities.

    For developers, this suggests that when building agent systems, priority should be given to designing tool interfaces with clear semantics and structured returns. This can significantly reduce model invocation errors.

    3. Financial Scenarios: Blind Testing in Highly Specialized Tasks

    3.1 Three Key Design Elements of the Evaluation Mechanism

    Financial scenarios have extremely high requirements for accuracy and compliance, and their evaluation systems are correspondingly stricter. They usually include the following elements:

    • Tasks built on real business processes. Evaluation tasks are not artificially constructed question banks but are extracted from real business workflows, including compliance review, data analysis, and report writing. This ensures that the evaluation distribution is consistent with the actual production environment.
    • Large-scale expert blind testing. Dozens or even hundreds of internal and external domain experts are organized to independently score task results without knowing which model produced them. Blind testing effectively eliminates scoring bias caused by brand preference.
    • Unified scoring standards. Standardized scales, such as a five-point scale, are used to comprehensively score dimensions such as accuracy, completeness, and professionalism, ensuring comparability across models.

    Public results show that leading open-source models score around 3 on such blind tests (five-point scale), with differences between models often ranging from 0.1 to 0.4 points. This indicates that in highly specialized vertical scenarios, flagship models have become relatively close in capability. What truly separates them is fine-grained domain knowledge and details of output standardization.

    3.2 Implications for Enterprises Deploying Financial AI

    Evaluation practices in financial scenarios offer two lessons for enterprise developers:

    1. The evaluation set is an asset. An evaluation set built from real business tasks is not only a basis for model selection but also a regression testing foundation for subsequent iteration and optimization. When introducing LLMs, enterprises should invest in building evaluation sets as early as possible.
    2. Human-machine collaboration remains necessary. Even top models are far from achieving a “perfect score” in expert blind tests. Retaining human review and fallback mechanisms in critical business processes is a prudent engineering choice at the current stage.

    4. Coding Ability and General Intelligence Level

    Coding is another important dimension for measuring model reasoning and tool-use capabilities. In coding evaluations that approximate real development workflows—for example, requiring the model to fix issues in a code repository within a limited number of interaction turns—Chinese-developed models have achieved solution rates above 30% and rank among the top in some evaluations. This is a significant improvement over previous-generation models and reflects progress across the full chain of code understanding, defect localization, and patch generation.

    On general intelligence indices, scores from third-party evaluation organizations show that top Chinese-developed models have entered the global top tier, but a visible gap remains compared with the most advanced level. This pattern of “leading in individual areas but still behind in aggregate” indicates that model capabilities are developing unevenly. When selecting models, organizations should examine performance specific to their own business scenarios rather than relying only on overall rankings.

    5. A Shortcoming That Cannot Be Ignored: Knowledge Reliability

    One of the most alarming findings in evaluation is that some models still force an answer when they “do not know” it, with forced-answer rates reaching 80%–90%. In other words, when lacking reliable evidence, models tend to generate content that appears plausible but is actually fabricated, rather than honestly expressing uncertainty.

    This problem is especially harmful in professional domains such as finance, healthcare, and law. A fluent but fabricated answer is far more dangerous than “I don’t know”—the latter triggers human intervention, while the former may flow directly into business operations when review is not rigorous enough.

    From an engineering perspective, hallucination mitigation can start at three levels:

    1. Retrieval-augmented generation (RAG). Anchor answers in verifiable knowledge sources and require the model to cite sources.
    2. Confidence calibration training. Introduce “abstention” samples during training and alignment so that the model learns to express uncertainty when evidence is insufficient.
    3. Post-output validation. Validate key entities, numbers, and conclusions using rules or models, and downgrade low-confidence content to human processing.

    A simple validation approach is illustrated below:

    1. def guard_answer(answer, knowledge_base, threshold=0.6):
    2. # 1. Extract key factual claims from the answer
    3. claims = extract_claims(answer)
    4. # 2. Compare each claim against the knowledge base and calculate support
    5. scores = [kb_support(c, knowledge_base) for c in claims]
    6. # 3. If too many key claims have insufficient support, abstain or escalate to a human
    7. if sum(s < threshold for s in scores) / len(scores) > 0.3:
    8. return {"action": "escalate", "reason": "too many low-confidence claims"}
    9. return {"action": "deliver", "answer": answer}

    Of course, this is only a fallback measure. The fundamental solution still depends on continued optimization of the model’s ability to “know what it knows,” which should also be a core consideration in selection evaluation. Enterprises are advised to include “no-answer questions” in their own evaluations and measure both refusal rates and fabrication rates.

    6. Summary and Recommendations

    Taken together, performance across office, finance, and coding points to several conclusions:

    • Agent capability has become the main battlefield of evaluation. Task pass rates, multi-step planning quality, and tool-call accuracy reflect a model’s production value better than traditional question-answering accuracy.
    • Gaps in vertical scenarios are narrowing, and details decide the outcome. In highly specialized scenarios such as finance, the gap between flagship models has entered the range of a single decimal place. Enterprise model selection should rely on self-built evaluations based on real tasks.
    • Reliability shortcomings must be addressed. High forced-answer rates mean that any production deployment needs retrieval augmentation, confidence validation, and human fallback mechanisms.

    For developers and enterprises, the recommended actions are: build dedicated evaluation sets from real business tasks, include both “task completion rate” and “refusal rate” as core metrics, strike a balance between automated evaluation and expert blind testing, and place LLM adoption on a quantifiable, regression-testable engineering foundation.