Agent Evaluation and Financial Scenario Engineering Practices

1. From “Taking Exams” to “Getting Work Done”: The Shift in Agent Evaluation

Over the past two years, the dominant approach to model capability evaluation has resembled standardized testing: given a single-turn question, compare whether the answer matches a reference solution. The advantages are strong comparability and low cost, but the drawback is equally clear—it measures “question-answering ability” rather than “delivery capability.” Real business tasks often span multiple interaction turns, involve tool calls, and require reading external data mid-process and adjusting strategy based on results. Instability at any step can lead to delivery failure.

The emergence of agent evaluation benchmarks is a direct response to this limitation. The core approach can be summarized in three points:

1. Task realism: Evaluation tasks come from real workflows, not artificially constructed questions.
2. Process observability: Beyond final results, intermediate tool calls, state changes, and error recovery are also recorded.
3. Cost measurability: Average consumption per unit task (tokens, tool call count, execution time) is tracked, making “effectiveness vs. cost” a two-dimensional comparison.

Under such an evaluation framework, a model’s overall leaderboard ranking only indicates its approximate position on the effectiveness dimension. What actually guides engineering selection is whether it occupies a position on the Pareto frontier of “optimal effectiveness at equivalent cost.” For two models with adjacent rankings, if the performance score gap is within 1%, the actual difference often falls within the evaluation noise range; in this case, cost difference becomes the deciding factor—a model with roughly 60% lower unit task cost will see its total cost of ownership advantage significantly amplified in large-scale batch execution scenarios.

This also addresses a common engineering judgment: performance is the first element of cost-effectiveness, and cost is the second, but both must hold simultaneously. When performance falls short, low cost is meaningless; when performance is close, cost becomes the only differentiator. The value of the Pareto frontier lies in placing both dimensions on the same chart, transforming selection decisions from “who scores higher” to “who is better under my cost constraints.”

2. Why Financial Scenarios Are a “Stress Test” for Agents

Among vertical domains, financial analysis places particularly demanding requirements on agents because its task structure carries three layers of complexity:

First, long processes with many states. A complete securities analysis report may involve data extraction, financial metric calculation, peer comparison, valuation modeling, and conclusion drafting—each stage depending on the output of the previous one. An agent needs to maintain contextual consistency across a long chain, avoiding “forgetting” key constraints mid-process.

Second, strong regulatory and procedural requirements. Fund research, asset allocation, taxation and accounting all have mature industry workflows and compliance requirements. Output cannot simply “look reasonable”—it must conform to business standards. This means the model needs to learn professional procedures, not just professional vocabulary.

Third, low fault tolerance. Numerical errors, metric definition confusion, and unit mismatches in financial scenarios directly propagate into incorrect conclusions. Evaluation tasks must be designed to expose these issues rather than merely assessing text fluency.

Therefore, capability enhancement for financial scenarios typically requires building large-scale workspaces and high-quality task sets: workspaces provide near-realistic file structures, data tables, metric definition documentation, and historical context, while tasks span the full spectrum from single-point calculations to end-to-end report generation. Publicly disclosed approaches involve constructing thousands of workspaces and tens of thousands of high-quality tasks covering fund research, securities analysis, quantitative strategy, asset allocation, corporate finance, and taxation/accounting, allowing models to encounter professional workflows and standards during training rather than facing them for the first time during evaluation.

3. Engineering Interpretation Behind Evaluation Data

Public large-scale expert evaluations in the financial domain provide another set of reference data: based on hundreds of real financial tasks with hundreds of participating experts, one model achieved a composite score of 3.04, ranking first among open-source models; comparison groups scored 2.68, 2.68, and 2.97 respectively. The engineering implications of these numbers require careful interpretation:

• Absolute scores have limited meaning; relative gaps are more valuable. The gap between 3.04 and 2.97 is not dramatic, suggesting that top-tier model capabilities on financial tasks are converging. The real differentiation lies in complex task subsets rather than average scores.
• The value of expert evaluation lies in task authenticity. Tasks designed by industry experts more closely approach the boundary cases of real workflows and can expose model weaknesses in metric interpretation and abnormal data handling.
• Open-source availability affects deployment paths. Open-source models enable private deployment, incremental fine-tuning on internal organizational standards, and keeping sensitive data on-premises—hard constraints in the financial industry.

Taking both sets of information together yields a practical conclusion: among models readily accessible to most teams, this class of model ranks in the first tier for comprehensive financial task performance, with a cost structure supporting large-scale invocation. For teams needing batch research report generation or batch financial data verification, this is an option worth including in the candidate set.

4. Implementation Practice: How to Verify Whether a Financial Agent Is Usable

Selection cannot rely solely on leaderboards. Below is a reusable validation framework organized into four progressive layers.

4.1 Layered Task Design

Divide candidate tasks into three complexity tiers with separate pass criteria:

tasks = {
    "L1_single_point_calculation": "Given a financial statement excerpt, calculate a specified ratio and provide the unit",  # pass rate > 95%
    "L2_multi_step_analysis": "Perform peer comparison across two reports, output a comparison table and conclusion",      # pass rate > 80%
    "L3_end_to_end_delivery": "Generate a complete analysis report with valuation range based on the workspace",           # human score > 3/5
}

L1 identifies foundational capability gaps, L2 tests long-chain consistency, and L3 corresponds to actual business delivery. The structure of three-tier pass rates is more informative than a single aggregate score: if L1 is high but L2 drops sharply, the model has a weakness in multi-step state maintenance; if L1 itself fails to meet the threshold, it should not proceed to business piloting.

4.2 Workspace Construction

A workspace should include: raw data files, metric definition documentation, historical output samples, and common error examples. Metric definitions are especially critical—many errors in financial tasks stem from misunderstanding indicator definitions. Writing definitions into the context effectively shifts part of the domain expertise from “model memory” to “contextual supply,” which is an effective means of reducing hallucination.

4.3 Cost Accounting

Unit task cost must be calculated based on actual call structure, not simply multiplied by unit price:

Unit task cost ≈ (input tokens × input price) + (output tokens × output price) + (tool call count × per-call cost)

Retry costs should also be included in batch scenarios. If a model has a low first-pass success rate, the additional consumption from retries will offset its unit price advantage—a classic trap where “low price does not equal low cost.”

4.4 Human Review Feedback Loop

Establish a sampling review mechanism for L3 tasks, recording error type distributions (numerical errors, metric definition errors, conclusion overreach, format mismatches), and use these types to trace back improvements to prompts and tool chains. Review data can simultaneously serve as corpus material for subsequent fine-tuning.

5. Selection Recommendations and Considerations

In summary, financial agent selection should follow these principles:

1. Set cost constraints first, then compare effectiveness. Filter models within an acceptable unit task cost range, avoiding excessive premiums for marginal performance improvements.
2. Use layered task validation rather than leaderboard ranking alone. Differences between adjacent-ranked models may be smaller than evaluation noise; business validation is the ultimate basis.
3. Prioritize options that support private deployment and fine-tuning. The sensitivity of financial data makes deployment form a hard constraint, not an optional preference.
4. Make metric knowledge explicit. Using workspaces and contextual supply to reduce reliance on model memory is a cost-effective way to improve stability.
5. Establish a continuous evaluation mechanism. Model versions iterate frequently, and evaluation conclusions have limited validity periods. Evaluation task sets and review processes should be institutionalized as regular engineering assets.

It should be noted that evaluation leaderboards reflect statistical results on specific task sets and cannot be directly equated with performance on an organization’s own business scenarios. Any selection should go through small-scale piloting, layered validation, and cost accounting before deciding whether to expand scope. For the financial domain—where regulatory requirements are strong and fault tolerance is low—a disciplined engineering validation process deserves more investment than model rankings themselves.