Evaluating Large Language Models for Financial Tasks and Research Report Generation

1. Background: Why Financial Scenarios Require Stricter Capability Evaluation

The financial industry has significantly different requirements for text generation models compared to general conversation and everyday writing scenarios. Financial tasks typically involve numerical computation, logical reasoning, compliant expression, risk disclosure, and precise use of professional terminology. A model that performs fluently in general Q&A may still produce numerical deviations, incorrect attributions, or non-standard expressions when faced with financial metric calculations, industry comparisons, or research report framework generation.

Therefore, before introducing large language models into financial analysis, investment research assistance, report generation, and similar business processes, a more rigorous evaluation system that closely mirrors real workflows must be established. This system cannot rely solely on static multiple-choice questions or subjective scoring. Instead, it should simulate the actual task formats of financial practitioners as closely as possible, including but not limited to:

• Financial statement data extraction and calculation
• Industry prosperity assessment and evidence citation
• Corporate operational risk identification
• Multi-period data comparison and trend explanation
• Automatic organization and professional expression of research report structure

Only through real task-driven evaluation can one accurately determine whether a model possesses the fundamental capabilities required to enter financial business scenarios.

2. Evaluation Method Design: Real Tasks and Expert Crowdtesting as the Core

To obtain more referenceable conclusions, this evaluation adopted a combination of “real task sets + expert crowdtesting” rather than relying solely on automated scoring scripts or small-scale manual sampling.

1. Building Real Financial Task Sets

Evaluation tasks were derived from the daily work of financial practitioners, covering multiple high-frequency sub-scenarios. Each item in the task set includes clear input conditions, output requirements, and evaluation dimensions. Typical task types are as follows:

Task Type	Example Description	Key Assessment Points
Financial calculation	Calculate gross margin and net margin changes based on consecutive multi-period income statements	Numerical accuracy and calculation logic
Text comprehension	Extract operational risk points from the management discussion section of annual reports	Information extraction and semantic understanding
Comparative analysis	Compare solvency indicators of different companies in the same industry	Attribution capability and conclusion robustness
Structured generation	Organize scattered data into research report summary paragraphs	Professional expression and structural organization
Risk assessment	Identify potential risks in a certain asset allocation plan	Risk awareness and compliance boundaries

Compared to general evaluation sets, these tasks place greater emphasis on “process inspectability, result verifiability, and error localizability.” For example, in financial calculation tasks, not only is the final numerical value checked for correctness, but intermediate steps are also examined for formula misuse, unit confusion, or inconsistent calculation standards.

2. Introducing Expert Crowdtesting Mechanisms

To ensure evaluation quality, multiple experts with financial industry backgrounds were invited to participate in scoring. Expert reviewers did not have access to model identity information and scored task outputs across multiple dimensions. Common scoring dimensions include:

• Accuracy: Whether conclusions are consistent with given data and calculations are correct
• Completeness: Whether key elements required by the task are covered
• Logicality: Whether the reasoning chain is clear and free of gaps or circular arguments
• Readability: Whether the language is standard and suitable for financial scenarios
• Risk and compliance: Whether misleading statements exist or necessary risk disclosures are missing

The advantage of expert crowdtesting lies in its ability to identify semantic issues that automated evaluation struggles to capture. For example, an answer with completely correct numerical values but incorrect attribution direction might receive a high score in automated scoring, but experts can identify its flaws from a business logic perspective.

3. Interpreting Evaluation Results: From Single Scores to Capability Profiles

After completing expert review on real task sets, comprehensive performance across different models can be obtained. Results show clear stratification among participating models, and this difference is more pronounced in specialized financial tasks than in general Q&A scenarios.

1. Composite Scores Reflect Task Adaptability

A composite score is not equivalent to an absolute ranking of a model’s “intelligence” but rather reflects its adaptability to a specific financial task set. Models with higher scores typically exhibit the following characteristics:

• Better step control for numerical calculation tasks
• Professional expression closer to the habits of financial practitioners
• More complete coverage of risk disclosures
• Maintaining contextual consistency in multi-step reasoning

These capabilities cannot be guaranteed solely by increasing model parameter scale. They are also related to training data distribution, instruction-following capability, and structured output training.

2. Score Differences Concentrate in High-Difficulty Tasks

On simple factual Q&A, all models generally perform well with small score gaps. However, on complex tasks involving multi-step calculations, cross-information verification, and research report paragraph generation, the gaps widen rapidly. This indicates that model selection in financial scenarios should focus on stability in high-difficulty tasks rather than just looking at average scores.

3. From Total Scores to Capability Profiles

Looking at a single total score can easily mask a model’s specific weaknesses. In practice, it is more advisable to break down evaluation results into multiple sub-dimensions to form “capability profiles.” For example:

• A model may excel in financial calculations but be overly conservative in risk assessment
• Another model may have fluent language expression but insufficient numerical accuracy
• Yet another model may be stable in structured generation but prone to logical errors in complex attribution

Such fine-grained profiles provide more direct guidance for subsequent model selection, prompt optimization, and fine-tuning direction choices.

4. Research Report Generation in Practice: From Task Evaluation to Content Output

The ultimate purpose of evaluation is to determine whether a model can undertake part of the financial content generation work in real business operations. Research report generation is one of the most representative scenarios.

1. Core Challenges of Research Report Generation

A qualified research report is not merely an accumulation of text. It also involves the following requirements:

• Data accuracy: All financial data and market indicators must be consistent with source data
• Rigorous logic: Conclusions must be supported by data and analytical processes
• Standard structure: Typically includes summary, body analysis, risk disclosures, and other modules
• Cautious wording: Avoid exaggeration and absolute statements
• Compliance awareness: Be cautious with forward-looking content to avoid misleading

These requirements determine that research report generation cannot rely entirely on open-ended free continuation. Instead, stricter task decomposition and process control are needed.

2. A Deployable Generation Workflow

In practical applications, a “step-by-step generation + intermediate validation + structured output” approach can be adopted rather than generating an entire report in one pass.

Step 1: Data Organization and Validation

Organize input data into a structured format using unified standards, for example:

Company name: Example Company
Reporting period: FY2024
Revenue: xxx billion CNY
Year-over-year growth: xx%
Net profit: xxx billion CNY
Operating cash flow: xxx billion CNY
Key risk items: Intensified industry competition, raw material price fluctuations

The model first performs consistency checks on the data, confirming no obvious contradictions among indicators, before proceeding to the analysis phase.

Step 2: Module-by-Module Analysis Generation

Break the research report into several logical modules, generating and validating each one:

• Business overview
• Profitability analysis
• Cash flow and solvency
• Industry and competitive environment
• Risk disclosures

Generating each module separately helps control output quality and locate errors. If a particular module deviates significantly, only that section needs to be regenerated without discarding the entire document.

Step 3: Conclusion and Risk Disclosure Supplementation

After completing the analysis modules, generate a summary conclusion. The conclusion section should avoid simply repeating previous content and instead distill key findings. Additionally, necessary risk disclosures must be included, explaining the assumptions and uncertainties underlying the analysis.

3. Structured Expression of Output Examples

A simplified structured output can be presented as follows:

[Business Overview]
Example Company achieved revenue of xxx billion CNY in FY2024, representing year-over-year growth of xx%, primarily driven by expansion in core business scale.
[Profitability]
Gross margin was xx%, up x percentage points from the previous year; net margin was xx%. Profitability improvement mainly came from product mix optimization.
[Cash Flow and Solvency]
Net operating cash flow was xxx billion CNY, indicating a sound cash flow position; short-term solvency indicators are within a reasonable range.
[Risk Disclosures]
Attention should be paid to the impact of intensified industry competition on future pricing, as well as cost pressures from raw material price fluctuations.

This structured output facilitates subsequent systematic processing and better meets the auditability requirements of financial business operations.

5. Practical Recommendations: How to Build Your Own Financial Model Evaluation and Generation System

For development teams or enterprise users hoping to introduce large language models into financial business scenarios, the following aspects can serve as starting points:

1. Build Task Sets That Align with Business Operations

Do not directly copy general evaluation sets. It is recommended to extract typical tasks from the real work records of business teams and classify and iterate them by task type, difficulty, and domain. Task sets should be updated regularly to reflect changes in business priorities.

2. Introduce Human Expert Review Mechanisms

Especially in the early stages of evaluation, human expert review cannot be omitted. A unified scoring rubric can be designed to clearly define each dimension and scoring criteria, reducing subjective differences among reviewers. As evaluation data accumulates, automated scoring can be gradually introduced as a supplement.

3. Focus on Fine-Grained Capability Metrics

Beyond composite scores, it is recommended to track model pass rates, average scores, and error distributions for each sub-task category. This helps teams quickly identify which tasks a model is suitable for and which tasks require additional rule-based safeguards or human review.

4. Design Controlled Generation Workflows

For high-risk content generation tasks such as research reports and analytical summaries, do not directly perform end-to-end free generation. Instead, adopt a “data validation — module-by-module generation — structural assembly — compliance check” workflow to keep model outputs within controllable boundaries.

5. Combine Logging with Version Management

Financial scenarios have high traceability requirements. It is recommended to retain complete logs for every model output, including input data version, prompt version, model version, and post-processing rule version. This allows quick tracing of problem sources when disputes arise.

6. Summary

The difficulty of applying large language models in the financial domain lies not in whether they can generate text, but in whether they can maintain sufficient accuracy, logicality, and compliance in real tasks. By building real task sets, introducing expert crowdtesting, and refining capability profiles, one can more objectively evaluate a model’s actual performance in financial business operations.

On this basis, adopting step-by-step generation and structured output can transform model capabilities into an auditable and controllable research report generation workflow. For technical teams, the key is to establish an evaluation and generation mechanism deeply integrated with business operations, rather than focusing solely on a model’s score performance on general leaderboards.