Evaluating Financial LLMs and Automating Research Report Generation

1. Background: Why Financial Scenarios Require Specialized Evaluation

Text-based tasks in the financial sector differ significantly from general conversational or writing tasks. Financial texts are characterized by high information density, strict compliance constraints, complex numerical relationships, and strict timeliness requirements. A model that performs well on general benchmarks may not maintain stable outputs in tasks such as financial statement analysis, policy interpretation, risk alerting, or research report writing.

Therefore, before deploying LLMs to support financial operations, it is necessary to build an evaluation set based on actual business tasks rather than relying solely on public leaderboards or subjective impressions. The core objective of evaluation is not to give a simplistic “good or bad” verdict, but to answer three specific questions:

1. Can the model understand financial semantics and contextual constraints?
2. How stable is the model in numerical reasoning, logical induction, and text generation?
3. Does the model output meet the requirements for business readability, compliance, and traceability?

2. Evaluation Design: Breaking Down Dimensions Based on Real Financial Tasks

When validating model capabilities for financial scenarios, you can construct an evaluation process covering multiple task types, involving numerous domain experts. For instance, an evaluation set can be formed by organizing over two hundred financial experts to define nearly six hundred real-world financial tasks. These tasks are not simple Q&A pairs or multiple-choice questions; instead, they closely replicate complex scenarios from actual workflows, including but not limited to:

• Financial report summarization and key metric extraction
• Industry policy impact analysis
• Corporate announcement interpretation
• Macroeconomic data commentary
• Research report fragment continuation and structure generation
• Financial risk identification and alerting
• Multi-document information fusion and cross-validation

During the evaluation process, experts score the model across multiple dimensions, commonly including:

• Information Accuracy: Whether key numbers, dates, entity names, and logical relationships are correct.
• Analysis Depth: Whether the output merely restates surface-level facts or extracts structural judgments.
• Expression Standardization: Whether the language is rigorous, clear, and aligns with financial writing conventions.
• Risk Awareness: Whether the model highlights uncertainties, data sources, or assumptions where necessary.
• Task Completion: Whether the response fully addresses the prompt without omissions or irrelevancies.

The benefit of this multidimensional human evaluation is that it reflects the model’s actual usability in real financial operations, rather than just measuring the fluency of the generated text.

3. Interpreting Results: What the Scores Represent

In comprehensive scoring, different models will exhibit clear stratification. Results are typically presented as average scores. For example, a specific financial LLM might score 3.04 in overall performance, while comparable open-source models might score between 2.6 and 3.0. Although the score gap may seem marginal, in complex financial tasks, slight differences often indicate significantly higher usability.

It is important to note that a high score does not equate to an “absolute accuracy rate.” Financial tasks often lack a single standard answer; expert scoring encompasses objective correctness judgments alongside a comprehensive weighing of logical quality, expression professionalism, and business operability. Therefore, evaluation reports are best used as a reference for model selection rather than the sole decision-making basis.

From an engineering perspective, a financial model demonstrates strong deployment potential if it performs stably in the following areas:

• Comprehension of long-text and multi-document materials
• Accurate transcription of numbers, percentages, and year-over-year (YoY) or quarter-over-quarter (QoQ) relationships
• Distinguishing between facts, inferences, and recommendations when generating conclusions
• Explicitly stating assumptions when information is insufficient
• Standardized output formatting for subsequent automated processing

4. Research Report Generation: From Task Decomposition to Prompt Design

Research reports are among the most representative long-text generation tasks in financial scenarios. When applying LLMs to report generation, you should not expect a complete report in a single pass. Instead, break it down into controllable subtasks to form a pipeline-style workflow.

A common research report generation workflow is as follows:

1. Data Preprocessing

Chunk, clean, and annotate metadata for materials such as financial reports, announcements, industry data, and news. Focus on retaining time, entities, metric names, and values to prevent confusion during subsequent generation.

2. Structured Outline Generation

First, prompt the model to output a report outline based on the available materials. The outline typically includes:

• Executive Summary
• Industry Background
• Company or Target Overview
• Financial Analysis
• Risk Factors
• Valuation and Outlook

The goal of the outline phase is to lock in the logical framework and prevent content divergence in later stages.

3. Section-by-Section Generation and Verification

Generate content chapter by chapter according to the outline. The prompt should explicitly require:

• Drawing conclusions strictly from the provided materials
• Retaining original units and time scopes for numerical data
• Citing source IDs or material locations for referenced information
• Using phrasing like “Based on available materials” or “Subject to uncertainty” for uncertain information

An example prompt structure is as follows:

You are a financial research assistant. Please write the "Financial Analysis" section based on the following materials.
Requirements:
1. Use only the financial data present in the materials;
2. Retain original time scopes and units;
3. Explain YoY and QoQ changes;
4. Explicitly state if the materials are insufficient;
5. Output in Chinese, approximately 500 words.
Materials:
[Material 1] ...
[Material 2] ...

4. Cross-Section Consistency Check

Once generation is complete, perform a consistency check focusing on:

• Numerical consistency for the same metric across different sections
• Logical conflicts between viewpoints
• Alignment between risk alerts and conclusions
• Consistent terminology usage

This step can be assisted by an LLM, but human review is still recommended for the final verification.

5. Compliance and Risk Review

Financial research reports involve strict compliance requirements. Add a review phase before official publication to check for:

• Predictive conclusions lacking sources
• Strong recommendation statements without risk alerts
• Absolute judgments on policies or market trends
• Conflation of facts and opinions

5. Key Challenges in Engineering Deployment

When deploying financial LLMs in production business systems, evaluation and generation are just one part of the lifecycle. The following issues require close attention during practice:

1. Alignment Between Evaluation Sets and Business Scenarios

If the evaluation set deviates significantly from actual business operations, a high score will not reflect real-world post-deployment performance. Enterprises should build small-scale internal evaluation sets based on their own business and cross-validate them with public evaluation results.

2. Model Output Stability

Financial scenarios demand high stability. Excessive output variation for the same task across different runs will directly disrupt downstream automated workflows. Fluctuations can be minimized through temperature parameter control, fixed prompt templates, and integrated self-verification steps.

3. Context Length and Information Loss

Research report generation typically requires feeding large volumes of data. Design a sound material selection strategy to avoid ignoring critical information by overloading the context window all at once. Techniques like chunked retrieval, summary compression, and hierarchical concatenation can improve context utilization.

4. Human Review Cost Control

Although LLMs can significantly boost generation efficiency, human review remains necessary in financial scenarios. A practical approach is to let the model generate the first draft and perform basic validation, focusing human effort on key judgments and compliance reviews.

6. Conclusion

The application of LLMs in financial scenarios should not focus solely on parameter scale or general conversational ability, but rather on comprehensive performance in real financial tasks. By building evaluation sets aligned with business operations, decomposing research report workflows, and introducing multi-stage validation mechanisms, LLMs can transition from merely “capable of writing” to “ready for production.”

For developers and enterprise users, the greater value lies not in ranking on a single evaluation, but in establishing a continuously iterable evaluation and generation system. As business data accumulates and evaluation dimensions refine, the model’s capabilities in financial text comprehension, research, and generation can truly be transformed into productivity.