Evaluating LLMs on Real Financial Tasks: Open-Source Model Performance

Why Financial Scenarios Demand More from LLMs

Finance has long been one of the more challenging industries for LLM deployment. Unlike general conversation or copywriting tasks, financial work involves rigorous numerical computation, complex business rules, precise interpretation of risk-related language, and sensitivity to regulatory compliance requirements. A model that performs well on general benchmarks may not produce reliable output when applied to real financial tasks.

Evaluating whether an LLM is suitable for financial scenarios therefore requires looking beyond public benchmark scores to its overall performance on real business tasks. Real financial tasks typically share the following characteristics:

• High information density: A research report, an earnings statement, or a risk control rule often contains multiple types of information simultaneously—timestamps, ratios, year-over-year changes, risk disclosures, and more.
• Highly condensed language: Financial texts frequently use specialized expressions such as “growth rate declined,” “turned from positive to negative,” or “impairment provision.” Models must accurately understand their meaning, not just perform surface-level matching.
• Strong context dependency: The same metric can carry different meanings across industries or reporting standards. Models need strong contextual reasoning capabilities.
• High cost of errors: Misinterpretation in financial contexts can lead to flawed investment decisions, compliance risks, or even capital losses, placing higher demands on model stability and explainability.

These characteristics mean that evaluating LLMs in finance cannot simply repurpose general NLP task formats. A more realistic evaluation framework aligned with actual business workflows is needed.

An Evaluation Methodology Based on Real Financial Tasks

To more objectively measure a model’s practical capabilities in finance, a reliable approach is to involve domain experts and conduct crowd-sourced evaluations using real business tasks. This type of evaluation typically follows this process:

1. Task collection: Extract high-frequency, representative tasks from actual financial work—for example, extracting key points from earnings reports, summarizing industry research, assessing risk events, or performing metric calculations and attribution analysis.
2. Expert annotation: Domain experts with financial industry experience standardize the tasks, specifying input conditions, expected outputs, and acceptable error margins.
3. Blind comparison: Outputs from different models for the same task are anonymized and presented to experts for scoring across dimensions such as accuracy, completeness, logical coherence, and usability.
4. Weighted aggregation: Weights are assigned based on task difficulty and business importance, then combined into a composite model score.

In this evaluation approach, the “usability” of model output becomes a key consideration. Usability refers to whether the model’s answer can directly enter the business workflow—for example, whether it can serve as a first draft for a report, help analysts quickly locate critical information, or flag areas requiring human review.

Consider a typical financial task: given a company’s annual earnings flash report, the model is asked to extract core metrics such as revenue, net profit, non-recurring-adjusted net profit, and operating cash flow, and to determine year-over-year trends. A high-performing model should not only extract values accurately but also recognize the definitional difference between “net profit attributable to shareholders of the listed company” and “net profit after deducting non-recurring gains and losses,” avoiding conflation of the two.

Key Differences Revealed by Multi-Model Comparison Data

In a comprehensive evaluation targeting real financial tasks, multiple mainstream LLMs were compared. The evaluation covered hundreds of real financial tasks, with scoring by several financial domain experts. Results showed that capability gaps between models are significantly amplified in real financial scenarios.

In terms of composite scores, some open-source models have already demonstrated strong competitiveness on financial tasks, even approaching or surpassing certain closed-source models in specific sub-capabilities. The score distributions of the models evaluated showed clear tiered differences. Leading models performed consistently on numerical extraction and trend judgment tasks, while some models exhibited notable weaknesses when multi-step reasoning and business definition judgment were required.

It is worth noting that a model’s fluency in general conversation does not fully represent its reliability on financial tasks. Some models excel in open-ended Q&A but tend to produce “plausible but incorrect” outputs when tasks require strict alignment with business rules and consistent output definitions. This type of problem is especially dangerous in financial scenarios because errors are often hidden beneath fluent prose.

From a technical perspective, the causes of these capability differences may include:

• Domain coverage of training data: The proportion of financial corpora used during pre-training and fine-tuning directly affects a model’s ability to understand specialized terminology and business logic.
• Instruction following and constraint satisfaction: Financial tasks typically come with explicit output format requirements—such as “present as a table,” “output only numerical values,” or “cite the data source”—and models must comply with these constraints accurately.
• Numerical reasoning and multi-step computation: Some tasks involve year-over-year calculations, ratio computations, or range judgments, requiring the model to have reasonably reliable numerical processing logic.
• How uncertainty is expressed: When information is incomplete or ambiguous, a strong model should provide qualified conclusions rather than force-generating definitive but incorrect answers.

Model Selection and Deployment Strategy Based on Evaluation Results

For developers and enterprise users, the value of evaluation data lies not just in “which model ranks higher” but in providing reference for practical model selection and deployment. When introducing LLMs into financial scenarios, the following dimensions should be considered holistically:

First, define the error tolerance boundary of the business scenario. If model output supports human analysis, the tolerance for errors is relatively larger; if it feeds directly into automated processes, accuracy requirements are higher. Under different tolerance requirements, the priorities for model selection will differ.

Second, recognize the role of domain fine-tuning and prompt engineering. Even models with strong baseline capabilities still need domain fine-tuning or carefully designed prompts to align output specifications with a specific financial institution’s business definitions. For example, a prompt can explicitly state “net profit uniformly adopts the definition of net profit attributable to the parent company’s shareholders” to reduce errors caused by definition confusion.

Third, establish continuous evaluation and regression mechanisms. Financial business rules and market conditions change constantly. After deployment, models need regular regression testing with new real tasks to monitor capability degradation or rule drift. An automated evaluation pipeline can be set up to continuously incorporate new real tasks into the test set.

Fourth, consider the controllability and deployment cost of open-source models. For financial institutions with strict data compliance requirements, open-source models allow deployment in local environments or dedicated cloud environments, keeping data within the internal network—a clear compliance advantage. Additionally, open-source model weights are visible, facilitating security audits and targeted optimization.

Below is an illustrative evaluation task prompt designed to guide a model through structured financial data extraction:

Task: Extract core financial metrics from the given text.
Requirements:
1. Extract only data explicitly present in the text; do not infer or fill in missing values.
2. If definitional differences exist, list them separately and label the definition name.
3. Output format is JSON with fields: metric name, value, unit, year-over-year change, source sentence.
4. If a metric does not appear in the text, fill the corresponding field with null.

This approach compresses ambiguity in financial tasks as much as possible, making model output more deterministic and verifiable.

Future Directions for Financial LLM Capabilities

Based on current evaluation results, the usability of LLMs in financial scenarios is improving rapidly, but large-scale unattended deployment remains some distance away. Future capability evolution may focus on the following directions:

Stronger numerical reasoning and computational reliability. The many metric calculations and trend judgments in financial tasks require greater stability in numerical processing. A “model + computation” collaborative approach that integrates external calculation tools or structured data sources may be an effective path to improving accuracy in the near term.

More granular understanding of business definitions. The same metric may have different definitions across financial institutions or regulatory frameworks. Models need stronger contextual awareness to dynamically adjust output behavior based on definitional specifications in a given document or prompt.

More robust output verification and explanation mechanisms. When providing conclusions, models should also supply the original textual evidence or reasoning chain supporting those conclusions, enabling rapid human review. This traceability is key to building trust in LLMs for financial scenarios.

More efficient domain adaptation approaches. Through parameter-efficient fine-tuning, retrieval-augmented generation, and similar techniques, the cost of domain adaptation can be reduced, enabling smaller financial institutions to build knowledge Q&A and analysis systems tailored to their own business needs using open-source models.

Overall, evaluation based on real financial tasks provides more valuable reference for LLM applications in finance than general benchmarks. As evaluation frameworks mature and model capabilities continue to improve, LLMs are expected to play a more substantive role in financial research, risk analysis, and intelligent investment research. For developers and enterprise users, the key at this stage is not pursuing the single best model but establishing evaluation, selection, and continuous optimization mechanisms that match their own business requirements.