From Research Reports to Trading Formulas: How to Evaluate LLM Coding Capabilities

As AI coding tools continue to evolve, deciding whether a model is truly useful requires more than reviewing benchmark rankings or trying one code-generation prompt. For developers, a more meaningful evaluation places the model in a workflow that resembles real work and measures whether it can complete the full process—from reading source material and understanding technical concepts to deriving logic, delivering code, and checking the result.

This article uses a representative technical task: extracting the definition of an indicator from an image-based or document-based research report and converting it into the formula syntax used by commonly adopted market-data platforms. Although the task may appear to involve only “reading a document and writing a formula,” it tests several core capabilities at once, including document parsing, domain-term interpretation, context retention, mathematical translation, code generation, self-review, and uncertainty handling.

1. Why a Single Code-Generation Test Is Not Enough

Traditional coding tests often provide a clear requirement and ask the model to produce code. This approach can overlook the most difficult parts of practical engineering work. Requirements are often incomplete, key definitions may be hidden in images, tables, footnotes, or surrounding explanations, and the final implementation must comply with the syntax of a specific platform.

For example, a user might provide only the following request:

Based on the definition in the research report, reproduce the ICU moving-average indicator and output a market-data formula that can be used directly.

To complete this task, a model must address at least the following questions:

1. Is the indicator name ambiguous?
2. Is the definition located in the main text, an image, a table, or a footnote?
3. Does “moving average” mean a simple moving average, an exponential moving average, or a specially smoothed series?
4. Does the formula use closing prices, typical prices, or another input?
5. Is the calculation period specified, and should it remain configurable?
6. Which functions, variables, and plotting statements are supported by the target formula language?
7. How can the generated formula be checked for semantic drift?

A useful evaluation should therefore examine more than whether the final code appears executable. It should determine whether the model has accurately reconstructed the original definition and can explain the basis for each conversion step.

2. How to Design the Complete Task Workflow

A reliable process can be divided into six stages.

2.1 Preprocess the Source Material

First, determine whether the source material can be analyzed directly. If the research report is a scanned image, optical character recognition should be performed first. If it is an electronic document, check whether text, tables, and images are mixed together.

The following information should be preserved:

• Document page numbers
• The location of the indicator name
• Screenshots of the original formula
• Explanatory context surrounding the formula
• Parameter definitions and units
• Chart titles and annotations
• Version, date, or revision information

The goal is not merely to convert a file into text. It is to preserve the relationship between content and location. For technical indicators, a formula without its surrounding context may not provide enough information to determine what its variables mean.

2.2 Extract the Original Definition

When extracting an indicator, the model should first produce a structured definition rather than immediately generating a formula for the target platform. A useful structure is:

Indicator name:
Input data:
Intermediate variables:
Core calculation steps:
Parameters and default values:
Output:
Special processing:
Boundary conditions:
Source in the document:

For a moving-average indicator, the following points must be made explicit:

• Whether the input sequence is the closing price or another price series
• Whether returns, differences, or weighted values are calculated first
• Whether the smoothing method is a simple average or exponential smoothing
• Whether the period is fixed
• How insufficient initial data is handled
• Whether normalization, offsets, or threshold checks are applied

If the source does not specify an item, the model should mark it as “requires confirmation” rather than filling it in without evidence. This is more reliable than producing a complete-looking formula with the wrong meaning.

2.3 Convert the Natural-Language Definition into a Formula

Only after the definition has been extracted should formula conversion begin. Three different layers should be distinguished.

Mathematical Definition

For example, a simple moving average can be expressed as:

MA(n) = (X1 + X2 + ... + Xn) / n

Here, X is the input sequence and n is the calculation period.

Pseudocode Definition

source = close
result = moving_average(source, period)

Target-Platform Syntax

Different formula environments may use different variable names, function parameters, plotting statements, and comment conventions. Conversion is therefore not a matter of replacing words. It requires semantic mapping:

Abstract concept	Handling in the target formula
Closing-price series	Use the platform’s standard price field
Moving average	Map to an average function supported by the platform
Period parameter	Use a constant or parameter variable
Output series	Use a plotting or return statement
Conditional logic	Map to the platform’s supported logical expressions

A generic example might look like this:

P := CLOSE;
N := 20;
ICU := MA(P, N);
ICU;

This example illustrates the conversion structure only. It does not mean that the ICU indicator in a particular research report is necessarily a standard moving average. If the source defines a special calculation, the original definition must take priority. The model should not apply a standard moving average simply because the indicator name contains the term “moving average.”

2.4 Handle Identical Names with Different Definitions

Indicator names are not always standardized across the industry. The same abbreviation may refer to different algorithms in different sources, or it may be a custom indicator created by a particular research team.

The model should therefore prioritize formulas and variable definitions rather than infer an algorithm from the name alone. The following checks are useful:

• Does the main text clearly explain the indicator name?
• Can every variable in the formula be found in the preceding material?
• Does the chart’s curve behave consistently with the formula?
• Are the period and parameters defined in an appendix or footnote?
• Could subscripts, minus signs, or parentheses have been lost?
• Could OCR have misread numbers, letters, or operators?

Image-based formulas are especially vulnerable to errors such as:

• Confusing 1 with I
• Confusing 0 with O
• Omitting subscripts
• Treating a minus sign as a space
• Converting a fraction into a continuous string of text
• Changing the nesting of parentheses
• Losing multiplication signs, dots, or decimal points

If a model does not identify these risks and simply outputs a syntactically formatted formula, that is not sufficient evidence that it has understood the source.

3. How to Evaluate Model Performance on This Type of Task

Instead of focusing only on whether the model can generate code, evaluation should cover the following dimensions.

3.1 Document Understanding

Can the model accurately locate the indicator definition? Can it distinguish explanations in the main text, chart descriptions, and sample data? Does it mistake irrelevant content for part of the formula?

3.2 Auditable Reasoning

The model should be able to explain:

• Why a particular input variable was selected
• Why a specific moving-average function was used
• Where each parameter came from
• Which details are explicitly stated in the source
• Which details were inferred from common industry practice
• Which issues still require human confirmation

Auditability is more important than an answer that merely appears reasonable. In financial and engineering scenarios, errors are often semantic rather than syntactic.

3.3 Executable Code and Formula Syntax

The generated result should comply with the constraints of the target environment, including:

• Legal variable names
• Functions that actually exist
• Correct parameter ordering
• Correct statement separators
• Comments that do not break parsing
• Output statements supported by the platform

If the target environment is unknown, the model should ask which platform is being used or provide an abstract formula together with the items that still require adaptation. It should not assume that all platforms use the same syntax.

3.4 Error-Detection Capability

A high-quality model should not only generate content but also review its own output. For example, it should:

• Compare the generated formula with the original formula line by line
• Calculate the first few values with hand-worked sample data
• Compare outputs under different periods
• Check behavior when there is insufficient data
• Verify that no future data is referenced
• Check whether variables are overwritten unintentionally

Future-data leakage is particularly dangerous in quantitative research. Some formulas run successfully but use data from future periods, producing results that appear excellent but cannot be achieved in live trading.

4. Recommended Validation Methods

4.1 Manual Sample Validation

Construct a simple dataset such as:

Price series: 10, 12, 14, 16, 18
Period: 3

If a simple moving average is used, the first three valid outputs should be:

(10 + 12 + 14) / 3 = 12
(12 + 14 + 16) / 3 = 14
(14 + 16 + 18) / 3 = 16

Compare these values with the output from the formula platform. This can quickly reveal errors in period direction, window boundaries, or initial-value handling.

4.2 Cross-Implementation Validation

Implement the same indicator separately in a spreadsheet, a scripting language, and the target formula environment, then compare the outputs. If the implementations differ, check the following first:

• Data ordering
• Missing-value handling
• Initial-window behavior
• Smoothing algorithm
• Decimal precision
• Whether the current period is included

4.3 Visual Consistency Checks

If the research report includes an indicator chart, compare the reproduced curve with the original chart’s trend. Visual comparison cannot replace formula verification, but it can reveal obvious direction errors, incorrect periods, and abnormal jumps.

4.4 Extreme-Condition Testing

At minimum, test the following cases:

• The data volume is smaller than the calculation period
• The input sequence contains missing values
• Prices rise continuously
• Prices fall continuously
• All input values are identical
• The parameter is set to the smallest valid value
• The parameter is empty or outside the permitted range

These tests help determine whether the model merely tailored an answer to the sample or actually considered the algorithm’s complete behavior.

5. How to Write Better Prompts for This Workflow

To reduce unsupported guesses, split the task into multiple rounds instead of asking the model to “read the document and generate a formula” in a single prompt.

The first round should focus only on extraction:

Locate all content related to the target indicator in the document.
Do not generate code.
List the original definition, variables, parameters, formulas, and any ambiguities.

The second round should reconstruct the mathematics:

Organize the extracted content into a rigorous mathematical definition.
Distinguish information explicitly stated in the source from inferred information.
If the definition is incomplete, list the questions that require confirmation.

The third round should perform syntax conversion:

Convert the confirmed mathematical definition into the target formula language.
Preserve the meaning of each parameter and explain which mathematical step each line of code represents.

The fourth round should validate the result:

Use a small sample dataset to derive the results by hand.
Check for period shifts, future-data references, and boundary-condition errors.

This staged approach has two advantages. It allows human intervention at each step, and it makes it easier to identify whether an error occurred during document recognition, mathematical interpretation, or syntax conversion.

6. A Practical View of Intelligent Coding Models

From a development perspective, newer models have made clear progress in code generation, context analysis, and tool coordination. They can save significant time on tasks such as:

• Reading multi-page technical material
• Organizing complex requirements
• Generating initial scripts
• Writing test cases
• Explaining error messages
• Refactoring repetitive code
• Adding documentation and comments

However, they still cannot replace professional review and final confirmation. For indicator reproduction, financial calculations, data analysis, and production code, models are best used as efficient drafting and review assistants—not as unreviewed final decision-makers.

Different models also have different strengths. Some are better at long-document summarization, some at code completion, and others at image understanding or complex reasoning. Evaluation should therefore use a task-specific test set based on real work rather than relying solely on public leaderboards or a single example.

Conclusion

Extracting an indicator from a research report and reproducing it as an executable formula is a small but representative engineering task. It demonstrates that the real value of an intelligent coding tool is not limited to writing code. The tool must also understand source material, identify ambiguity, explain its reasoning, follow the target syntax, and support repeatable validation.

For developers, the safest approach is to establish a closed loop of extraction, definition, conversion, validation, and review. For organizations, real business documents, boundary cases, and known error samples should be included in the evaluation set. Only by testing a model across the complete workflow can its production value be assessed without being misled by a single apparently successful code-generation result.