Evaluating Next-Generation AI Agents for Data Analysis and Financial Research

    As large language models evolve from conversational systems into agents that can plan tasks, call tools, and generate files, they are changing how data analysis and financial research are performed. This article explains why deterministic scripts, data-driven charts, traceable workflows, and human review are essential for reliable results, and presents a practical framework for evaluating an AI agent’s numerical accuracy, data understanding, tool use, document generation, and explainability.

    As large language models evolve from simple question-answering systems into agents that can plan tasks, call tools, and generate files, data analysis and financial research are adopting new workflows. In the past, users typically had to organize spreadsheets, write analysis scripts, create charts, and manually transfer the results into daily reports or research documents. Today, an AI agent can interpret a natural-language request, break it into subtasks, read data, perform calculations, retrieve relevant information, generate charts, and produce a structured document.

    However, completing a task does not mean that the result is fully reliable. In quantitative analysis, financial data processing, and percentile calculations, accuracy cannot be judged by the fluency of the generated text alone. The data source, calculation process, metric definitions, and final values must also be verified. When evaluating next-generation AI models, the key question should not simply be whether the answer sounds plausible. It should be whether the system can establish a workflow that is verifiable, reproducible, and traceable.

    1. AI Agents vs. Conventional Conversational Models

    A conventional conversational model primarily relies on its internal language reasoning capabilities to generate text from the surrounding context. This approach is efficient for general knowledge, conceptual explanations, and simple summaries. However, when a task involves large spreadsheets, multiple metrics, or complex charts, relying only on model memory and probabilistic inference introduces significant risks.

    An AI agent typically adds several stages:

    1. Task decomposition: It breaks down a request such as “analyze the data and generate a research report” into subtasks, including file reading, field identification, data cleaning, metric calculation, background research, chart generation, and document composition.
    2. Tool use: It calls a script execution environment, file parsing component, retrieval system, charting component, or document-generation tool as needed.
    3. Intermediate-result feedback: After each step, the agent reads the execution result and decides what to do next instead of producing the final answer in a single pass.
    4. Exception handling: If the file format is invalid, fields are missing, a calculation fails, or the retrieved information is insufficient, the agent can revise its plan.
    5. Result aggregation: It combines numerical results, explanations, charts, and source information into a daily report, research report, or other structured file.

    The central value of this approach is that it transforms the model from a text generator into a task coordinator. The model interprets requirements, writes execution logic, and organizes the output, while deterministic tools handle data loading, mathematical calculations, and file processing.

    2. Why Data Analysis Should Be Delegated to Scripts

    Large language models can encounter three major types of problems when handling precise numerical tasks.

    The first is replacing exact calculations with approximations. For example, when asked to calculate a percentile, a model may produce a plausible percentage based on the context instead of strictly applying data sorting, interpolation rules, and sample counts.

    The second is misalignment in structured information. When a table contains multiple dates, instruments, metrics, and conditions, the model may confuse one column with another or omit a filtering condition.

    The third is lack of process traceability. If the final output consists only of a sentence, it is difficult for users to determine whether the result came from an actual calculation over the source data or from an estimate generated from general patterns.

    A more reliable approach is to have the agent generate and execute a script. The script should be responsible for:

    • Reading the original spreadsheet or data file;
    • Checking field names, data types, and missing values;
    • Filtering and sorting records according to explicit rules;
    • Performing statistical calculations;
    • Outputting intermediate and final results;
    • Saving the data required for charts;
    • Preserving the execution log.

    Script execution provides determinism. When the input data, formulas, and parameters remain unchanged, the same result can generally be reproduced. Even when the final report is written by an AI model, key figures should be linked to script output rather than recalled or estimated by the model.

    3. Charts Should Be Generated from Real Data

    In financial analysis, charts are not merely decorative. Price movements, basis changes, inventory distributions, volatility ranges, and percentile positions may directly affect research conclusions.

    When charts are generated programmatically, every point, line, and bar is derived from specific data and calculations. Users can then verify whether:

    • The x-axis and y-axis correspond to the correct fields;
    • Dates are displayed in chronological order;
    • Units are consistent;
    • Missing values have been handled correctly;
    • Outliers have been hidden or altered;
    • Chart titles and legends match the underlying data;
    • The filtering conditions match the descriptions in the report.

    This is fundamentally different from generating an image that merely looks like a chart. Image-generation methods can create visual effects, but they should not be used for precise statistical analysis because line positions, labels, and proportions may not strictly reflect the original data.

    In practical prompts, users can specify that the agent should:

    • Automatically generate trend, distribution, or comparison charts;
    • Build charts from calculations over the actual data;
    • Label the data range, sample count, and units;
    • Add annotations to important points;
    • Combine charts and written content into a consistent research document;
    • Preserve data sources and calculation notes at the end of the report.

    This allows the agent to produce more than a report that simply contains charts. It can produce analytical material with audit value.

    4. A Complete Automated Research-Report Workflow

    A relatively reliable workflow can be divided into seven stages.

    1. Define the Inputs and Outputs

    At the beginning of the task, specify the input files, time range, analysis subjects, metric definitions, and required output format. For example, instead of saying only “analyze inventory and basis,” define the inventory field, the basis formula, the observation period, and the required chart types.

    2. Create a Task Checklist

    The agent should first generate a checklist such as:

    • Verify that the file can be read;
    • Confirm field names;
    • Count missing values;
    • Calculate current values and historical percentiles;
    • Retrieve relevant market information;
    • Generate trend and distribution charts;
    • Write the analysis and conclusions;
    • Produce the final report.

    A checklist reduces omissions and allows users to review execution progress during the process.

    3. Read and Validate the Data

    The agent should not begin calculating immediately after loading the data. It should first inspect the table structure, including the number of fields, data types, date range, duplicate records, and outliers. For important fields, it should output the minimum value, maximum value, sample count, and number of missing records.

    4. Perform Deterministic Calculations

    All operations involving sorting, averages, standard deviations, percentiles, year-over-year changes, and range statistics should be performed by a script. The report-generation module should explain the results, not recalculate them.

    5. Retrieve External Information

    If the report needs to include policy developments, industry news, or market events, the system should clearly distinguish between conclusions from the source data and viewpoints from external materials. Retrieval results should record the title, date, source summary, and relevance to the current analysis. Unverified information should not be inserted directly into the report.

    6. Generate Charts and Documents

    Charts should use consistent fonts, colors, units, and date formats. The report should include an executive summary, data notes, key metrics, charts, risk warnings, and conclusions. Any information that cannot be confirmed should be clearly marked for further verification.

    7. Review the Results

    Before delivering the final output, perform another review:

    • Do the figures in the text match the calculation results?
    • Do chart titles correspond to the actual data?
    • Are percentages and decimal places formatted consistently?
    • Is the time range consistent throughout the report?
    • Are there conclusions without supporting sources?
    • Has correlation been incorrectly described as causation?

    5. Why Percentile Calculations Often Differ

    Percentiles may appear simple, but several definitions are commonly used. Different calculation methods can vary in the following areas:

    • Whether the nearest-rank method or linear interpolation is used;
    • How small sample sizes are handled;
    • Whether missing values are included;
    • Whether records are first completed by calendar day;
    • Whether outliers are removed;
    • Whether the full historical sample or a fixed rolling window is used;
    • Whether the current value is included in the historical sample.

    Therefore, a report should not state only that “current inventory is at a certain historical percentile.” It should also specify the observation period, sample count, and calculation method. Otherwise, two systems may both produce reasonable results but still differ substantially because they use different definitions.

    For this type of metric, a suitable description is:

    Within the specified observation period, the current value’s positional percentile is calculated from valid observations. Missing records are excluded, and the percentile uses a predefined interpolation rule.

    The system should also preserve the original sorted values and calculation parameters so that the result can be reviewed if questions arise.

    6. How to Evaluate an AI Agent’s Real Capabilities

    Evaluation should not rely on a single demonstration. Instead, it should use a multidimensional testing framework.

    1. Numerical Accuracy

    Use datasets with known answers to verify averages, percentiles, change rates, rankings, and other results. For metrics with many decimal places, define strict error thresholds.

    2. Data Understanding

    Intentionally introduce field aliases, null values, duplicate rows, and different date formats. Observe whether the agent can identify these issues instead of generating conclusions immediately.

    3. Tool-Calling Capability

    Check whether the agent proactively uses scripts for complex calculations, can revise its logic after an execution error, and preserves intermediate results.

    4. Chart Reliability

    Verify the points, axes, labels, and units in the charts. The goal is to determine whether the visualization accurately represents the data, not merely whether it looks polished.

    5. Document Generation

    Review the report structure, layout, relationship between text and charts, and file completeness. Pay particular attention to truncated charts, overflowing text, and numerical formatting changes.

    6. Explainability and Traceability

    A capable agent should provide more than a conclusion. It should explain where the data came from, which algorithms were used, what steps were executed, and which items still require human confirmation.

    7. Practical Boundaries and Recommendations

    AI agents are well suited to repetitive tasks with relatively clear rules and multiple stages of coordination, such as daily report generation, spreadsheet cleaning, metric aggregation, and preliminary research analysis. However, they should not be treated as fully autonomous decision-making systems.

    In trading, investment, risk management, and externally published research, a human review stage should be retained. In particular, the following questions should not be answered solely by a model:

    • Is the data source authoritative?
    • Does the research conclusion contain a logical gap?
    • Is external news authentic and valid?
    • Does the chart conceal unusual volatility?
    • Is the result applicable to the current business context?
    • Could the conclusion be misunderstood as a deterministic forecast?

    Ultimately, the value of next-generation AI models is not simply that they “write more like humans.” Their greater value lies in translating natural-language requirements into executable workflows. A reliable solution should use a collaborative model in which the AI model handles planning and communication, scripts handle calculations, retrieval systems provide supporting evidence, and humans perform the final review.

    Only when data, code, charts, documents, and review procedures form a closed loop are AI agents suitable for quantitative analysis and financial research production. For users, the most important question is not whether a single answer is impressive. It is whether the system can complete tasks consistently, process data accurately, and provide a clear path for locating and correcting errors when they occur.