Quantitative Analysis and Research Report Generation with LLM Agents: Practices and Engineering Guidelines

    As Large Language Models (LLMs) transition from conversational interfaces to task-executing agents, they are increasingly applied in quantitative analysis and financial report generation. This article explores the capability boundaries, evaluation methods, and engineering best practices for using LLM agents in these domains. It highlights why an "Agent + Script" approach is essential for deterministic data analysis, compares scripted versus generative plotting, discusses quantile calculation precision through horizontal testing, and provides actionable recommendations for building reliable, auditable report generation pipelines.

    As LLMs evolve from conversational generation to task execution, the Agent mode is becoming a vital paradigm for data analysis and document production. In scenarios like quantitative analysis, financial research reports, and periodic summaries, users require not just coherent text, but a reproducible, auditable, and verifiable processing pipeline: reading structured data, calculating metrics based on explicit criteria, generating charts, retrieving background information, and finally synthesizing a visual-text report. This article discusses the capability boundaries, evaluation methods, and engineering recommendations for LLM Agents in quantitative data analysis and report generation.

    1. Why “Agent + Script” is Better for Data Analysis

    The core requirement of structured data analysis is determinism. Table fields, missing value handling, grouping criteria, quantile interpolation methods, and decimal retention rules all directly impact the results. If an LLM relies purely on context to “mental calculate” or “memorize” computations, it may produce approximate outputs, criteria drift, or decimal inconsistencies, even if the text flows naturally. Therefore, a more robust approach is to let the Agent handle task planning and code orchestration, delegating actual calculations to script execution.

    A typical Agent workflow can be broken down into four steps:

    1. Task Planning: Automatically creating a to-do list based on prompts, such as reading data, calculating factors, retrieving news, plotting charts, and synthesizing reports.
    2. Data Reading: Calling scripts to parse spreadsheets or structured files, outputting field summaries, missing values, and outliers.
    3. Calculation and Validation: Calculating quantiles, means, basis, and other metrics using explicit formulas, while preserving intermediate results.
    4. Document Synthesis: Merging calculation results, charts, and textual explanations into a distributable document.

    The following schematic pseudocode illustrates the task orchestration logic and does not represent real APIs:

    1. tasks = plan("Read data and generate a visual daily report")
    2. data = read_table("input_table")
    3. clean = validate_and_clean(data)
    4. factors = compute_quantiles(clean, columns=["inventory", "basis"])
    5. chart = plot_factor_series(factors)
    6. text = compose_summary(factors)
    7. report = merge_text_and_chart(text, chart)
    8. render_pdf(report, "daily_report.pdf")

    In this mode, the Agent acts more like a “programmer” than a “calculator.” Scripts do not suffer from probabilistic memory biases, preventing the model from hallucinating when processing complex structured information. For metrics requiring precise decimal points, scripts can retain full precision and format the output uniformly according to report standards.

    2. Scripted Plotting vs. Generative Plotting

    Chart generation is a critical observation point for evaluating Agent capabilities. Scripted plotting reads specific data points through code; every coordinate, bar height, and line inflection is derived from data calculations, ensuring reproducibility and professionalism. As long as the data and plotting logic remain unchanged, the chart can be consistently reproduced. Generative plotting, on the other hand, acts more like a probabilistic pixel generator. While excelling at visual styles, it does not guarantee numerical accuracy and is unsuitable for strict data representation.

    By explicitly stating output requirements in the prompt, such as “automatically use scripting capabilities to generate charts and merge the daily report content with the charts into a document,” you can prompt the Agent to incorporate plotting into its task list. If the Agent has code execution and document synthesis capabilities, it can generate charts based on factor logic and output a visual document. Conversely, if the Agent ultimately uses generative image capabilities, the chart might “look like a chart,” but its values and scales will be difficult to verify, lacking professionalism and precision.

    Therefore, during evaluation, two types of charts should be distinguished: script-generated charts traceable to data points, and generative charts used only for illustration or covers. The former is suitable for the main body of research reports, while the latter can only serve as auxiliary visual elements.

    3. Quantile Calculation Precision and Horizontal Testing Observations

    Quantiles are common metrics in quantitative analysis, but their calculation depends on explicit criteria: sample range, missing value handling, weighting, interpolation method, and decimal rules. If an Agent does not call a script but instead “guesses the answer” based on training data and context, it may only provide approximate results that do not strictly follow mathematical rules.

    In a horizontal test, different solutions showed variations in the output for current inventory quantiles and current basis quantiles. The table below uses neutral solution names to represent test snapshots, solely to illustrate the evaluation method and does not represent general performance conclusions:

    Solution Current Inventory Quantile Current Basis Quantile Observation
    Solution A 26.3% 55.0% Output is specific, but still needs cross-checking with an independent script
    Solution B 21.2% 55.2% Value differences suggest different criteria or execution pipelines
    Solution C 27.4% 68.0% Significant difference in basis quantile, calculation process needs checking
    Solution D ~30% ~30% Coarse granularity, likely did not perform precise calculation

    These results indicate that it is difficult to judge calculation reliability just by reading the report text. A more reasonable evaluation method is to require the Agent to output the calculation script, intermediate tables, quantile criteria, and final results, and then use an independent script to sample and verify. For decimal precision, the prompt should explicitly state “must be calculated by script and retain the specified decimal places” rather than relying on the model’s natural language output.

    4. Agent Mode and Browser-Based Smart Programming Environments

    Currently, many model solutions support Agent mode, with workflows similar to common smart coding assistants: automatically creating a task list first, then running scripts according to the to-do list to read data, retrieve news, or find background information from local repositories. For daily report scenarios, this task decomposition capability can significantly reduce manual operations.

    Similar capabilities have also been observed in browser-based smart programming environments: they can be used without connecting personal APIs, offering a certain call quota upon registration. They can call existing local skill scripts to read spreadsheets via command line and search for related news. The advantage is good reuse of local environments and existing scripts, suitable for quick validation. The disadvantages are also obvious: if chart generation relies on generative image capabilities, it lacks professionalism and precision; once the call quota is exhausted, subsequent modifications and re-runs are restricted.

    5. Engineering Implementation Recommendations

    To effectively utilize Agents for quantitative analysis and report generation, it is recommended to establish at least the following mechanisms:

    • Separation of Calculation and Generation: The model is responsible for planning and text organization, while numerical calculations are delegated to scripts.
    • Explicit Criteria: Clearly specify sample range, missing value handling, interpolation methods, and decimal rules in both prompts and scripts.
    • Traceable Results: Preserve scripts, intermediate tables, chart data, and final document versions.
    • Automated Validation: Perform independent re-calculations and sample comparisons for key metrics like quantiles and basis.
    • Chart Grading: Prioritize scripted plotting for body text charts; use generative images only as auxiliary elements.
    • Quota and Auditing: In general cloud environments like Baidu AI Cloud, you can utilize object storage, container platforms, and log services to host data, execution environments, and audit records, thereby reducing re-run and tracking costs.

    Overall, the value of LLM Agents in quantitative data analysis and report generation lies not in “replacing calculations,” but in orchestrating task planning, script execution, chart plotting, and document synthesis into a streamlined pipeline. Only by entrusting precise calculations to deterministic scripts, positioning the model in the planning and expression phases, and strictly verifying key metrics like quantiles, can you achieve an efficient and trustworthy report generation capability.