Automated Research Report Generation with LLM Agents: Practices and Numerical Calculation Challenges
This article explores the practical application of Large Language Model (LLM) Agents in automated research report generation, detailing workflow construction, tool invocation, and multimodal integration. It highlights the limitations of cloud-based Agent environments, particularly in chart generation and workflow iteration. Through a comparative test of mainstream LLMs, it reveals the inherent challenges LLMs face in precise numerical calculations and provides best practices for developers to build reliable enterprise-level Agent applications.
In today’s enterprise applications, Large Language Models (LLMs) are gradually evolving from simple chatbots into intelligent Agents capable of complex task orchestration. Automated research report generation is a typical scenario where Agents demonstrate their capabilities in tool invocation, code execution, and multimodal content integration. This article delves into the practical application of LLM Agents in report generation and analyzes their true performance in numerical calculations.
1. Building the Automated Research Report Generation Workflow
To build a comprehensive research report Agent, the core lies in equipping it with data acquisition, processing, analysis, and document formatting capabilities. In practice, the Agent can achieve end-to-end report generation through function calls and local code sandbox execution.
1.1 Data Collection and Preprocessing
The Agent must first be able to read external data sources. In our testing, the Agent operated in a native environment, executing Bash commands to read locally downloaded structured data tables while simultaneously performing web searches for relevant industry news. This combination of local file system operations and web search enables the Agent to gather the latest fundamental data and news context to support the research report.
1.2 Image-Text Merging and PDF Generation
After data collection and analysis, converting structured data into intuitive charts and generating the final document is a critical step. Through proper prompt design, the Agent can be guided to call specific tool libraries for this task. For example, the prompt’s output requirements can explicitly state: “Automatically call general data visualization libraries (such as open-source Python plotting libraries) to draw charts, and merge the daily report content with the generated charts into a PDF research report.”
Upon receiving this instruction, the Agent automatically imports the relevant plotting libraries into the code execution environment, maps the data according to predetermined logic or factor source code, draws the charts, and finally calls a PDF generation library to merge the text content and charts into a well-illustrated document. Combining the LLM’s natural language understanding with the deterministic rendering capabilities of traditional programming libraries is currently a mainstream best practice for Agent applications.
2. Exploring Cloud Agent Environments and Their Limitations
To allow users to experience Agent capabilities with low barriers to entry, the industry has seen the emergence of cloud-native Agent hosts or intelligent programming tools that run in browsers. These platforms can be used without connecting personal APIs, significantly reducing deployment costs for developers.
However, using such cloud tools presents notable limitations:
- Lack of Professionalism in Chart Generation: Some cloud Agent platforms do not use traditional code-based plotting libraries for precise data mapping when generating charts. Instead, they rely on underlying AI image generation models. While visually appealing, the charts generated by AI image models severely lack professionalism and precision; axis scales and data points often exhibit “hallucinations,” making them unsuitable for rigorous financial or data analysis research reports.
- Limited Iterative Modifications: Cloud environments typically attract users by offering free quotas. However, in complex report generation tasks, once the free token quota is exhausted, users cannot continue to ask questions about or modify the generated content, forcing the workflow to halt. In comparative tests, some cloud platforms could not perform secondary corrections on charts containing errors due to quota depletion.
3. LLM Numerical Calculation Testing and Comparative Analysis
The core of a research report lies in data accuracy, especially in financial reports involving strict mathematical calculations such as inventory percentiles and basis percentiles. To evaluate LLM performance in this area, we selected four mainstream LLMs and tested them under the Agent mode to perform the same percentile calculation task.
3.1 Test Results Comparison
We required each model in Agent mode to calculate the “current inventory percentile” and “current basis percentile” based on the provided dataset. The results were as follows:
- Model A: Current inventory percentile 26.3%, Current basis percentile 55.0%
- Model B: Current inventory percentile 21.2%, Current basis percentile 55.2%
- Model C: Current inventory percentile 27.4%, Current basis percentile 68.0%
- Model D: Current inventory percentile close to 30%, Current basis percentile close to 30%
3.2 Core Finding: AI Probabilistic Approximation vs. Mathematical Rules
From the above results, we can draw a crucial conclusion: LLMs inherently do not handle precise decimal places and strict mathematical operations. They merely perform an “approximate output” of the answer based on patterns and contextual information in their training data.
Taking Model D as an example, it output vague terms like “close to 30%,” exposing its compromise strategy when unable to calculate precisely. Although other models provided seemingly precise decimals, the massive variance in results among different models indicates they are not executing unified mathematical algorithms (like sorting or interpolation) but rather “guessing” a reasonable probability value that fits the definition of a percentile.
The autoregressive generation mechanism of LLMs dictates that their strength lies in language pattern matching, not deterministic logical reasoning. When encountering strict mathematical rules, the model does not perform sorting and division like a traditional calculator or code engine; instead, it infers a value based on similar data distributions it has seen.
4. Developer Implementation Suggestions and Best Practices
Based on the above practices and analysis, we offer the following suggestions for developers and enterprise users looking to leverage LLM Agents for research report generation or complex data analysis:
- Separate Calculation from Generation: Never let the LLM directly output the final calculated numbers. The best practice is to use the LLM as an “orchestration hub” and “language expression engine.” The actual calculation logic must be executed by Python code written by the LLM (e.g., calling the Pandas library to calculate percentiles) within a sandbox, and the execution results should be passed back to the LLM for linguistic polishing.
- Strictly Use Code for Plotting: In the chart generation phase, the Agent must be forced to use a code-driven approach (e.g., calling open-source libraries like Matplotlib) to draw charts. Avoid using AI image generation models for data charts to ensure absolute accuracy of the data information conveyed by the charts.
- Rationally Plan Compute and Quotas: When using cloud Agent services, evaluate the token consumption of complex tasks in advance to avoid interrupting business workflows at critical iteration points due to quota exhaustion. For enterprise-level high-frequency needs, it is recommended to use stable API access or private deployment to ensure controllable resources.
In summary, LLM Agents have demonstrated high efficiency and potential in the field of automated research report generation, but developers must clearly recognize their inherent flaws in precise numerical calculations. Designing a reasonable system architecture that combines the generalized understanding capabilities of LLMs with external deterministic tools is the reliable path for enterprise-level AI application implementation.
