Using Code Execution for Agent-Driven Data Analysis and Research Report Generation

    This article explains how AI agents can use deterministic code execution to handle data analysis and research report generation. It covers the rationale for delegating calculations to scripts, the typical six-stage workflow, key design practices for data validation, percentile calculation, visualization, and report assembly, as well as prompt design, quality control, and implementation recommendations. Readers will learn how to improve reproducibility, auditability, and numerical consistency in agent-generated analytical documents.

    1. Why Data Analysis Should Be Delegated to Code Execution

    In scenarios such as research reports, business analysis, and operational reviews, agents are often asked to read spreadsheets, calculate metrics, generate charts, and produce documents. Language models are good at understanding requirements, organizing language, and planning steps, but their output is probabilistically generated text. For calculations that must strictly follow mathematical definitions—such as decimal places, percentiles, year-over-year changes, and period-over-period changes—models may provide approximate values or even guess based on experience when context is incomplete.

    A more reliable approach is to delegate calculations to deterministic programs. With the same input and the same code, every run produces the same result, and each step can be checked line by line. The model’s role then shifts to decomposing tasks, writing or invoking calculation scripts, and explaining the results, while the numbers themselves must come from program output.

    This division of labor provides two direct benefits. First, results are reproducible: anyone with the same data and scripts can obtain consistent conclusions. Second, results are auditable: every value can be traced back to a specific calculation step.

    2. Typical Workflow for Agent Data Analysis

    A complete analysis task usually consists of six stages:

    1. Task planning: The agent first breaks the requirement into a checklist, such as reading data, calculating metrics, retrieving information, drawing charts, and generating documents.
    2. Data reading: Scripts parse spreadsheet or database export files and check field types, missing values, and time ranges.
    3. Information retrieval: The agent searches the web or queries a local information repository to collect background material related to metric changes and records the sources.
    4. Metric calculation: Statistical calculations are executed according to a unified definition, and intermediate result tables are output.
    5. Chart generation: Data visualization tools are used to draw charts based on calculation results. Each coordinate point in the chart corresponds to a specific data row.
    6. Report assembly: Text analysis, numerical tables, and charts are merged according to a template and exported as a fixed-format document such as PDF.

    Structured data should be passed between stages to avoid having the model restate numbers in natural language. For example, a calculation script outputs a JSON file, a charting script reads that file, and the report template reads the same file. In this way, numbers are not altered during multiple transfers.

    3. Design Considerations for Key Stages

    3.1 Data Reading and Structural Validation

    The reading stage is the foundation for all subsequent calculations. It is advisable to explicitly perform the following checks in the script:

    • Whether field names match expectations;
    • Whether numeric columns can be correctly converted to numeric types; values that cannot be converted should be recorded and reported rather than silently ignored;
    • Whether there are gaps in time series and whether dates are sorted in ascending order;
    • Whether the sample size meets statistical needs; if the sample is too small, this should be noted in the report.

    If validation fails, the workflow should pause and return an error message rather than continuing to generate the report.

    3.2 Metric Calculation: Historical Percentile Example

    Percentiles are often used to determine where a current value sits within a historical series. The definition should be unified at the beginning of the project; otherwise, the same data may produce different results. Common definitions include:

    • Empirical percentile: The ranking proportion of the current value within historical samples, i.e., the number of samples less than or equal to the current value divided by the total number of samples.
    • Interpolated percentile: Linear interpolation between adjacent samples, producing a smoother result but depending on the specific interpolation rule.

    Whichever definition is chosen, the report should state the methodology, window length, and sample range. The logic can be illustrated as follows:

    1. Input: history (time-ascending), current value
    2. Steps:
    3. 1. Filter nulls and check whether the valid sample count n is at least the configured minimum
    4. 2. Count samples satisfying x <= current as k
    5. 3. percentile = k / n, rounded to two decimals by a fixed rule
    6. 4. Write n, k, window start/end dates, and the result to the result file

    This logic should be strictly executed by the program. Floating-point rounding rules should also be fixed in code, for example by consistently using the same rounding method to avoid inconsistent decimal places across stages.

    3.3 Visualization: Binding Charts to Data

    The value of a chart lies in faithfully presenting data. In practice, the following principles should be followed:

    • The charting script should only read result files generated during the calculation stage and should not accept values provided directly by the model;
    • Each chart should include a data source note, including the time range and metric definition;
    • Axes, units, and legends should use a unified template to ensure consistent styling across multiple generations;
    • Images should be exported at a higher resolution and embedded in the final document.

    If the model needs to interpret a chart, the interpretation should be based on the data table corresponding to the chart, and the source of specific values should be noted.

    3.4 Report Assembly and Format Output

    Report assembly can be template-driven. Placeholders are reserved in the template, such as metric tables, chart positions, and analysis paragraphs. The script fills calculation results into the corresponding positions and then exports to PDF.

    The task prompt can explicitly specify the output structure, for example:

    1. Output requirements:
    2. 1. Present the core metric table first; all values must reference the calculation result file.
    3. 2. Add one interpretation paragraph after each chart, describing only changes directly visible in the chart.
    4. 3. List all data sources and calculation script versions at the end.
    5. 4. Do not add numbers that do not exist in the calculation results.

    Such constraints can significantly reduce deviations caused by the model’s free-form generation.

    4. Prompt Design Principles

    Prompts define the working boundaries of the agent. Effective prompts usually contain four types of information:

    • Task objective: Which question is being analyzed, and for which audience;
    • Data definitions: Metric definitions, time range, and data sources;
    • Execution constraints: Which calculations must be completed by scripts, and which stages may allow retrieval or reasoning;
    • Output format: Section structure, number of charts, and file type.

    The more specific the prompt, the more controllable the agent’s behavior. For example, instead of “analyze inventory conditions,” rewrite it as “calculate the empirical percentile of the current value using inventory data from the most recent 250 trading days, generate a comparison chart, and output a PDF report.” The execution path then becomes much clearer.

    5. Quality Control and Risks

    5.1 Information Hallucination

    Retrieval results should retain original links or document identifiers. Every piece of external information in the report should be traceable to a source, and content that cannot be verified should be marked as pending confirmation.

    5.2 Numerical Consistency

    When the same metric appears in the body text, tables, and charts, it should be rendered from the same variable. After report generation, the numbers in the document can be automatically compared with the result file, and publication should be terminated if inconsistencies are found.

    5.3 Data Security

    When agents read local files, they should follow the principle of least privilege and only expose directories required for analysis. Tasks involving sensitive data should be executed in controlled environments to avoid transmitting raw data to unrelated external services.

    5.4 Runtime Stability

    Script execution should have timeouts and resource limits, and logs should be retained on failure. Intermediate artifacts should be archived by date for troubleshooting and traceability.

    6. Implementation Recommendations

    1. Validate the workflow with a small sample first, confirm that the input and output of each step meet expectations, and then expand to full data.
    2. Include calculation scripts in version control and note the script version in reports to ensure reproducibility.
    3. Write test cases for key metrics and use known answers to verify whether the calculation logic is correct.
    4. For documents involving high-impact conclusions such as investment judgments or business decisions, retain a manual review step.