Agent Mode in Practice: Letting AI Use Code for Rigorous Data Analysis

    This article explains why LLMs should compute data results by running code instead of estimating numbers in their output. It covers the typical agent workflow of planning, writing and running code, retrieving information, and producing reports. It also compares code-based charts with image-model charts and recommends checking key figures against script output.

    1. Background: From Conversational Q&A to Agent Mode

    Previously, data tasks with LLMs usually meant pasting data into a chat window and asking the model to work out the answer “in its head.” This approach works for simple cases. However, once a task involves structured tables, multi-step calculations, or batch processing, the probabilistic nature of model output becomes a problem.

    Mainstream LLM platforms now commonly offer Agent mode, also called tool-calling mode. The key change is that the model no longer answers directly. Instead, it plays the role of a programmer. It plans the task and writes code, and the runtime environment executes that code and returns the results. This “compute first, then answer” approach is a key step in moving LLMs from language generators to task executors.

    2. Why Data Analysis Requires a Code Execution Path

    2.1 Probabilistic Generation vs. Deterministic Computation

    An LLM is fundamentally a probabilistic sequence predictor. When asked to output a quantile, mean, or regression coefficient directly, it guesses a plausible number based on its training data and context. It does not strictly apply mathematical rules. This is unacceptable in precision-sensitive fields such as finance and statistics.

    The code execution path works differently. A deterministic program performs each calculation, and every data point in a chart comes from real numerical computation rather than probabilistic pixel generation. The accuracy gap between the two approaches is fundamental, and prompt tuning cannot close it.

    2.2 Structural Methods for Reducing Hallucinations

    When handling complex structured information, such as multi-level-header Excel files or nested JSON, models that read the data directly are prone to memory errors and hallucinations. If the agent generates a parsing script and that script reads the actual file contents, the script works as a fact-checking layer. The model sees the data the program returns, not its own imagined version of the data.

    3. Typical Agent Workflow

    When a code-based agent receives a data analysis task, it usually follows this process:

    1. Task planning: Automatically create a task list that breaks a vague request into executable steps.
    2. Code generation and execution: Write and run scripts step by step to read, clean, and calculate the data.
    3. Information retrieval: Search public information online as needed, or retrieve relevant material from a local knowledge base.
    4. Visualization and output: Call Python visualization libraries to draw charts, then combine the analysis conclusions and charts into a document such as a PDF.

    Below is an example of agent-generated code (pseudocode):

    1. import pandas as pd
    2. # Read local tabular data
    3. df = pd.read_excel("inventory.xlsx")
    4. # Compute quantiles deterministically
    5. stock_quantile = df["inventory"].rank(pct=True).iloc[0]
    6. basis_quantile = df["basis"].rank(pct=True).iloc[0]
    7. print(f"Inventory quantile: {stock_quantile:.1%}")
    8. print(f"Basis quantile: {basis_quantile:.1%}")

    Adding explicit instructions to the prompt, such as “automatically call a visualization library to draw charts, and combine the text and charts into a PDF research report,” lets the agent deliver the output end to end, from raw data to finished product.

    4. Comparing Two Visualization Methods

    In practice, there are two common ways to generate charts, and their results differ significantly:

    Dimension Code-based charting Image model generation
    Data source Points plotted from real computed results Probabilistic pixel generation
    Numerical accuracy Strictly consistent with the data Coordinates and trends may be distorted
    Reproducibility Scripts can be rerun and audited Output differs each time
    Suitable scenarios Financial research reports, statistical analysis Concept illustrations, creative visuals

    For report scenarios that require high rigor, ask the agent to chart with code first. Image models are better suited as decorative supplements, such as cover images or illustrations.

    5. Accuracy Pitfalls: Don’t Trust a Model’s Mental Arithmetic

    A typical pattern appears in practice. When different models are asked to estimate the same set of quantiles without running code, their approximate values often differ from each other, and none match the script’s actual results. The reason is simple. Models do not perform precise decimal arithmetic. They produce only probabilistic approximations based on training data and context.

    This leads to two engineering recommendations:

    • Keep conclusions traceable: Every numerical conclusion should trace back to executable code, not to the model’s direct statement.
    • Cross-check key metrics: Compute a baseline value with code first, then check whether the model’s statement matches it. When they differ, the code result takes precedence.

    6. Practical Recommendations and Summary

    A reliable AI data analysis pipeline should follow these principles:

    1. Prefer tool-calling mode with code execution, so the model works as a programmer.
    2. State the deliverables in the prompt: Specify the data source, calculation method, charting tool, and output format (such as PDF).
    3. Require a structured task list, so reviewers can check the basis for each step.
    4. Verify against code: Treat the model’s natural-language conclusions as summaries, not as facts.

    The real value of Agent mode is that it makes the model compute correctly. Hand deterministic calculation to code, and hand planning and explanation to the model. This split produces data analysis workflows that are both efficient and trustworthy.