Using LLM Agents for Data Analysis: A Code-Execution-Based Practical Guide
This article explains why you should not let a large language model perform numerical computation directly, and makes the case for a code-execution-based approach in which the model handles intent, planning, and code writing while a deterministic runtime handles all computation and visualization. It walks through a five-stage agent workflow, deterministic rendering, an automated chart-to-document delivery pipeline, and covers reliability, security, cost, evaluation metrics, and common pitfalls, closing with a practical implementation checklist for building reproducible, auditable data-analysis agents.
1. Starting Point: Why You Shouldn’t Delegate Numerical Computation to Probabilistic Generation
Handing a table and a metrics-definition document to an LLM and asking it to directly answer questions like “what historical percentile does the current metric sit at?” or “which interval has the basis converged to?” is what many people first try when automating analysis. In practice, though, you keep running into the same class of problem: the same data and the same prompt produce unstable numbers across repeated queries, and some results show an obvious “rounding” signature, where the last decimal digits look filled in rather than computed.
This isn’t an accident of any single call — it follows from how generation works. An LLM’s output is fundamentally token-by-token conditional probability sampling; it learns statistical patterns in its training corpus, not arithmetic and statistical rules. A string of decimals often appears in the corpus in a patterned form, and the further out you go, the less reliable signal the model has to draw on, so it simply continues writing along the “string that most looks like an answer.” The result is that errors happen silently: no exception is raised, the format still looks tidy, but the numbers can’t be trusted.
Worse, caliber differences get mistaken for model differences. Percentiles alone have multiple definitions (linear interpolation, nearest rank, whether to exclude missing and suspended samples, how long the rolling window is), and basis involves aligning the timestamps of spot and contract prices, whether adjustment is applied, and so on. When several models return inconsistent results, it’s hard to tell whether it’s a model capability problem or whether they’re each guessing at different calculation conventions. To tell these apart, you first have to separate computation from generation.
2. Core Idea: Turn a Probabilistic Problem into a Deterministic One
The correct engineering approach is to change the division of responsibility: the model handles intent understanding, task decomposition, writing code, and writing conclusions; all numerical computation and chart rendering is done by a deterministic runtime.
| Dimension | Model generates results directly | Code-execution-based analysis |
|---|---|---|
| Value source | Parameterized probability sampling | Deterministic computation from data and code |
| Reproducibility | Affected by sampling randomness | Reproducible given a data snapshot and script |
| Auditability | Black box, hard to locate error sources | Scripts and execution logs verifiable line by line |
| Failure mode | Silent errors, no exceptions | Out-of-bounds, missing columns caught by assertions |
| Charts | Probabilistic pixel generation | Precisely rendered from data coordinates |
| Iterative changes | Usually requires full regeneration | Re-run after changing parameters or config |
This shift brings more than “more accurate numbers.” More importantly, the analysis process becomes a deliverable asset: scripts can be put under version control, reused, regression-tested, and re-run by whoever takes over next.
3. The Five Key Stages of an Agent Workflow
An agent with tool-calling ability fits this kind of task because it maps naturally onto a “plan–execute–verify–summarize” loop. A workable process usually includes the following stages.
Task decomposition. First produce a structured to-do list instead of writing code directly. The list should explicitly state: what data is needed, what caliber to compute under, what charts are needed, and what format the final deliverable should take. The value of an explicit plan is that deviations surface early, rather than after you’ve generated 20 charts and only then discover the caliber was wrong.
Information retrieval. This includes reading local files, searching a local knowledge base or document library, and searching the web. The key constraint here is traceable data sources: retrieval results should record the source and retrieval time, write it into the report’s citation location, and avoid conclusions that can’t be traced back.
Sandbox code execution. Code runs in an isolated environment, and the model gets back execution results rather than its own guesses. Here you must assume “the code the model writes will have errors,” so retries, exception handling, and missing-dependency handling are standard configuration, not extra features.
Deterministic visualization. Rendered by a plotting library from computation results, detailed in the next section.
Deliverable assembly. Conclusions, data notes, charts, and appendix scripts are assembled into a document by a fixed template and output.
Illustrative pseudocode (syntax close to a general scripting language, only expressing the process structure):
def run_analysis(task, tools):plan = planner(task) # 1. output a structured to-do listartifacts = {}for step in plan.steps:if step.type == "retrieve":artifacts[step.key] = tools.search_local_kb(step.query)elif step.type == "compute":code = coder(step.spec) # 2. the model only writes coderaw = tools.sandbox.run(code) # 3. the runtime executes itvalidate(raw, step.expected_schema, step.ranges)artifacts[step.key] = rawelif step.type == "chart":tools.sandbox.run(render_spec(step.key, artifacts))return assemble(plan, artifacts) # 5. assemble deliverables by fixed template
4. Visualization: Why Deterministic Rendering Can’t Be Replaced
Generative image models are a common but wrong choice here. They can draw “professional-looking charts,” but the tick marks on the axes, the position of data points, and the legend’s correspondence all come from pixel-level approximation and aren’t guaranteed to match the underlying values; and once a parameter needs adjusting, there’s no way to fix it via “change the config and re-run.”
A plotting library offers exactly the opposite: every pixel position is computed from the data through a coordinate transformation, so there is a strict functional relationship between the chart and the data.
# Illustrative pseudocode: data-driven deterministic plottingvalues = compute_percentile(series, window=252, method="linear")fig = plot.figure(title="Historical percentile of the metric", figsize=(10, 5))ax = fig.axes()ax.plot(dates, values, label="percentile")ax.axhline(threshold, linestyle="--", label="threshold")ax.set_ylim(0, 1) # explicit boundary constraint; outliers stand out at a glanceax.legend(); ax.grid(True)fig.save("chart_percentile.png", dpi=200)
A few engineering details are easy to overlook: when non-Latin fonts and negative-sign display are needed, they must be configured uniformly during environment initialization, or you’ll get garbled boxes; chart outputs should carry a clear filename and generation timestamp so they can be matched to document references; and the same class of charts should reuse the same style configuration to avoid style drift during batch generation.
5. From Charts to Documents: An Automated Delivery Pipeline
Making the delivery requirements explicit in the prompt is the lowest-cost step to connect the whole chain. For example, ask the agent: “Automatically call the plotting capability to generate figures, plot according to the metric logic defined in the script, and merge the text conclusions with the figures into a PDF daily report.”
To truly achieve figure–text consistency, you need to bind “chart–text–data” to the same intermediate result: the computation script first produces structured results and writes them to disk, the plotting script only reads that result, and the text conclusions are also rendered from the same result rather than dictated separately by the model. That way, even if the model’s wording is off, the numbers and the graphics won’t contradict each other.
pipeline:1. ingest read the data snapshot, record the file fingerprint and timestamp2. compute compute under the fixed caliber, output structured results3. render generate all charts driven by the results4. narrate generate text conclusions based on the results5. assemble merge into a document, with computation scripts and data notes attached6. verify automatically check that numbers in the document match the result files
Step 6 is missing from many solutions, yet is often the cheapest and highest-payoff: since the numbers already exist in structured form, a script can compare whether the numbers appearing in the document body match, automatically intercepting problems like “the model misremembered a number.”
6. Reliability, Security, and Cost
Layered validation. Validate fields and types at the input layer, validate value ranges at the computation layer (e.g., percentiles must fall within the defined interval, ratio-type metrics must not exceed bounds), and validate document structure and numerical consistency at the output layer. When validation fails, the system should abort and feed the error back to the model to retry, rather than letting it keep writing.
Data and caliber versioning. The prerequisite for reproducible numbers is a fixed data snapshot, a fixed code version, and explicitly declared caliber definitions. Miss any one of the three and what you reproduce is just “another result.”
Sandbox and permissions. The code execution environment should default to no internet access or only whitelisted domains, filesystem read/write should be confined to the task directory, dependency versions should be locked rather than pulling the latest each time, and sensitive fields should be desensitized before entering the model context.
Resources and cost. Token consumption for analysis agents concentrates on reading large tables, multiple rounds of debugging, and long-document generation. Practical ways to rein this in include: first running statistics summaries with a script and having the model consume the summary rather than the raw table, limiting the maximum retries per task, caching retrieval and computation results, and turning high-frequency fixed analyses into reusable script templates.
Hosting. When running this kind of workload in the cloud, you can typically use a container platform to run the isolated execution environment, object storage to archive data snapshots and outputs, and logging and monitoring services with alerting to track task success rate, execution time, and failure reasons, forming an operable long-chain task system. Platforms like Baidu AI Cloud provide exactly this kind of general-purpose capability; specific selection should be evaluated against task-duration distribution, concurrency scale, and data-compliance requirements.
7. How to Evaluate: Don’t Use “Looks Professional” as the Acceptance Standard
To prove that code-execution-based analysis beats direct generation, you need a reproducible benchmark set: pick a fixed data snapshot and fixed caliber, first produce standard answers with a human-verified script, then have the agent run end-to-end and compare field by field.
At minimum, cover the following metrics:
- Numerical error: the absolute/relative deviation of target fields from the baseline, with particular attention to the last decimal digits
- Conclusion consistency: whether the agent’s directional judgment matches the baseline computation
- Figure–text consistency: whether the data expressed by the charts matches the result files
- Format and completeness compliance rate: whether all required sections and charts are output per the template
- Reproducibility: whether repeating N times on the same input yields completely identical results
Among these, “reproducibility” is the most easily skipped yet the most critical for analysis scenarios. An analysis system whose results differ every time cannot bear the responsibility of supporting decisions, even if any single answer is correct.
8. Common Pitfalls and an Implementation Checklist
Common pitfalls:
- Treating decimal places as a capability metric. When the last decimals have no basis, precision is just format, not accuracy.
- Replacing data-driven plotting with generative image generation, sacrificing the consistency between numbers and graphics.
- Comparing different results without defining a caliber, leading to caliber differences being misjudged as model differences.
- Only looking at the final document’s aesthetics without doing numerical comparison and consistency checks.
- Having the model consume raw data from large tables directly, which both wastes context and raises the error rate.
Implementation checklist:
- Write the caliber definitions into a document before starting automation.
- Fix the data snapshot and code version as the basis for reproducibility.
- Have the model write code, the runtime execute, and the plotting library render — separate the responsibilities.
- Use assertions to turn “silent errors” into “explicit exceptions.”
- Write intermediate results to disk in structured form, and generate both the text and the charts from the same result.
- Add an automatic comparison step between the document numbers and the result files.
- Build a benchmark set and accept based on reproducibility metrics rather than appearance.
- Make sandboxing, whitelisting, dependency locking, and desensitization defaults, not options.
Conclusion: The ceiling of an analysis agent’s capability doesn’t depend on whether the model can “say” the right answer, but on whether the system can constrain computation to a deterministic path. Put the model in the position of planning, orchestration, and expression; hand computation and rendering to code; and insert validation and consistency comparison in between. Only then can automated analysis move from a demo effect to a production capability usable for daily decision-making.
