# Agent Data Analysis in Practice: Using Code Execution to Beat LLM Hallucinations
Large language models are more reliable in data analysis when they write and run code instead of answering from memory, so every number can be traced to a reproducible, auditable script. This article explains the Agent data analysis pattern and why code execution beats direct answers. It also walks through the standard workflow, how to choose a visualization backend, the remaining precision limits of Agents, how to run model-generated code safely, and practical tips for building reliable data intelligence applications.
What Is Agent Data Analysis: From Answering Questions to Computing Them
As large language models (LLMs) move into data analysis, the industry has converged on a more reliable approach. Instead of having the model recall answers from its training data, let it act as a programmer: plan the task, write code, actually run it to get results, and then keep reasoning from those results. This Agent pattern is widely adopted by mainstream coding assistants today, and it is what this article refers to as Agent data analysis.
The shift is about more than writing a few extra lines of code. In conversational mode, a model’s answers about numbers and statistics are essentially approximate reproductions of its training data: probabilistic language generation. In Agent mode, every number in the output can be traced to executable, reproducible, auditable code. The first approach guesses a plausible-looking answer; the second computes a real one. Their reliability is not in the same league.
Why Code Generation Is More Reliable Than Direct Answers
When handling complex structured data (joining multiple tables, nested JSON, wide-to-long conversions), letting the model “do the math in its head” easily produces hallucinations:
- mismatched column names;
- counts reported as sums;
- absolute values reported as percentages.
What makes these errors dangerous is how hard they are to notice. If the model instead writes a script to parse the data, every step becomes readable, editable, and re-runnable code. Deterministic code brings at least three benefits:
- Reproducibility: The same input and script always produce the same output, regardless of sampling randomness.
- Auditability: Anyone can open the script and check the logic. When something goes wrong, the bug can be traced to a specific line.
- Scalability: Scripts keep working when data grows from thousands to millions of rows, without running into the model’s context window limit.
The reference code below includes some robustness handling:
import csvdef load_table(path):# Stream rows with a generator so data isn't fully loaded into memory; handles million-row fileswith open(path, newline='', encoding='utf-8-sig') as f: # Compatible with BOM-prefixed files exported from Excelyield from csv.DictReader(f)def quantile(values, q):if not (0 <= q <= 1):raise ValueError("q must be in the [0, 1] range")cleaned = []for v in values:try:cleaned.append(float(v))except (TypeError, ValueError): # Skip empty and non-numeric values so they don't distort the statisticscontinues = sorted(cleaned)if not s:return Nonepos = (len(s) - 1) * q # Matches NumPy's default linear interpolationlo = int(pos)hi = min(lo + 1, len(s) - 1)return s[lo] + (s[hi] - s[lo]) * (pos - lo)rows = load_table("data.csv")print("Median (P50):", quantile((r["value"] for r in rows), 0.5))
A simple experiment shows the difference. Ask the model directly, “What is the P50 of current inventory?” and it will likely return a plausible-looking decimal without explaining where it came from. Write a script and run the quantile() function above instead, and you get a definite number. You can also print the row count and empty-value rate for cross-checking.
Two errors come up often during debugging:
KeyError: Usually caused by a wrong column name, or by spaces or BOM characters in the header. Check the columns withreader.fieldnames.ValueError: Usually caused by a quantile outside the valid range, or by a column that cannot be converted to float. Add assertions and failure retries during data cleaning.
The Standard Workflow for AI Agents in Data Analysis
A mature Agent works much like a professional coding assistant. Its workflow can usually be broken down into four steps:
- Task decomposition: Break a request such as “generate a quantitative analysis report” into subtasks, for example: read data → clean and validate → compute metrics → look up supplementary information online → generate charts → compile the report.
- Step-by-step execution: Write and run code for each item in the checklist. The Agent reads local or database tables and, when needed, searches the web or matches material from a local knowledge base.
- Result validation: Sanity-check intermediate results (magnitude, sign, empty-value rate), roll back and fix any anomalies, and recompute key metrics a second time.
- Deliverable assembly: Combine the computed results, charts, and narrative text into a structured document.
Agents usually run end to end when three conditions hold. First, you state the constraints clearly in the request, for example, “Automatically generate charts and merge the text and figures into a single PDF research report.” Second, the requirements are unambiguous. Third, the tools are available and the data is well-structured.
The real differentiator is not whether an Agent can produce charts. It is whether every point on a chart was actually computed from data.
Agents are not failure-proof, though:
- Dirty data can lead to incomplete cleaning.
- Ambiguous business definitions (such as inconsistent definitions of “active user”) can introduce systematic bias.
- On a rerun, the model may rewrite an already-verified, correct script.
These steps still need humans to define the metrics and sign off on the results.
Automated Visualization and Report Generation: Making Results Clear
Visualization is the final step in data analysis, and the choice of plotting “backend” directly affects credibility:
| Aspect | Code-driven plotting | Generative image plotting |
|---|---|---|
| Data source | Real data from calculation scripts | Pixels generated probabilistically |
| Reproducibility | Same input and script produce the same output | Results may differ each time |
| Auditability | Coordinates and axes can be traced and checked | Hard to map to real data |
| Typical use cases | Professional analysis, research reports, financial statistics | Concept illustrations, decorative images |
Visualization backends differ across Agent implementations. Some default to professional visualization libraries to ensure accuracy, while others integrate image generation for better visual appeal. When choosing an Agent, treat “are the charts driven by real data?” as a must-have requirement.
Precision Limits: The Hallucination Problem Agents Cannot Escape
Agent mode is not a cure-all. When key steps still rely on the model reasoning about numbers directly, hallucinations persist.
In a small internal test, several mainstream models were given the same quantile question, and their answers varied widely:
- inventory percentile answers clustered roughly between 21% and 30%;
- the basis-point percentile answers fluctuated between 55% and 68%.
This was a single informal test from around the first half of 2025, not a formal benchmark. Results can vary substantially across models, versions, and datasets, so it should not be treated as a general conclusion. Even so, it shows that when handling exact numbers, models still produce approximate output. They guess a plausible-looking decimal based on the training distribution and context, without necessarily performing the actual computation. Any step that depends on exact numbers should therefore be handled by code execution.
Running Model-Generated Code Safely
Security boundaries must be set before you execute model-generated scripts. Recommended practices include:
- Run scripts in a sandbox or restricted container.
- Use data accounts with least-privilege, read-only access.
- Limit network access and execution time to prevent accidental deletion, unauthorized access, and data leaks.
- Review outputs manually before they go live.
- Mask sensitive data before sending it to third-party models or external services through web search, and make sure this follows internal compliance requirements.
Practical Tips for Building Reliable Data Intelligence Applications
- Compute numbers with code: Use scripts for all statistics, quantiles, and derived metrics.
- Enforce validation: Build in sanity checks and a second computation, and stay alert to magnitude, units, and empty values.
- Keep chart data consistent: Generate charts from data produced by calculation scripts, and avoid purely generative image approaches.
- Keep workflows auditable: Preserve the full execution trace and intermediate artifacts so they can be reviewed and traced later.
- Design for elastic deployment: On mainstream cloud platforms, container services can host sandboxed execution, object storage can hold intermediate artifacts and charts, and logging and monitoring can record execution traces. These services typically use pay-as-you-go pricing and scale with task volume; actual details depend on usage.
FAQ
Q: How should I choose a visualization backend?
A: Prefer code- or real-data-driven solutions, where coordinates and values can be traced and verified. Generative images are only suitable for concept illustrations, not professional research reports.
Q: How can I troubleshoot common errors?
A: A KeyError usually means a mismatched column name or BOM characters; check the columns with fieldnames. A ValueError usually means a quantile out of range or a failed type conversion; add assertions and retries during data cleaning.
Q: Is this approach safe?
A: Only if execution is isolated. Always run scripts in a sandbox or restricted container with least-privilege access, network and time limits, and manual review of outputs. Mask sensitive data and follow compliance requirements before sending it out.
Conclusion
The real progress of LLM Agents in data analysis is not that they “answer questions better,” but that they have learned to “do the math.” Leaving precise computation to code and understanding and orchestration to the model is the right way to curb hallucinations and approach professional quality. Keep one point in mind: as long as key steps still depend on probabilistic generation, the precision limit remains. The higher the stakes and the greater the need for accuracy, the more tightly you should keep deterministic computation in the hands of code.
