LLM Agents for Data Analysis: Capability Boundaries and a Precision Pitfall Guide

    LLM-powered agents are rapidly moving into data analysis workflows, automating task chains that once required manually stitching together multiple tools—reading tables, retrieving reference material, invoking statistical methods, generating charts, and compiling reports. Yet in real production environments, output becomes unstable once tasks involve precise values, statistical metrics, financial reconciliation, or chart scales. This article explains why LLM agents produce inaccurate results, how to avoid percentile errors, and how to choose the right approach for chart generation. It maps out the genuine strengths and boundaries of LLM agents and provides practical engineering safeguards. The core conclusion is simple: let the model handle understanding and expression, and let code handle calculation and verification.

    Introduction: Why Can’t LLMs Do the Math in Data Analysis?

    Over the past two years, agents built around large language models have been rapidly entering data analysis workflows. A typical agent can carry out a task chain of “read a table, retrieve reference material, invoke statistical methods, generate charts, and compile a report,” automating analysis processes that once required a person to manually connect several tools. This capability clearly improves efficiency. However, in real production environments, developers often run into a critical problem: when an analysis task involves precise values, statistical metrics, financial reconciliation, or chart scales, the model’s output may not be stable.

    This article therefore examines questions such as “why are LLM-based data analyses inaccurate” and “how do you avoid percentile errors, and how should you choose between chart generation approaches.” It systematically reviews the capability boundaries of LLM agents and offers engineered fallback solutions. The core conclusion can be summed up in one sentence: let the model handle understanding and expression, and let code handle calculation and verification.

    1. What Can LLM Agents Do, and Which Layer Do They Belong In?

    Before discussing limitations, we first need to clarify the real strengths of LLM agents. Only by placing the model in the right position can you both capture its efficiency and control the risk.

    First, large models are good at tool orchestration and intent understanding. A user states a need in natural language—for example, “read the current inventory table and compute the historical percentile of the current inventory level”—and the agent can break that request down into steps such as reading the file, cleaning the data, calling a statistical function, and producing a conclusion. This mapping from a vague intent to executable actions is an important source of value for an agent compared with a traditional script.

    Second, a large model can read structured data through scripts or tools. For example, when using a data-processing library to open an Excel file, a CSV file, or a database table, the model is not “guessing numbers from memory”; it calls deterministic tools to obtain fields, row counts, means, extremes, and other basic information. In this situation, the source of the numbers is controllable.

    Third, large models are well suited to information retrieval, summarization and synthesis, and report organization. For retrieving, de-duplicating, and consolidating external material, and for organizing analysis results into sectioned text, a large model can perform reliably.

    That said, the applicable boundaries need to be clear: LLM agents suit analysis workflows where tools provide coverage, inputs are controllable, and results are verifiable. They are not suited to directly entering high-stakes financial decision-making or compliance scenarios without data validation, fallback mechanisms, or a clearly defined calculation methodology.

    One key principle: any action that a deterministic tool can complete—code, a database, a statistics library, a plotting library—should be handed to the tool. The large model is mainly responsible for “understanding intent” and “organizing expression.”

    2. Why Are LLM Data Analyses Inaccurate?

    When no deterministic tool or code execution is invoked, the model cannot guarantee precise numerical computation. This is the boundary to watch most carefully in LLM-based data analysis.

    There are three main reasons.

    First, there is a mismatch between the generative paradigm and the computational paradigm. Large models are essentially good at probabilistic language generation, not at the strict, deterministic arithmetic that a calculator performs. For tasks such as multi-digit multiplication and division, floating-point accumulation, iterative statistics, and complex sorting, the model does not naturally possess a “calculator-like” capability for exact modeling.

    Second, once numbers are split into tokens, their numerical relationships are weakened. Digits may be broken into different fragments during input and output, and carry, borrow, rounding, and decimal-place alignment can all be affected by context, producing small but non-negligible errors.

    Third, the model tends to generate results that “look reasonable.” To keep the context coherent, it may output a number that fits the narrative logic rather than going back to the raw data and checking row by row. This can make several decimal places look trustworthy while the value actually deviates from the true one.

    Therefore, in scenarios such as actuarial work, statistical inference, financial reconciliation, and risk-control metrics, you cannot let the model “mentally calculate” the result. The correct approach is to let a deterministic computation library produce the result, while the large model only explains, summarizes, and expresses it.

    3. How Do You Avoid Percentile Errors? Seeing the Deviation on the Same Dataset

    To illustrate the problem, we can design a neutral example. Suppose we have historical inventory data and want to compute “the historical percentile of the current inventory level.” Here we use a common definition: the proportion of historical samples that the current value is not lower than—that is, the share of historical samples less than or equal to the current value.

    The example dataset is:

    [120, 98, 143, 87, 205, 176, 154, 132]

    The current inventory is the last record: 132. Under the “less than or equal to the current value” definition, the samples that satisfy the condition are 120, 98, 87, 132—4 samples out of 8 total. So the true value is 4 / 8 = 50.0%.

    The table below shows illustrative output from different agents on the same dataset. The numbers illustrate the phenomenon and are not a judgment on any specific product:

    Calculation mode Metric output: historical percentile of current inventory True value (same dataset)
    Agent mode · Model A 26.3% 50.0%
    Agent mode · Model B 21.2% 50.0%
    Agent mode · Model C 27.4% 50.0%
    Agent mode · Model D close to 30% 50.0%

    This table shows that even with the same data and the same metric definition, different agents can produce clearly different results. A percentile depends on the sample distribution and on locating a position; once the model fails to call a deterministic computation tool, it can produce approximate or even wrong judgments.

    To avoid this kind of error, statistical computation should be encapsulated into a tool function. The following is illustrative code you can put into practice. The recommended runtime is Python 3.10 or later, with numpy 1.26 or newer; the input is a list of numbers, and all items must share the same unit.

    1. import numpy as np
    2. def current_rank(values, current, include_equal=True):
    3. """
    4. Compute the percentile of the current value within the historical samples.
    5. Definition: when include_equal=True, compute the share of samples less than
    6. or equal to the current value. Sorting does not change this proportion; if
    7. you use an interpolated percentile, keep the definition consistent across
    8. the team.
    9. """
    10. arr = np.asarray(values, dtype=float)
    11. if arr.size == 0:
    12. raise ValueError("Input cannot be empty")
    13. if np.isnan(arr).any():
    14. raise ValueError("Input contains NaN; clean it first")
    15. if current is None:
    16. raise ValueError("Current value cannot be None")
    17. current = float(current)
    18. if np.isnan(current):
    19. raise ValueError("Current value cannot be NaN")
    20. if include_equal:
    21. return float(np.mean(arr <= current))
    22. return float(np.mean(arr < current))
    23. inventory = [120, 98, 143, 87, 205, 176, 154, 132]
    24. current = inventory[-1]
    25. rank = current_rank(inventory, current, include_equal=True)
    26. assert 0.0 <= rank <= 1.0
    27. print(f"Historical percentile of current inventory: {rank:.1%}")

    This code fixes the result as a deterministic output. For empty lists, NaN, and invalid input, the code raises an error directly rather than silently generating a plausible-looking value. In engineering practice, it is best to register functions like this as tools the agent can call; the model is only responsible for invoking them when needed and reading the return value.

    4. How Should You Choose Between Chart Generation Approaches? Beauty Is Not Rigor

    Data analysis reports usually need visualization. Two common approaches exist today.

    The first is to use a general-purpose image-generation model to “draw” the chart directly. Its advantages are fast generation and rich styling, which suit covers, illustrations, and conceptual graphics. But it cannot guarantee that axis scales, data-point positions, value labels, and proportions are precise. It is essentially “a picture that looks like a chart,” not a data chart driven by real data.

    The second is to use a deterministic plotting library to generate charts. A plotting library takes data arrays as input and can precisely render axes, scales, labels, and legends, with reproducible and auditable results. In professional research reports, financial analysis, statistical reports, and similar scenarios, this is the only reliable approach.

    Therefore, chart generation should follow these minimum standards:

    1. Make the data source explicit—for example, the raw table, the cleaned dataset, or the metric calculation table.
    2. Make the calculation definition explicit—for example, the definition of mean, median, year-over-year, month-over-month, or a percentile.
    3. Label the axis units, and avoid mixing quantities, amounts, and percentages.
    4. Explain how outliers are handled—for example, removal, truncation, retention, or separate annotation.
    5. Choose the chart type according to the task: use a line chart for trends, a bar chart for categorical comparisons, a histogram for distributions, and a scatter plot for exploring relationships.
    6. State a clear prohibition: generated images must not be used as precise data charts.

    A sensible engineering trade-off is to produce precise data charts with a plotting library, while covers, illustrations, and decorative elements can be supplemented by a generative model. This preserves visual appeal while avoiding numerical distortion.

    5. How Do You Implement This in Engineering? Six Best Practices

    Overall, enterprises can adopt the following practices in production environments.

    First, layer responsibilities. Clearly separate the “language layer” and the “computation layer.” The language layer—handled by the large model—covers intent understanding, information retrieval, text organization, and expressing conclusions. The computation layer—handled by deterministic libraries—covers all numerical operations.

    Second, encapsulate tools. Package capabilities such as percentiles, means, year-over-year and month-over-month comparisons, regression, and anomaly detection into functions or interfaces, so the agent obtains results through tool calls rather than relying on the model to compute them itself.

    Third, make results reproducible and auditable. Every key metric should retain its input data, computation code, version number, and intermediate results. That way, when a result is questioned, it can be traced, rechecked, and diagnosed.

    Fourth, cross-validate and provide fallback on failure. High-stakes metrics can be cross-checked with multiple methods—for example, computing the same percentile with different functions. If the results disagree, trigger an alert and route the case to human review. When a tool times out, permissions fail, data is missing, or results are abnormal, the flow should automatically degrade to human handling or block the process, preventing wrong results from flowing further.

    Fifth, govern resources and quotas. Calling tools, retrieving material, and generating text all consume resources. You need quota monitoring and degradation policies within the workflow to avoid task interruption or unfixable results caused by an exhausted quota.

    Sixth, standardize visualization. Produce precise charts with a plotting library, and label the data source, calculation definition, and units in the chart. Generated images can serve only as supporting illustration and cannot take on the responsibility of precise data expression.

    If you need to build the technical foundation that supports agents like these, you can use common infrastructure such as a container platform, object storage, message queues, logging services, and monitoring and alerting. The main selection criteria are four: isolated runtime environments with elastic scaling, persistence of data and report artifacts, task decoupling and asynchronous scheduling, and logging, monitoring, and observability. By carrying runtime and operations on common infrastructure, teams can focus more on business logic and analysis quality.

    6. Conclusion: The Model Is a Smart Dispatcher, Not a Precise Calculator

    Large-model agents are reshaping how data analysis is produced, but their value is not as a “precise calculator.” It is as a “smart dispatcher and expresser.” Only by acknowledging and avoiding their weakness in numerical precision when no tool is called, and by using deterministic code to backstop key calculations, can you truly build an efficient, reliable, and auditable automated analysis pipeline.

    Understanding this boundary is the first step toward using large models well in production.