AI Agents for Data Analysis: From Ad Hoc Q&A to Reusable Workflows

1. An Experiment That Fell Short of Expectations

A company’s data team once ran an experiment: they made a large set of historical analysis queries available to an AI agent, hoping it would find answers in the analysts’ past work. The team checked execution logs and confirmed that the agent had indeed read the materials. In the end, however, accuracy improved by less than one percentage point.

At first glance, this result seems odd. Historical queries were written by analysts for real problems, and agents are good at reading code, so following existing analysis should have worked. The issue is that “being able to read a query” and “being able to judge whether that query applies to the current problem” are two different things. Historical statements carry implicit premises such as metric definitions, time ranges, deduplication rules, and business exceptions, and these premises are rarely written in the statement itself. The agent gets the result logic but lacks the decision context.

This point clarifies the design requirement: to improve analysis quality, what needs to be captured is judgment rules and validation methods, not just code snippets.

2. Three Stages of Evolution

The combination of data analysis and AI agents can be roughly divided into three stages.

Stage 1: File-Level Assistance

A user uploads a spreadsheet, and the model generates and executes data processing code to complete cleaning, calculation, and chart output. The value of this stage is lowering the tooling barrier. Work that once required remembering function syntax and repeatedly adjusting chart parameters becomes a matter of describing the need in natural language and correcting the result through a few follow-up rounds.

But the outcome depends heavily on the user. The user needs to know which table to use, what the fields mean, and which records should be excluded. The model may make mistakes, such as treating a total row as a detail row or missing refund orders. Only the user can identify and correct these problems. This stage is therefore essentially “the human makes judgments, and the model executes.”

Stage 2: Direct Data Source Connections

The next natural requirement is to reduce manual movement. Through standardized remote interfaces, an agent can directly read project records, task status, and organizational knowledge from business systems.

This reduces upload and download steps, but it does not solve the analysis method problem. The model finds the data, but the user still needs to explain metric definitions, calculation logic, and output requirements. Connectivity expands the range of accessible data, but judgment capability does not improve at the same time.

Stage 3: Workflow Codification

At this stage, the agent can generate spreadsheets, documents, and presentations, covering the entire process from data processing to report delivery in a single session. A new problem then appears: repetitive work. Every session requires restating the report structure, anomaly criteria, and target audience.

The solution is to package repeatedly validated practices. A complete analysis skill package usually contains three types of content: operating instructions, executable scripts, and reference materials. When the agent encounters a related task, it reads these materials and executes according to the established workflow. Team members can share the same workflow, and new team members do not need to start from scratch.

3. Why Historical Queries Are Hard to Reuse Directly

Returning to the experiment at the beginning, its result can be explained from three angles.

First, metric definitions are implicit. The same term “active user” may refer to different behavioral definitions in different business lines. A historical query reflects only the definition used in one analysis; it does not explain the scope in which that definition applies.

Second, timeliness matters. Metric definitions, table schemas, and business rules change. A query written several months ago may reference deprecated fields or omit filtering conditions added later.

Third, the basis for judgment is missing. Before writing a query, an analyst has usually ruled out several incorrect directions and confirmed the source of anomalous data. These reasoning steps are rarely preserved. The agent sees only the final statement and cannot reconstruct the judgment made at the time.

Therefore, effective codification should capture “why it is calculated this way,” not just “how it is calculated.”

4. Designing a Reusable Analysis Workflow

4.1 Write Metric Definitions as Explicit Contracts

Before the workflow starts, write metric definitions, time ranges, inclusion and exclusion rules into a structured configuration. Below is an illustrative configuration snippet; the field names are for explanation only:

metric: active_users
definition: Deduplicated users with at least one valid login during the period
time_range:
  start: "{{ period_start }}"
  end: "{{ period_end }}"
exclude:
  - internal_test_accounts
  - duplicate_device_records
validation:
  max_daily_drop_ratio: 0.3
  require_row_count_match: true

The purpose of the configuration is to make implicit premises explicit. After reading it, the agent can check whether the inputs satisfy these constraints before execution.

4.2 Use Layered Validation

Data analysis errors often do not throw exceptions. Instead, they quietly produce a number that looks reasonable. Validation therefore needs to be layered:

• Input layer: confirm that the data source time range, field types, and row counts match expectations.
• Calculation layer: cross-check key metrics, for example, ensuring that the sum of subitems equals the total.
• Output layer: check for outliers and compare against historical ranges, flagging values beyond thresholds for manual review.

Example code:

def validate_metric(current: float, previous: float, max_drop: float) -> bool:
    """Check whether the metric drop versus the previous period exceeds the threshold."""
    if previous == 0:
        return current == 0
    drop_ratio = (previous - current) / previous
    return drop_ratio <= max_drop

When validation fails, the workflow should stop producing conclusions and explain the specific reason for failure, rather than presenting an unchecked number.

4.3 Keep a Record of Judgment

At the end of each analysis, record three items: the metric definition version used, which data was excluded and why, and which results were manually confirmed. These records allow later analysts to trace conclusions and allow the workflow to be continuously corrected.

5. Implementation Steps

1. Start with high-frequency tasks that have clear rules, such as weekly report checks or budget-versus-actual spending comparisons. These tasks have stable definitions and are easy to validate.
2. Have business experts organize metric definition documents and mark steps where errors have occurred in the past.
3. Write metric definitions as structured configuration and validation rules as executable scripts.
4. Use historical data for replay validation: run the workflow for periods with known results and compare the output with manual conclusions.
5. Run a small-scale pilot, collect misjudgment cases, and update configuration and validation thresholds.
6. After confirming stability, expand to other business lines.

Replay validation is the step most likely to be skipped, and it is also the key to judging whether a workflow is reliable. Only when the output matches known conclusions can the workflow be shown to have learned judgment methods, not merely how to execute statements.

6. Boundaries and Risks

AI agents are suitable for repetitive, rule-clear work such as data cleaning, standard report generation, and initial anomaly screening. Strategic judgment, causal inference, and conclusions that require accountability should still be led by humans.

The following points also require attention:

• Data permissions should follow the principle of least privilege, and the agent should access only the datasets required for the task.
• Analysis results should be output together with data sources and metric definitions so they can be reviewed.
• Externally published numbers should undergo manual review, especially content related to finance and compliance.