AI Data Analysis Over the Past Two Years: From Reading Files to Reading Methods

Over the past two years, the boundary of what AI can do in data analysis has kept moving outward. But one experiment pointed in the opposite direction: a data team opened a large volume of historical SQL to an agent and confirmed that the agent had indeed read the material, yet the accuracy gain came in under one percentage point. Seen against two years of capability gains, the bottleneck is not whether the model can do the work, but whether the team has delivered the right method — and whether results can be verified.

1. Four Stages: How AI Data Analysis Capability Evolved

Stage 1 (H2 2024 – H1 2025): Files in the Loop — Starting from a Single Uploaded File

In this period, the data analysis capability of mainstream assistants mostly began with “read one file.” After a user uploaded a CSV or a spreadsheet, the assistant would write and run code to clean the data, compute results, and draw charts.

The change in experience was immediate. Previously you had to look up functions, write queries, and tune charts; now you describe the need, answer a few follow-up questions, and get a reasonably complete first pass. What this changed was mainly execution cost, not judgment cost. Take reconciling budget against actual spend: an agent can quickly rank the projects with the largest variance, compute their shares, and plot the trend — but it has no way of knowing which table is the final version, which adjusting entry should not be counted, or which definition the finance team accepts.

The answers to those questions usually live only in the heads of people who know the data. The agent can execute “list the variances” efficiently, but cannot judge whether a variance makes sense in the first place. This also explains why the same capability produced such different results in different hands: someone who understands the business context spots gaps quickly and corrects course, seeing a clear efficiency gain, while someone who does not can only accept a result that “looks reasonable.”

Stage 2 (Around May 2025): Integrations in the Loop — No Manual Data Shuffling, but Intent Still Needs Spelling Out

The next need followed naturally: if the agent can handle files, can it go get the data itself? By connecting business systems through tool access protocols, the assistant could read project history, task status, organizational knowledge bases, and similar material, reducing the manual work of moving data around.

But finding material is not the same as knowing how to analyze it. Given the same project data, one person cares about schedule risk, another about resource allocation, another about delivery quality. Once the agent has the material, the analysis goal, the metrics of interest, and the judgment criteria still have to be defined by a human. So this step automated the transport of input; the hard part — expressing intent — stayed on the human side.

Stage 3 (H2 2025 – Early 2026): Deliverables in the Loop — From Data to Report in One Session

Starting in the fall of 2025, mainstream AI assistants could create and edit spreadsheets, documents, and presentations, so a single session could carry an analysis all the way from data processing to a delivered report.

Once the analysis was “produced,” a new kind of repetitive work surfaced: explaining again and again how the report should be laid out, which anomalies deserve flagging, and who the report is ultimately for. Each session required the same instructions to be repeated from scratch, and the layout and rules settled after rounds of revision last time started from zero again. What users needed was not stronger generation, but the ability to carry the last approach forward.

Stage 4 (Since October 2025): Methods in the Loop — Turning Practice into a Reusable Asset

The “skills” mechanism introduced in the same period provided a vehicle for preserving practice: instructions, scripts, and reference material are packaged together, and the agent reads them on demand when a relevant task comes up. This is a direct response to the “start over every time” problem.

A set of analysis steps that an individual has repeatedly validated can be written up as an operating procedure and shared with colleagues along with the scripts and references, so they do not have to figure it out from scratch. At this point, the collaboration interface for AI data analysis has expanded from a single file to a reusable method.

Stage	What is handed to the agent	Core friction
Files in the loop	A single file plus a request description	A human must download and upload files
Integrations in the loop	Tool access plus an analysis goal	Material is reachable, but intent still has to be expressed
Deliverables in the loop	The full task from data retrieval to a finished report	Delivery requirements must be restated every time
Methods in the loop	Validated practice plus scripts	How practice is written, checked, and propagated

2. Back to That Counterintuitive Experiment: Why “More SQL” Barely Helped

The team behind the experiment hoped the agent would find answers in analysts’ past work: historical SQL was written by humans against real problems, and the model is good at reading code. The result was a limited accuracy gain — even though execution logs confirmed the agent had read the material. The problem, then, was not whether it was read, but what could be learned from reading it.

1. Historical SQL Is Full of Context Dependencies, and the Patterns May Not Transfer

Analysts’ SQL is usually written around a specific question at a particular moment, and the table names, fields, time windows, and filter conditions all carry that moment’s business context. The agent sees the statement, not the reason those conditions were chosen. More importantly, many business definitions exist only in people’s experience. “Valid order,” for example, may implicitly exclude test orders, internal transfer orders, and orders that straddle periods — and neither the schema nor the SQL may carry any such annotation. Imitating the pattern can therefore inherit the wrong filter conditions along with the right ones.

2. Retrieval Is Not Understanding: The Structure of the Method Is Never Abstracted

“Confirming the agent read the material” only shows that text entered the context; it does not show that the practice within it was understood and correctly transferred. Without consistent naming conventions, comments, and documentation, it is very hard for an agent to distill common principles from hundreds or thousands of historical queries. It is like dumping several years of company reports on a new hire without explaining where the definitions came from or what the judgment criteria were — that alone will not make their analysis more accurate.

3. Without a Verification Loop, the Agent Cannot Tell Whether the Result Is Right

Historical SQL shows “how it was done” but never says “whether the result was correct.” Analytics tasks rarely have a natural right-or-wrong signal the way unit tests do: the same monthly revenue report can differ noticeably depending on how returns are netted. Lacking executable checks and reference answers, the agent can only mimic form and cannot correct outcomes.

This is exactly where the methods-in-the-loop approach adds value: it forces people to make tacit practice explicit, clarifying metric definitions and fixing the steps while writing the procedure, and attaching executable check scripts. Those procedures are then exercised repeatedly by colleagues, forming a loop of write → use → feedback → revise. In short, “feeding SQL” stays at the imitation layer, while “writing the method” has reached the engineering layer. What remains is to close the verification loop.

3. Reusable Analysis Playbooks: Turning Practice into an Asset

1. Structure and Writing Guidelines

A reusable analysis playbook typically contains six parts: trigger conditions, metric definitions, steps, scripts, reference material, and acceptance criteria. Their responsibilities break down as follows:

• Trigger conditions: describe which tasks should use the playbook, helping the agent match the scenario accurately.
• Metric definitions: spell out metric formulas, time windows, and filter rules — the foundation of accuracy.
• Steps: describe the analysis flow in an ordered, executable way.
• Scripts: encapsulate deterministic data processing and computation logic.
• Reference material: hold business context, field descriptions, and report templates.
• Acceptance criteria: give quantifiable, automatically executable checks on the result.

A sample directory layout:

revenue_report/
├── PROCEDURE.md            # Trigger conditions, steps, acceptance criteria
├── scripts/
│   ├── load.py             # Data loading and field validation
│   ├── metrics.py          # Metric computation (reuses shared functions from the org)
│   └── validate.py         # Result validation
└── references/
    ├── metric_glossary.md  # Metric definitions
    └── report_template.md  # Report structure and formatting conventions

PROCEDURE.md might read like this:

# Monthly Revenue Analysis Report
## When to use
Use when the user asks for a monthly revenue analysis, a weekly or monthly
revenue report, or a regional revenue comparison.
## Metric definitions
- Revenue = gross order amount including tax − refund deductions, excluding
  internal transfer orders.
- Valid order: payment status is "paid" and payment time falls within the
  reporting month.
- Region assignment: map the province of the shipping address to its region.
## Steps
1. Run scripts/load.py to validate field completeness and data types.
2. Run scripts/metrics.py to compute monthly revenue and month-over-month
   change for the total, by region, and by product line.
3. Flag anomalies: any item with a month-over-month swing above 15% or a
   refund rate above 5% must include an explanation.
4. Generate the report file following references/report_template.md.
## Acceptance criteria
1. Regional revenue summed together differs from total revenue by ≤ 0.5%.
2. Refund deductions match the totals in the refund detail table.
3. Orders without a payment status in the reporting month must not be counted.

The two most important parts are metric definitions and acceptance criteria: the former tells the agent what “computed correctly” means, and the latter lets the agent judge on its own whether the result is right.

2. Locking Practice Down with Tests: Unit Tests for Analytics Tasks

Descriptive acceptance criteria still need to be turned into executable tests. For each analysis task you can maintain a set of “golden cases”: input data that has been manually verified, plus the expected output (key metric values, the anomaly list, the report structure). Whenever the analysis pipeline changes, run the regression tests first to confirm nothing moved unintentionally.

import pandas as pd
from scripts.metrics import calc_revenue
def test_revenue_matches_manual_result():
    df = pd.read_csv("tests/fixtures/sample_orders.csv")
    result = calc_revenue(df, period="2026-05")
    # Expected result, verified manually
    assert result["total"] == 1_250_000
    assert result["region"]["华东"] == 420_000
def test_region_sum_equals_total():
    result = calc_revenue(pd.read_csv("tests/fixtures/sample_orders.csv"),
                          period="2026-05")
    assert abs(sum(result["region"].values()) - result["total"]) <= 0.005 * result["total"]

These tests are written once by a human and reused many times. After producing an analysis, the agent can run the same suite to check its own work. When a business definition changes, updating the definition document and the cases together is enough to version the practice.

3. From Personal Experience to Organizational Asset: Governance Points

As a playbook moves from an individual to a team, it needs supporting management:

• Review and versioning: metric definition changes should go through review and keep a version history, so every analysis is traceable to the definition it used.
• Permissions and security: write only how to access data in the playbook, never hard-code credentials; scripts retrieve data through a shared permission system, and sensitive fields are masked according to permissions.
• Layered context: the base layer holds organization-wide definitions and templates, the team layer holds analysis methods for specific business domains, and the individual layer holds personal preferences — avoiding the same rule being duplicated and conflicting across playbooks.

4. Engineering Practices for Reliable AI-Run Analysis

1. Three Layers of Context: Goal, Data, and Method

The quality of an agent’s analysis depends on whether it can obtain all three kinds of information at once:

• Goal layer: the business question, success criteria, and output format.
• Data layer: data sources, table structures, field meanings, and data quality.
• Method layer: metric definitions, analysis steps, anomaly-handling rules, and acceptance criteria.

The experiment that stalled at the start mainly supplied raw material for the method layer without providing executable definitions or verification criteria. The better approach is to provide all three layers in a structured way: the goal is stated explicitly by a person, the data layer is obtained automatically through integrations, and the method layer is captured in a playbook and then pinned down by tests.

2. Executable Validation: Turning “Looks Right” into “Verifiable”

Analysis code should carry guard conditions before it runs. If the data is validated right after loading, an input that does not match expectations halts the pipeline immediately instead of running through to charts and a report in a broken state:

def validate_input(df: pd.DataFrame) -> None:
    required = {"order_id", "status", "paid_at", "amount", "return_amount", "province"}
    missing = required - set(df.columns)
    if missing:
        raise ValueError(f"Missing required fields: {sorted(missing)}")
    if df["order_id"].duplicated().any():
        raise ValueError("Duplicate order IDs found; deduplicate by latest status first")
    invalid = df[~df["status"].isin(["paid", "refunded", "cancelled"])]
    if not invalid.empty:
        raise ValueError(f"Unknown payment status: {invalid['status'].unique().tolist()}")
    # Administrative division codes are 6 digits
    bad_province = df[~df["province"].astype(str).str.fullmatch(r"\d{6}")]
    if not bad_province.empty:
        raise ValueError(f"Invalid province code: {bad_province['province'].unique().tolist()}")

3. Bounded Automation: Make the Flow Correct First, Complete Second

Stable automation should be bounded: inputs and outputs are clearly defined, and when data does not match expectations the flow pauses and asks rather than assuming. If the reporting month is missing, for instance, it should ask the user to confirm the month instead of defaulting to the most recent one. As cases accumulate and definitions stabilize, the agent’s autonomous scope can be widened step by step.

4. Platform Support: Traceability and Controllable Access

For an analysis pipeline to run on a platform, it usually also relies on these common capabilities:

• Object storage and a data warehouse for storing and querying data.
• A workflow engine to schedule recurring analysis jobs.
• A logging service that records each step the agent takes and the scope of data it read.
• Monitoring and alerting that notify promptly when a job fails or a metric looks abnormal.
• A unified permission and audit system that keeps data access compliant.

As an individual, you do not need to stand up the whole stack at once. Picking one high-frequency, repetitive analysis — a weekly report or a monthly business review — writing a single playbook with a few tests usually delivers visible improvement right away.

Conclusion

Looking back over these two years, progress in AI data analysis did not come from the model getting stronger alone. From files to integrations, from deliverables to reusable methods, each capability gain was essentially another layer added to the interface between people and AI. The “more SQL” experiment shows the same thing from the other side: once execution capability is strong enough, what determines the quality of the outcome is whether a person has stated the goal, the metric definitions, and the verification criteria clearly.

For practitioners, the best investment is not a larger context window but two things: writing practice into reusable playbooks, and turning correctness into executable tests. With those two in place, AI stops being merely a faster query writer and actually takes over a piece of analysis you can trust.