Claude and Data Analysis: Two Years of Capability Shifts and Engineering Lessons

1. A “failed” experiment: full materials, limited gain

In mid-2026, the data team at a model provider publicly reviewed a not-so-successful experiment. Framed around Claude and similar AI data-analysis agents, the experiment opened a large number of historical queries to an agent: SQL statements that analysts had written for specific questions. The hope was that the agent could “find answers” from past experience. The team checked execution logs and confirmed that the material had indeed been read. Yet the final accuracy improvement was less than one percentage point.

This result deserves serious attention. The SQL had been written by experienced analysts for real business problems and contained substantial tacit knowledge, while today’s large models are particularly good at understanding and generating SQL. In theory, copying prior analysts’ approaches should have worked. Why, then, was the gain so limited?

Looking back two years, the answer becomes clearer: the capability boundary of AI in data analysis expanded rapidly, but the decisive bottleneck shifted from “can it do it?” to “does it know how it should be done?” Simply accumulating material does not automatically create the latter.

2. Three levels of capability evolution

Over the past two years, the data-analysis capabilities of mainstream agent platforms have broadly gone through three stages. Each stage reduced part of the human workload and exposed the next bottleneck.

Stage	Time	Input method	Main human work	New bottleneck
Level 1: Single-file analysis	From fall 2024	Upload one spreadsheet file	Choose the table, judge whether results are correct, and ask the next question	Manual data handling
Level 2: Connected data sources	From 2025	Connect to business systems through tool protocols	Explain analysis goals and steps	Finding material is not the same as knowing how to analyze
Level 3: End-to-end delivery and asset reuse	From fall 2025	Read data, directly produce finished files, and reuse predefined workflows	Design workflows, accumulate standards, and review deliverables	Workflow reuse creates consistency, reuse, and compliance requirements

2.1 Level 1: Starting with a single file

In fall 2024, mainstream AI assistants began shipping code-execution capabilities. A user could upload a CSV file, and the assistant could write and run JavaScript or Python to clean, calculate, and chart the data. The improvement in user experience was obvious: instead of looking up functions, writing scripts, and adjusting styles, users could state a requirement, ask a few follow-up questions, and obtain a reasonable analysis.

Today, analysts, operations staff, and even business owners use office agents, AI assistants, and AI coding tools for analysis. The entry point is often a spreadsheet at hand: reconciling the difference between budget and actual spend, comparing sales performance across regions, or finding the cause of abnormal gross margin in a product line. Ad hoc analyses that once required scheduling and analyst support can now be attempted by the requester.

One point is often overlooked: this smooth experience depends first on the user’s experience. The user knows which table to upload, what the fields mean, and which test records should not be counted. If the model misses a type of anomalous order or shifts a time series by one month, the user can spot it and point it out. AI takes over the “doing” part, while judgment remains with the human.

2.2 Level 2: Connecting tools and data sources

Soon, user requests changed from “calculate this for me” to “find it for me.” Manually exporting and uploading files every time is cumbersome, especially when reports span multiple systems. Starting in 2025, mainstream platforms commonly supported connecting to internal systems through remote tool protocols, allowing assistants to read project records, ticket status, and organizational knowledge according to permissions. The Model Context Protocol (MCP) is a representative approach in the industry: servers expose tools, resources, and prompt templates, and the assistant calls them when needed. This “connector” pattern also became a common design across several platform vendors.

This approach reduces the friction of moving data, but it does not solve the analysis problem itself. The assistant can retrieve data, but it still does not know what the reporting scope should be for the quarter, which anomalies need to be flagged, or who should receive the result. This is another version of the opening case: the material is present, the capability is present, but “how it should be done” is still missing.

2.3 Level 3: From processing data to delivering outputs, then reusing workflows

By fall 2025, leading agent platforms could generate and edit office files such as spreadsheets, documents, and presentations. In one conversation, they could go from data cleaning to a finished report. The “last mile” of analysis work began to be completed.

A new problem appeared. Every time users produced a weekly report, they had to explain again how the report should be arranged, which anomalies were worth marking, and who would receive it. It would be better if these requirements could be reused next time.

In the same season, mainstream platforms began introducing a “Skills” mechanism: steps, executable scripts, reference material, and cautions can be packaged into a folder, and the assistant reads them when a task matches. A carefully refined monthly channel-review workflow can be handed to colleagues together with validation scripts and metric-definition notes, so everyone follows the same standard.

In capability terms, the three levels gave the model three things:

• Execution capability: the model writes code, runs calculations, and produces charts;
• Data access: the model obtains multi-source data on demand;
• Process knowledge: the model completes work according to established methods and specifications.

3. Why “more material” often fails

Back to the opening case: at least three reasons explain the limited gain.

First, conflating “retrievable” with “reusable.” Historical SQL was written for specific tables and time windows. Once the time window, partition pruning, or field naming changes, copying the old query as-is fails. For a model, these are examples that are hard to generalize to new questions, not executable methods.

Second, lack of explicit expression of “how to do it.” A SQL query can say which fields to select and how to aggregate, but it rarely says why test orders should be excluded or which document takes precedence when there is a metric-definition dispute. These implicit rules are not written into the code, so the model cannot learn them.

Third, context overload. The more material a model reads, the more easily key rules are buried. “It was read” is not the same as “it was used.” Execution logs can prove that access happened, but they cannot prove that it changed the conclusion.

Separating knowledge makes this easier to understand. In data analysis, knowledge useful to a model can be divided into three types:

Knowledge type	Example content	Effective carrier
Factual knowledge	Where tables are, field meanings, data freshness	Data catalog, metric dictionary
Procedural knowledge	Metric-definition rules, analysis steps, validation methods	Executable scripts, workflow templates, checklists
Preference knowledge	Chart styles, anomaly thresholds, reporting recipients	Report templates, output specifications

Historical SQL mainly belongs to the second type, but its style depends heavily on context and mixes in a large amount of factual knowledge. Feeding it directly to a model as “experience” is like asking a person to read tens of thousands of lines of someone else’s uncommented script: lots of information, low signal-to-noise ratio.

4. Engineering practices: making analytical capability durable

To move AI data analysis from “occasionally useful” to “reliably stable,” the key is to make implicit knowledge explicit and structured, and to give the system self-check and correction capabilities.

4.1 Build a metric semantic layer

Add a translation layer between the model and the data: translate table and field names into business concepts, and express metric definitions as executable rules. Open-source semantic-layer projects such as Cube and the dbt Semantic Layer are essentially doing the same thing. A simplified metric-definition example:

# metric_defs.yaml: single source of truth for metric definitions
metrics:
  net_revenue:
    description: "Net revenue, after refunds"
    formula: "SUM(order_amount - refund_amount)"
    grain: ["stat_month", "channel", "region"]
    filters:
      - "status NOT IN ('test', 'internal')"
    freshness: "<= 24h"
    owner: "data-platform-team"
dimensions:
  channel:
    values: ["online", "offline", "partner"]
    alias: { "Online": "online", "Offline": "offline" }

With this layer, the assistant does not need to guess metric definitions from historical SQL. It can look up the definition first and then generate the query. Metric definitions also become objects for review and change management, so metric definitions are no longer one person’s private experience.

4.2 Turn workflows into templates and skills

For high-frequency reports, freeze validated queries, calculations, and validation logic into parameterized templates. The assistant then fills in parameters, calls scripts, and interprets results instead of writing SQL from scratch. Example:

-- channel_review.sql: channel-review template; {{ }} placeholders are parameters
WITH base AS (
  SELECT stat_month, channel, SUM(net_revenue) AS revenue
  FROM dws_sales_monthly
  WHERE stat_month BETWEEN '{{start_month}}' AND '{{end_month}}'
    AND status NOT IN ('test', 'internal')
  GROUP BY 1, 2
)
SELECT *,
       revenue - LAG(revenue) OVER (PARTITION BY channel ORDER BY stat_month) AS mom_delta
FROM base
ORDER BY stat_month, channel;

Compared with hand-written SQL, templates bring three benefits: parameterization keeps metric definitions stable, query plans can be optimized in advance, and changes need to be made in one place. Under the Skills mechanism, these scripts can be packaged as assets together with validation checklists, distributed within a team, and upgraded uniformly.

4.3 Use deterministic validation to cover model uncertainty

Large models are inherently probabilistic, so analytical conclusions cannot rely only on “looks reasonable.” Deterministic checks should be handled by code execution:

def validate(df, expected_total: float) -> list[str]:
    errors = []
    if df["revenue"].isna().any():
        errors.append("Revenue records with null values exist")
    if not df["region"].isin({"north", "south", "east", "west"}).all():
        errors.append("Unknown region values appear")
    if abs(df["revenue"].sum() - expected_total) / expected_total > 0.005:
        errors.append("Total revenue deviates from the general ledger by more than 0.5%")
    return errors

Common checks include:

• Reconciliation: whether the total deviates beyond a threshold from authoritative sources such as the financial general ledger;
• Null values, duplicate primary keys, and invalid enum values;
• Time continuity: whether any months are missing from a sequence;
• Magnitude sanity: whether month-over-month changes fall outside the historical distribution.

If anomalies are found, the assistant should pause before outputting conclusions and explain the reason, rather than continuing to write a report while carrying problems forward.

4.4 Build an evaluation set and baseline

Improvement cannot be based on feel. Organize historical real-world questions into a fixed evaluation set. Each case includes the question, expected result, and allowed tolerance:

cases:
  - id: channel_mom_2025Q3
    question: "Month-over-month net revenue changes by channel for Q3 2025"
    expected_file: fixtures/channel_mom_2025Q3.csv
    tolerance: 0.001

Whenever prompts, model versions, or context strategies change, run the evaluation set and compare two metrics: accuracy and number of regressions. The earlier team’s lesson is exactly this: if this mechanism had been in place, a gain of “less than one percentage point” could have been discovered before launch, rather than only in a real scenario.

4.5 Feed human corrections back into the system

Every correction during use is high-value feedback. If a metric definition is wrong, add it to the metric definition. If a chart is wrong, add it to the output specification. If test orders were missed, add a validation rule. The key is to let feedback enter a versioned, reviewable asset library, rather than remain in a one-off conversation. Preferences maintained only by conversation memory are unstable and cannot be audited.

4.6 Security and governance

Data analysis naturally involves sensitive data. At a minimum, engineering should consider:

• Permission alignment: the assistant can only access data that the caller is authorized to see;
• Least privilege: do not carry sensitive fields unrelated to the analysis in context; mask them first if necessary;
• Auditability: record which queries were generated, which data was accessed, and who confirmed the result;
• Reproducibility: retain parameters, script versions, and metric-definition versions together.

In addition, analysis tasks are suitable for being hosted as platform services: scheduling by container platforms, raw data and intermediate results in object storage, tasks and call chains in logging services, and error rates and execution time integrated with monitoring and alerts. Cloud platforms such as Baidu AI Cloud provide generic cloud capabilities of this kind, so teams do not need to rebuild infrastructure and can focus on semantic layers and validation systems.

5. Common pitfalls and how to avoid them

Pitfall	Symptom	Improvement
More context is always better	Stuffing all historical scripts into context	Retrieve only the minimum necessary information per task
Runs successfully = correct	Only checking code execution success	Add reconciliation and sanity checks
No baseline	Judging “better” by feeling	Fix an evaluation set and run regular regressions
Treating ad hoc scripts as assets	Rewriting the same analysis every time	Template, version, and reuse
Ignoring data freshness	Drawing conclusions from stale data	Declare freshness requirements in metric definitions

6. Conclusion

Over the past two years, progress in AI data analysis has not mainly come from models being better at writing SQL. It has come from engineering systems gradually filling three gaps: connecting data, accumulating methods, and validating conclusions. However rich historical material is, it cannot replace explicit metric definitions, executable templates, and quantifiable evaluation. The effective approach is to make AI work in an environment with clear rules, strict validation, and closed-loop feedback, rather than hoping it will discover best practices by itself from a huge pile of old files.