The Evolution and Governance of AI Data Analysis Capabilities

Over the past two years, how AI participates in data analysis has changed significantly: from users uploading spreadsheets and writing ad-hoc scripts, to connecting to business systems for automated data retrieval, and finally to encapsulating analysis steps, scripts, and validation requirements into reusable skill packages. While capability boundaries continue to expand, a core issue has emerged: AI being able to read materials does not mean it can use them correctly; being able to execute analysis does not mean it can stably reuse analytical experience.

A notable signal comes from a research team that exposed a vast amount of historical SQL to an analysis Agent, hoping it would learn answers from past analysts’ work. The team verified the execution logs and confirmed the Agent indeed read the materials, but the accuracy improvement was less than one percent. Historical SQL is written by analysts for real-world problems and theoretically contains a wealth of business metrics and processing experience, so why didn’t it yield significant gains? Answering this requires understanding the evolutionary stages of AI data analysis capabilities.

Phase 1: Starting with Single-File Analysis — Low Barrier, but Relies on User Context

The early form was straightforward: users uploaded data files like CSVs, and AI wrote and executed scripts to complete cleaning, calculation, and chart generation. For those familiar with data, this experience was a significant productivity boost—no need to repeatedly look up functions or tweak charts; just state the requirement and ask a few follow-up questions to get a preliminary analysis.

However, this capability had an implicit prerequisite: the user provided the critical context. They knew which table to upload, what the fields meant, which records should be excluded, and whether outliers were reasonable. When AI made omissions, the user could promptly spot and correct them. In other words, early AI analysis was more of an efficient “execution assistant” rather than an independent business analyst.

The keyword for this phase is “point efficiency.” It suits tasks like budget reconciliation, regional sales comparisons, and ad-hoc data exploration, but the analysis quality heavily depends on the user’s understanding of the data.

Phase 2: Connecting to Business Systems — Less Data Shuffling, but Not Necessarily Semantic Understanding

When users wanted to eliminate the hassle of downloading, uploading, and organizing data, AI data analysis began connecting to business tools via standardized remote integration protocols to fetch project history, task status, and organizational knowledge. Data retrieval became more automated and manual shuffling decreased, but “finding the data” and “knowing how to analyze it” remained two different things.

Data in business systems often carries complex semantics: the same metric might have multiple calculation definitions, the same field could mean different things across business lines, and outliers need to be evaluated in the context of activity cycles. The fact that AI can retrieve tables and data doesn’t mean it automatically understands these implicit constraints. Therefore, this phase solved “data accessibility” but did not fully solve “analytical semantic correctness.”

Phase 3: Delivery Loop and Skill Packages — Solidifying Experience into Executable Assets

Subsequently, AI began to create and edit files like spreadsheets, documents, and presentations, allowing a single conversation to extend from data processing to delivering a report. It seemed analytical work could now be handed over to AI, but users soon encountered another type of repetitive labor: having to repeatedly explain how to format the report, which anomalies to flag, and who the final audience is.

Reusable skill packages emerged as a response to this. Operation instructions, scripts, reference materials, and validation requirements can be bundled together for AI to read when encountering related tasks. Analysis steps repeatedly verified by one person can be written into a skill pack, complete with scripts and checklists, and shared with colleagues. This way, experience no longer stays buried in individual chat histories but becomes a reusable organizational asset.

The keyword here is “reuse.” However, reuse has varying levels of quality: saving a prompt is reuse, but saving a complete skill package with scripts, data definitions, anomaly rules, and acceptance criteria is high-quality reuse.

The Historical SQL Reuse Surprise: Reading ≠ Knowing How to Use

Returning to the historical SQL experiment: massive amounts of SQL were exposed to the Agent, and execution logs proved the materials were read, yet accuracy gains were limited. The reason usually isn’t that “the model didn’t read it,” but rather “SQL as a knowledge carrier has structural flaws.”

First, SQL is a result, not an intent. A piece of SQL records how something was calculated at the time, but it may not record the user’s original question, the business background, why certain records were excluded, or how the result was verified. What the Agent sees is code, not the complete decision chain.

Second, semantics are implicit and environment-dependent. SQL might contain specific table names, fields, dialects, time windows, and filter conditions. Once removed from its original environment, the semantics can drift. Directly retrieving similar SQL easily leads to answers that are “formally similar but semantically inconsistent.”

Third, success and failure samples are mixed. Historical SQL isn’t always best practices; some are ad-hoc queries, one-off fixes, or deprecated logic. Without quality labels, the Agent struggles to judge which ones are worth learning from.

Fourth, lack of question-answer-validation pairs. A high-quality knowledge base requires “natural language question, SQL, semantic explanation, result sample, and validation method” to appear together. With only SQL, the retrieval granularity is too coarse, making it hard for the Agent to map code snippets to the current problem.

Fifth, no execution feedback loop. After reading the materials and generating new SQL, if there are no result validations, regression tests, or manual reviews, errors won’t be promptly exposed. Execution logs can only prove “read,” not “used correctly.”

From Data Feeding to Executable Knowledge: A Governance Framework

To move AI data analysis from “can calculate” to “reusable and verifiable,” we need to combine knowledge governance with analysis engineering.

1. Establish a semantic layer and metric dictionary: Standardize metric definitions, dimensions, time semantics, filtering rules, and owners. AI should query the semantic layer before writing SQL.
2. Structured example library: Each example should include a problem description, standard SQL, semantic explanation, result sample, applicable boundaries, and validation assertions.
3. Skill package encapsulation: Write high-frequency analysis processes into executable skill packages containing step instructions, scripts, input/output constraints, anomaly rules, and acceptance checklists.
4. Evaluation set and regression testing: Establish baseline answers for typical questions. Run regressions after every prompt, retrieval, or model adjustment to observe accuracy, semantic consistency, and explainability.
5. Observability and human-in-the-loop: Record retrieval sources, generated SQL, execution results, and validation status; retain manual review checkpoints for key reports.

A sample skill package configuration might look like this:

skill: monthly_budget_variance
inputs:
  - budget_table
  - actual_table
steps:
  - validate_schema
  - normalize_period
  - join_by_department
  - compute_variance_ratio
checks:
  - total_amount_reconciles
  - null_ratio_below_threshold
  - top_anomalies_have_explanation
output:
  - variance_report
  - anomaly_list

The evaluation loop can be abstracted as:

for case in evaluation_set:
    context = retrieve_semantic_layer(case.question)
    sql = agent.generate_sql(case.question, context)
    result = execute(sql)
    score = compare(result, case.expected, case.assertions)
    log_trace(case, context, sql, result, score)

Implementation Suggestions

When teams implement this, they should first choose high-frequency analysis scenarios with relatively stable semantics, such as monthly business reports, budget variances, or regional sales comparisons. Next, precipitate these scenarios into skill packages. Then, establish a small-scale evaluation set and regression mechanism. Only after these steps should they expand to historical asset reuse. On general cloud platforms like Baidu AI Cloud, basic capabilities such as object storage, logging services, and monitoring alerts can be used to host knowledge assets, execution traces, and evaluation records, but the core remains semantic governance and engineering closed-loops.

The competition in AI data analysis is shifting from “can it generate SQL” to “can it understand business semantics, reuse organizational experience, and stably validate results.” Historical SQL holds value, but it is not a plug-and-play answer library. Only by governing the intent, semantics, boundaries, and validation methods behind the SQL together can AI data analysis evolve from a demo-level capability to a production-level capability.