From One File to a Reusable Method: Two Years of AI Data Analysis

Intro: More Historical Data, No Gain in Accuracy

Imagine this strategy: take the historical analysis material a team has built up over time, including queries and report logic, hand it to an AI analysis agent in bulk, and hope the agent learns the right way to work from what its analysts have done. It seems reasonable on the surface. The material comes from real business problems, and LLMs are naturally good at reading and writing code, so copying the pattern seems like it should just work. The team even checked the execution logs and confirmed that the agent had indeed read the material.

The result was surprising: accuracy improved by less than one percentage point.

This gap is worth digging into. The question is not whether the agent read the material, but what could actually be transferred out of it. To really understand this setback, you have to zoom out and see how the way AI does data analysis has changed over the past two years. Each change solved something, and each one also left a gap behind.

Stage 1: Starting with a Single File — Low-Barrier On-the-Spot Analysis

About two years ago, AI analysis started very simply: users uploaded one data file, and the model wrote and ran code to clean the data, calculate metrics, and generate charts.

The experience easily surprised people. Work that used to mean looking up functions, writing code, and tweaking charts again and again could now be done by describing what you needed and asking a few follow-up questions, and you’d end up with a decent first draft. For people who already knew their data, it was especially helpful. They could point out where the calculation was wrong, or ask another question to see the data from a different angle.

Worth noting is that making this stage actually work took a lot of contribution from the users themselves. They knew which table to pick, what each field meant to the business, and which anomalous records shouldn’t be counted. The model inevitably missed things, but a knowledgeable user could catch and correct them quickly. In other words, AI’s capabilities were anchored by people. What actually made the analysis correct was the combination of someone who understood the data and a model that could execute. The model did not carry the full judgment load on its own.

Even today, plenty of people still start by dropping their spreadsheets in and running them when reconciling budget against actual spending or comparing sales across regions. Work that used to mean asking an analyst can now be attempted by business staff themselves as a first pass.

Stage 2: Connecting to Business Systems — No More Manual Data Shuffling

Once people got used to this convenience, their needs naturally grew: did they really have to download and re-upload files every time? Couldn’t the AI just fetch the data itself?

Around a year ago, integration capabilities started to mature. Using standardized connections, intelligent analysis tools could connect to business systems and pull in project history, task status, organizational knowledge, and other information scattered across systems. This cut down on the repetitive manual effort of shuffling data and let analysis draw on more than just a local file.

Its limits are worth being clear about, though. Integration saved the effort of getting data; it did not save the effort of deciding what to analyze. Data could flow in automatically, but the user still had to spell out what kind of analysis to run, how to slice the data, and who the findings were meant for. Integration solved the logistics of what to feed in and how to feed it, not the more fundamental question of what the analysis was actually for.

Stage 3: Capturing Reusable Methods — No More Explaining Yourself Every Time

At some point in the year that followed, AI could already create and edit deliverables such as spreadsheets, documents, and presentations. A single conversation could take you from raw data to a polished report.

It looked like analysis could now be handed off entirely, but users soon ran into a different kind of repetitive work. How should a report be laid out? Which anomalies deserved to be flagged? Who was the final audience? Every time, these preferences had to be explained and adjusted all over again. What users really wanted was a way to save the requirements they had refined through trial and error last time and reuse them directly next time.

Reusable “analysis skills” were a direct response to this need. A skill bundles instructions, executable scripts, and reference material together, and the model reads it on demand when a relevant task comes up. When one person repeatedly tests an analysis workflow and confirms it works, they can turn it into a reusable method. They can then hand it to colleagues along with its scripts, references, and validation criteria. Those colleagues don’t have to figure everything out from scratch; they can build on an approach that has already been proven to work.

The real shift at this stage was that AI analysis output, for the first time, could turn from individual know-how into a team asset.

Back to the Original Question: Mimicking the Form Is Not the Same as Transferring the Judgment

Put the three stages together, and the puzzle from the beginning becomes easier to explain.

The reason large volumes of historical SQL had so little effect is that the historical material encodes what an answer looks like, but not why it was answered that way. Behind each query an analyst wrote were many implicit judgments: which table to choose, how to define each metric, which dirty data to exclude, and which business events caused anomalies in a given period of data. These judgments are the real source of analysis accuracy. Yet they are almost never written into the SQL itself. They live only in the analyst’s head and in the context of the conversations where the analysis took place. The model could mimic the syntax and structure, but it could not learn the business reasoning behind the choices.

This also explains why the integration and skills stages were more effective:

• Integration closes the “context availability” gap, giving the model enough first-hand information instead of secondhand conclusions.
• Skills make judgments explicit, turning analysts’ vague preferences into clear rules that can be executed and checked.

A more direct approach is to make tacit knowledge explicit. Rather than handing over a hundred historical queries that “look right,” it’s better to write out metric definitions, exclusion rules, and anomaly criteria in plain terms. With the former, the model has to guess. With the latter, it knows exactly what to do, so the results naturally differ by a wide margin.

There is another point that often gets overlooked: more context is not always better. A flood of historical material dilutes the model’s attention and introduces noise, and even definitions that contradict each other. That makes it harder for the model to zero in on the key constraints. Compared with piling on more and more material, a curated and structured context often works better.

Practical Advice for Teams

• Separate “getting data” from “setting goals”: integration capability solves input availability, but analysis goals, metric definitions, and target audiences still need to be defined by people. Don’t expect automation to replace judgment.
• Make tacit rules explicit first: writing out exclusion criteria, metric definitions, and anomaly rules in clear terms improves accuracy more than feeding in large numbers of examples.
• Turn validated workflows into assets: once a type of analysis has been refined through repeated testing, turn it into a reusable method and share it so the team doesn’t keep falling into the same traps.
• Focus on context quality, not quantity: curated, deduplicated, and structured material beats dumping in everything without filtering.
• Keep people in the loop: especially for data cleaning and explaining anomalies, timely correction from someone who understands the business is key to results you can trust.

After two years, the path of AI in data analysis is fairly clear. It has gone from helping you get a single file working, to helping you avoid moving data by hand, to helping you turn your methods into something reusable. Nearly every real improvement along the way has come from making human business judgment more available and more explicit, not from handing the model more historical material. Once teams understand that, they can put AI analysis on a path that is both trustworthy and reusable.