Intelligent Analytics Agents: Lessons from Two Years of Data Analysis Practice

Over the past two years, the role of intelligent analytics agents in data analysis has shifted clearly: from reading a single table and writing scripts, to connecting business systems for background context, to generating deliverables and reusing analysis skills. As capabilities expanded, a common misconception followed: if an agent is good at reading code and queries, opening a large corpus of historical queries should let it learn analysis automatically.

One data team tried this without much success. They exposed a large set of historical queries and confirmed from execution logs that the agent actually read them. Accuracy improved by less than one percentage point. Historical queries are real analyst assets, and agents can read code—so why did “following existing analysis” fail to help much?

Answering that requires the capability path of analytics agents, the gap between “context supply” and “analysis skill,” and then engineering methods.

1. Capability evolution: from single-file work to reusable assets

1.1 Interactive single-file analysis: lowering the first barrier

Early agents delivered the most immediate value when users uploaded a spreadsheet, then the agent wrote and ran scripts for cleaning, calculation, and charts. Users described needs in natural language and iterated to a first-cut analysis instead of looking up functions and chart APIs.

This worked because users usually knew their data: which file to upload, what fields meant, which rows to exclude. When the agent slipped, users could catch it. Accuracy came from “a user who knows the data + an interactive agent,” not from the agent alone.

That is why many teams started with narrow tasks such as budget vs. actuals or regional sales comparison. Clear task boundaries, controllable data scope, and results that humans can judge were the conditions for success.

1.2 Connecting business systems: less file shuffling, not business understanding

Agents then began reaching business tools via remote connectors or integrations—project history, task status, organizational knowledge. Users no longer had to download and re-upload files; the agent could fetch related information.

That solves document logistics, not metric definitions. Finding documents does not mean knowing which metric definition to use, which anomalies to drop, how to align time windows, or how to handle business exceptions. After more context is available, correctness depends even more on metadata, definitions, and constraints. Without them, the agent can get lost in more context.

1.3 Deliverable generation and skill reuse: from one-off analysis to organizational assets

More recently, agents can create and edit spreadsheets, documents, and presentations, so a conversation can run from data prep through a report. Users then hit a new kind of repetition: restating layout, which anomalies to flag, and who the audience is.

Reusable skill packs became a natural need. Procedures, scripts, references, and checks can live together and be loaded for related tasks. Steps that one person iterated and validated can be packaged with scripts, references, and checks and shared. Analysis experience then lives as versioned organizational assets, not only inside a chat.

2. Why historical queries barely improved accuracy

2.1 Semantic gap: a query is not a business definition

Historical SQL records how something was computed then, not necessarily why. A sales table may include refunds, test orders, and period adjustments. A query may filter them out without documenting the reason. The agent may copy surface structure and still not know whether a new question should use the same definition.

Reusable assets should include at least: the query, metric definitions, field meanings, time grain, filters, business exceptions, how to validate results, and applicability bounds.

2.2 Retrieved-and-related is not executable

A common pattern is to let the agent retrieve “related” queries. Related is not executable: definitions may have changed; only some tables may be covered; parameters may be hardcoded; several related queries may conflict.

Without metadata filters, versioning, and conflict resolution, the agent gets “related but not directly runnable” material. More context can mean more noise.

2.3 Execution logs are not understanding

Logs prove the agent read files, not that it understood them. Reading is a behavior signal; understanding needs outcome evidence: correct definitions, exception handling, totals checks, and explained variances. Without assertion-style checks, gold results, and sampling, logs create the illusion of learning.

2.4 Context overload and evaluation bias

Historical queries are often numerous, inconsistent, and uncommented. Dumping them increases overload: the agent sifts low-density text and may miss the real definition. If evaluation uses a few easy questions or only checks that a number is close, without process and applicability, historical assets look more valuable than they are.

3. Engineering: turn experience into verifiable assets

3.1 Layer assets: queries, definitions, skills, evaluation

• Raw query layer: keep historical SQL with version, author, valid time range, and related tables.
• Definition layer: metric definitions, field meanings, filters, exceptions, known issues.
• Skill-pack layer: executable steps for a class of tasks—inputs, procedure, checks, references.
• Evaluation-case layer: fixed questions, expected results, and check rules.

The agent should not face every raw query. It locates a skill pack via metadata, then loads needed definitions and reference queries.

3.2 Skill-pack structure (illustrative)

A skill pack is not a loose document; it includes executable steps and verifiable checks. Generic sketch, not tied to a specific API:

skill:
  name: regional_sales_variance
  version: 1.0
  intent: Compare regional sales performance and flag anomalies
  inputs:
    - sales_table
    - region_mapping
    - metric_definition
  procedure:
    - step: Validate time range and region mapping completeness
    - step: Compute sales and year-over-year variance with a single definition
    - step: Flag regions beyond a threshold
    - step: Produce conclusions and open questions
  checks:
    - type: row_count_nonzero
    - type: total_matches_source
    - type: no_unmapped_region
    - type: anomaly_has_reason
  references:
    - metric_definition.md
    - known_exceptions.md

The value is freezing previously iterated requirements and running checks with the steps. For a matching task, the agent loads the pack, then executes and validates.

3.3 Retrieve with constraints: filter, plan, then verify

Do not dump queries unordered. Filter by metadata, rank semantically, load skill packs and definitions, plan, validate before run, assert after run.

candidates = metadata_filter(query, domain, time_range, table_lineage)
ranked = semantic_rank(candidates, query)
context = []
for item in ranked[:k]:
    if item.conflicts_with_policy:
        continue
    context.append(load_skill_or_reference(item))
plan = llm_plan(query, context)
validate_plan(plan)          # tables, fields, definitions, permissions
result = execute_sandbox(plan)
assert_checks(result)        # totals, row counts, nulls, anomaly explanations

The point is not “see more,” but “see the right things under constraints.” Metadata cuts noise; skill packs supply steps; definitions bound the problem; sandboxing limits risk; assertions provide outcome evidence.

3.4 Verification loop and metrics

Each analysis should leave an audit trail: inputs used, skill-pack version, checks run, anomalies flagged, conclusions needing human sign-off. Metrics can include:

• Result accuracy vs. human review
• Reproducibility on the same question over time
• Rework: extra explanations or edits
• Explainability: trace conclusions to definitions, queries, and checks
• Latency and cost within acceptable bounds

These support continuous improvement: which packs fail, which definitions are missing, which checks miss exceptions—feed that back into the asset layers.

4. Practical recommendations

First, start with high-value narrow scenarios: clear data scope, verifiable results, cheap human review—budget variance, regional comparison, anomalous orders.

Second, keep humans in the loop. The agent executes, organizes, and drafts explanations; people confirm definitions, handle exceptions, and make the final call. Reading materials is not independent analysis.

Third, permissions and audit. Historical queries and business data can be sensitive. Run in a controlled environment and retain inputs, plans, checks, and results.

Fourth, version skill packs. Definitions and schemas change; packs need versions, change logs, and applicability bounds. When sharing, share checks and known limits too.

Fifth, build an evaluation case library. Assess with fixed questions instead of one-off demos.

5. Conclusion

Two years of evolution show that analysis quality is not determined by context volume. Single-file analysis relies on users who know the data; system connections improve access; deliverables improve throughput; skill packs capture experience. Historical queries are only part of the asset. Without definitions, bounds, and a verification loop, opening lots of SQL does not become accuracy.

A more workable path is layered management of queries, definitions, skills, and evaluation; constrained retrieval and execution; checkpoints and metrics that prove the agent understood and reproduced the analysis. That is how agents move from looking like they can analyze to completing analysis in an auditable, reusable, verifiable way.