Industry Analysis with AI: Building an Evidence-Driven Workflow
Generic LLMs can draft structured industry reports quickly, but they often produce conclusions that look plausible and are hard to audit. This article explains four hard constraints of general-purpose models in industry analysis and a six-layer, evidence-driven human–AI pipeline covering data access, metric dictionaries, citation checks, multi-stakeholder views, hypothesis testing, and review.
Industry analysis is not “ask a model to write a complete-looking report.” It is knowledge work with strict requirements on evidence quality, statistical definitions, analytical viewpoints, and reasoning depth. General-purpose large language models can generate structured text quickly. In real business settings, handing the question straight to a model often yields conclusions that seem reasonable but cannot be verified. To use AI for industry analysis, embed the model in a controlled, traceable, auditable workflow. Do not treat the model as the only information source or the final judge.
1. Four hard constraints of general-purpose LLMs in industry analysis
1. Uncontrolled sources and a fragile evidence chain.
Model answers may come from training data, web retrieval, or internal synthesis, while citations often lack a traceable original source. Links may be dead, second-hand reprints, or summaries without a stated statistical definition. In industry analysis, a one-unit difference in definition, a one-quarter difference in period, or a one-region difference in sample can reverse the conclusion. AI output must therefore carry an evidence record: source, collection time, statistical scope, metric definition, location of the original table, and review status.
2. Missing definitions and metadata, so conclusions cannot be reused.
Common metrics include market size, growth rate, penetration, average order value, inventory turnover, and channel markup. The same name can mean different things across reports. If the model only summarizes text, it can splice incompatible definitions. Build a metric dictionary and a metadata layer that records, for each metric, definition, unit, time window, geography, calculation method, and source grade, and require conclusions to cite metric IDs.
3. A single viewpoint that irons out contradictions.
Real analysis must see brand, channel, investor, supply chain, and end-user slices at once. Growth for the brand can coexist with inventory pressure for the channel, valuation froth for investors, and utilization issues for supply. General models tend to emit a “mainstream consensus” and compress conflict into vague judgment. A better pattern is role-specific views, explicit conflict points, and human weighting.
4. Shallow depth: weak causal chains and little quantitative checks.
Models summarize well but do not always propose testable hypotheses. Analysis must move from phenomenon to mechanism: is growth from price, channel expansion, or repurchase? Is inventory pressure from weaker demand or stuffing the channel? Require hypotheses, supporting evidence, counter-evidence, key variables, and a validation plan—not conclusions alone. For material judgments, add human review, data recalculation, and red-team challenge.
2. Overall architecture: an evidence-driven human–AI pipeline
Split AI-assisted industry analysis into six layers: data ingest, evidence governance, retrieval and citation, multi-view analysis, validation and evaluation, and report delivery. The model does not replace the data layer. It synthesizes, compares, hypothesizes, and writes on controlled evidence.
Indicative flow:
Problem definition-> Confirm metrics and definitions-> Controlled data ingest (reports, tables, interview notes, public statistics)-> Evidence records and metadata registration-> Retrieval augmentation and citation checks-> Multi-role analysis-> Conflict points and hypothesis list-> Quantitative recalculation and human review-> Report output with evidence chain
The critical artifact is the evidence record. A generic JSON shape:
{"evidence_id": "E-2024-001","source_type": "public statistics / internal notes / channel interview","source_time": "2024-Q2","metric_id": "market_size","metric_definition": "annual market size on an ex-factory basis","unit": "CNY 100 million","region": "nationwide","sample_scope": "above-scale enterprises","raw_excerpt": "original excerpt or table locator","trace_url_or_path": "auditable path","confidence": "high/medium/low","review_status": "pending/reviewed"}
Any conclusion must cite evidence_id and state which claim it supports and what counter-evidence exists. A citation checker verifies that evidence exists, time matches, and definitions align. Failed citations must not enter the final report.
3. Data layer: controlled sources and a metric dictionary first
The first step is not asking the model. Define the problem boundary: object, time window, geography, value-chain stage, audience, and decision. Then list controlled sources—public statistics, industry reports, company filings, channel interviews, internal operating data—each with a source grade and allowed use.
Build a metric dictionary. “Market size” must state whether it is ex-factory, retail, or platform GMV. “Growth rate” must state YoY, MoM, CAGR, or comparable-store. Before generating a conclusion, match definitions. On conflict, output “not directly comparable” instead of merging. Dual check with rules plus the model:
For each conclusion:Extract cited metric_idCheck that all citations share the same definitionIf definitions differ:Flag "definition conflict"Explain the conflictForbid a single numeric conclusionIf definitions match:Allow a range estimate with an evidence list
4. Retrieval and citation: every claim must return to evidence
On controlled data, use retrieval-augmented generation: retrieve evidence snippets, then answer. Rerank and filter: prefer primary sources, time match, definition match, and auditable paths. Generation should follow “conclusion—evidence—counter-evidence—uncertainty.”
Example prompt constraints:
Answer only from the given evidence. Do not introduce external facts.Output format:1. Conclusion: no more than three sentences;2. Supporting evidence: evidence_id and key excerpts;3. Counter-evidence or conflict: contrary evidence or missing information;4. Definition notes: metric definition, time, geography;5. Confidence: high/medium/low and why.If evidence is insufficient, say "insufficient evidence." Do not fill gaps.
This reduces fabricated citations and over-generalization but does not replace humans. Monitor citation coverage, citation reachability, and definition consistency as pipeline metrics.
5. Analysis layer: multi-role views and preserved conflict
To counter a single viewpoint, analyze by role, then aggregate conflicts. Roles are analytical slices, not personas: brands care about revenue and share; channels about inventory and turns; investors about growth quality and valuation; supply chain about capacity and cost; end users about price and experience. All roles cite the same evidence pool and may disagree.
Do not force consensus. Output consensus, disagreement, and key uncertainty. Example:
Consensus: demand growth is slowing, evidence E1/E3/E7;Disagreement: whether channel inventory has peaked—channel view sees rising pressure, brand view sees better sell-through;Key uncertainty: definition differences make growth rates not directly comparable;Suggested check: add three consecutive months of channel inventory and sell-through.
Preserving conflict makes decision risk explicit. For each disagreement, require observable metrics and a validation window for follow-up.
6. Depth layer: from summary to hypotheses, mechanisms, and quantification
Shallow work is “phenomenon without mechanism.” Ask the model for a hypothesis tree: what is the phenomenon, what causes are possible, what evidence each cause needs, what is already available, what is missing, how to falsify. Recalculate key hypotheses—for example, decompose growth into price, volume, and channel expansion; cross-check inventory turns with sell-through.
Validation has three checks: data (citations exist, definitions match), logic (the conclusion follows from the evidence), and adversary (the model must seek counter-evidence and alternative explanations). The final report should attach evidence, hypothesis, conflict, and review tables so analysis is reproducible, auditable, and iterable.
7. Implementation advice
Start small: monthly tracking of one industry, competitor comparison, or channel inventory. Close the loop of evidence records, citation checks, multi-view output, and human review, then expand sources. Useful metrics: citation coverage, traceability, definition consistency, human-review pass rate, conflict detection, and recalculation error. Treat model output as a draft. Material conclusions need evidence re-check and human confirmation.
The right posture is not “let the model give the answer.” Put the model in an evidence-driven engineering workflow: constrain input with controlled data, constrain process with metadata and citation checks, keep conflict with multi-role views, and constrain output with validation. That uses the model’s synthesis and writing while keeping industry analysis credible, inspectable, and reusable.
