Evaluating and Deploying LLMs for Enterprise Business Q&A

    This article provides a practical guide for assessing and deploying large language models in enterprise business Q&A systems. It breaks down how to define verifiable evaluation criteria, organize source documents with time and permission markers, separate question analysis from evidence retrieval and answer generation, and build test sets that distinguish model failures from knowledge gaps. It also covers launch acceptance risks and ongoing maintenance practices.

    Turning “Good Answer” into Verifiable Requirements

    “Please describe this company’s overall business landscape” is a useful open-ended test question. Unlike a single-fact query, it requires the model to first determine the statistical scope, then extract facts from multiple documents, and finally organize them into a clear, hierarchical explanation. A single fluent response only indicates acceptable performance on that particular task. Without unified source materials, questions, and scoring rules, one cannot conclude that the model is more reliable than a previous version.

    The most common error in such questions is not misnaming a business, but placing real information at the wrong hierarchical level. For example, a digital content service may be a business category, membership subscription is a monetization method, and a mobile entry point is a distribution channel. All three may appear in the same document, but they should not be listed as three parallel business segments. Similarly, user coverage can indicate reach, but it cannot replace revenue contribution, nor can user scale alone be used to infer the profitability of a business.

    Therefore, an enterprise business overview answer should cover at least three elements: the classification basis, the time period described, and the key evidence used. If the user is asking about operating analysis, the answer should prioritize formally disclosed business or financial segments. If the user cares about product operations, it can be organized by user scenarios or product lines. Both approaches are valid, but operational classifications should not be treated as financial classifications, nor should they be mixed without explanation.

    Organize Source Materials Before Designing Retrieval

    1. Add Timestamps, Versions, and Permission Markers to Documents

    Enterprise documents are often complex because the same name may appear in annual reports, historical announcements, business descriptions, and internal documents from different years, with meanings that may differ. When building a knowledge base, record at least the document category, publication date, content validity period, version status, source location, and access permissions. The publication date and the applicable period should be stored separately: a document published later may describe a previous reporting period, so its figures should not be treated as current data merely because the file is newer.

    Source priority should also vary by question type. For financial segments and revenue attribution, use officially disclosed materials for the corresponding period. For business processes and responsibility assignments, current policies and verified descriptions are more suitable. When two sources appear to conflict, first check whether they use different statistical scopes or definitions. Only when the scopes are consistent but conclusions still differ should the system flag a conflict, rather than letting the model arbitrarily choose one.

    2. Preserve Structure; Avoid Fragments That Lose Context

    Retrieval units need to be small enough for precise matching, yet complete enough to support judgment. When splitting text, keep the section path, and try to keep definitions, applicability conditions, and exceptions in the same retrieval unit. For tables, associate metric names, row and column headers, units, reporting periods, and relevant footnotes with the data. Extracting only a single value—even if verbatim—can lead to incorrect answers when the unit, currency, or statistical scope is missing.

    Business names may have abbreviations, historical names, or similar terms. A terminology mapping can assist retrieval, but mapping does not mean two names are equivalent across all periods. If an old name applies only to a specific period, include the time condition in the mapping rule to avoid carrying adjusted business attributions into current answers.

    Separate Question Analysis, Evidence Retrieval, and Answer Generation

    1. First Determine What the User Is Asking

    The system can break down a user question into several elements: the subject, time range, required scope, and answer granularity. For example, “what businesses do you have” focuses on classification and coverage; “what do you mainly earn from” focuses on revenue sources and monetization; “what changed this year” requires comparing at least two periods under the same scope. The three questions share some materials but have different retrieval targets; reusing the same overview can easily miss the point.

    When the user does not specify a time range and the knowledge base has a clear latest complete reporting period, answer based on that period and state it at the beginning. If “business segments” may refer to either formal disclosure segments or internal product lines, the system can ask for clarification first; if both scopes can be explained concisely, present them separately. The criterion for asking is whether the ambiguity substantially changes the answer, not whether every field in the prompt is filled.

    2. Retrieve Evidence per Sub-Question, Then Assemble the Answer

    A business overview question should not rely entirely on one broad retrieval, because high-frequency similar documents may occupy multiple candidate positions, leaving a key segment without evidence. A more reliable flow is: first find materials that define the classification framework, then query each segment for business definitions, target customers, and monetization methods. If cross-period changes are involved, separately search for adjustment notes and materials from relevant periods. Keyword search is suitable for locating formal terms, reporting periods, and metric names; semantic search helps match user colloquial expressions with formal document terms. Neither can replace version and permission filtering.

    When generating content, the structure can follow “business category—core activity—target customer—monetization method,” but do not fill in information not provided by the materials just to complete the structure. For conclusions that are easily misused—such as revenue share, year-over-year changes, or business attribution—provide locatable evidence nearby. Citation checks cannot stop at confirming that the document exists; verify that the relevant passage actually supports the conclusion. For example, a material stating that a service has been launched does not prove that it generates separate revenue, nor that it belongs to a formal financial segment.

    3. When Evidence Is Insufficient, Provide a Bounded Answer

    If the materials only confirm that an activity exists but its financial attribution cannot be verified, separate the two: “Available materials confirm the activity, but they are insufficient to determine its disclosed segment.” If segment definitions changed between two periods and no restated data under a comparable scope exists, do not directly calculate growth rates. Such an answer may look less complete than a broad overview, but it helps readers judge which parts can be adopted directly and which require further verification.

    Use Test Sets to Distinguish Model Problems from Knowledge Problems

    1. Define Verifiable Fact Points for Each Question

    Open-ended questions need not have a single standard answer, but each question should predefine required facts, acceptable expression variations, valid evidence, and typical errors. Besides business overview questions, the test set should also cover categorizing by specified period, distinguishing channels from business lines, comparing cross-period changes, handling conflicting materials, and responding to insufficient evidence. Prepare multiple phrasings of the same question to check whether the system changes factual conclusions due to wording variations.

    For internal Q&A, additionally test the same question asked by identities with different permission levels. Permission control should occur during retrieval, not by first letting the model read restricted materials and then relying on prompt instructions not to reveal them. During acceptance, check not only whether the body text exceeds permissions but also whether cited snippets, document titles, or other attached information expose restricted content.

    2. Score Coverage, Accuracy, and Citations Separately

    A single composite score easily masks real problems. Track the following metrics separately:

    • Required fact coverage rate: number of correctly answered required fact points divided by the total required fact points, used to locate omissions.
    • Factual statement accuracy: number of verified correct factual statements divided by the total factual statements checked, used to find errors actively added by the model. Statements that cannot be verified at the moment should be marked separately; they should neither be counted as correct nor judged incorrect without verification.
    • Citation support rate: number of citation relationships that directly support the corresponding statement divided by the total checked citation relationships. Also record key statements that should have citations but do not; otherwise, citing only one source can yield a high support rate.
    • Scope and time consistency: check whether the answer mixes segments, channels, and monetization methods, whether historical materials are treated as current, and whether figures and descriptions come from comparable periods.
    • Quality of handling insufficient evidence: check whether the answer accurately identifies gaps while preserving what the available materials confirm. Retaining a valid conclusion is not the same as refusing the entire question.

    When scoring, split compound sentences into independent statements. For example, “activity A belongs to segment B and contributes the main revenue” contains two judgments—attribution and revenue contribution—and both must be verified separately. Expression quality can also be scored, but it should be separate from factual accuracy: natural wording is valuable but cannot offset factual errors.

    3. Use Controlled Comparisons to Locate Failure Points

    To compare model versions, keep the question, reference materials, instructions, and output constraints identical, and have human reviewers evaluate the same set of answers. A more practical approach is to run two parallel groups: one provides already verified evidence and observes whether the model can summarize correctly; the other runs the full retrieval pipeline and observes whether the system can find all materials before generating.

    If the model answers correctly when given valid evidence but incorrectly in the normal flow, first check document collection scope, chunking strategy, retrieval ranking, and time filtering. If the correct evidence is already in the context, but the model still classifies a channel as a business segment, check the task instructions, category definitions, and model comprehension capability. This comparison is more effective than repeatedly adjusting writing style because it clarifies whether to change the materials, the retrieval, or the generation step.

    Launch Acceptance Should Cover Business Risks

    Internal knowledge navigation can allow the answer to first provide an overview and then guide the reader back to the original source. For management decisions or external publication, add manual review for numbers, important business attributions, and cross-period comparisons. The acceptance threshold should be determined by the actual use case, but risks such as permission leakage, major factual errors, and outdated scopes should have separate limits and cannot be offset by high scores on other questions.

    After launch, record material versions, evidence citations, and user corrections while complying with permission and retention requirements. Add typical failure cases to regression tests. When the reporting period changes, business attribution is adjusted, or the knowledge base is updated, re-run relevant questions and focus on whether old materials are still mistakenly treated as current and whether definitions and tables in new documents are retrievable.

    Deployment can start with a small set of high-frequency business questions and a few authoritative documents. Ensure that every key conclusion can be traced back to corresponding materials, then gradually expand coverage. The most useful signal of system improvement is not that the overview becomes longer, but that readers can verify facts more easily and maintainers can locate errors more quickly.