How to Build a Reliable Knowledge Base with Retrieval-Augmented Generation (RAG)

    This article explains why LLM-based knowledge base Q&A systems still hallucinate, and how to control that risk end to end. It walks through a complete retrieval-augmented generation pipeline: document loading with metadata, structure-aware chunking, hybrid retrieval with metadata filtering and reranking, prompt-level answer boundaries, explicit refusal logic, and verifiable answers with citations. It also lists common design mistakes, their fixes, and a practical rollout sequence for turning a RAG demo into a stable production system.

    Why Knowledge Base Q&A Still Hallucinates

    Large language models are strong at understanding and generating language, but fundamentally they predict the most probable continuation of a context rather than executing strict factual queries the way a traditional database does. Even after connecting enterprise documents, the model can still fail in the following ways:

    1. No relevant content in the knowledge base, but the model answers anyway. When a user asks something outside the scope of the documents, the model may fill the gap with general knowledge acquired during training and produce a conclusion that sounds reasonable but cannot be verified.

    2. Relevant content is retrieved, but the answer adds extra inference. A retrieved passage may describe only part of a process, and the model may expand it into a complete policy, pushing the answer beyond the meaning of the source text.

    3. Retrieval results are inaccurate or poorly ordered. If the recalled fragments are not semantically similar enough to the question, the model composes an answer on top of the wrong context.

    4. The documents themselves conflict or are out of date. A knowledge base may hold both an old and a new policy at the same time. Without version, timestamp, and permission information, the model cannot determine which document it should cite.

    Reducing hallucination therefore cannot rely on prompt wording alone. It requires coordinated governance across four layers: data processing, the retrieval pipeline, model constraints, and result evaluation.

    The Overall RAG Pipeline

    A fairly complete retrieval-augmented generation flow usually includes these steps:

    1. Raw documents
    2. ↓
    3. Document parsing and cleaning
    4. ↓
    5. Text chunking and metadata enrichment
    6. ↓
    7. Vectorization and index building
    8. ↓
    9. User question rewriting
    10. ↓
    11. Similarity retrieval
    12. ↓
    13. Filtering and reranking of results
    14. ↓
    15. Evidence-grounded answer generation
    16. ↓
    17. Citation display and answer validation

    The most important principle here is:

    The model may only use retrieved, trusted content as the basis for an answer. When the evidence is insufficient, it should explicitly refuse rather than fill in the gaps itself.

    Step 1: Load and Parse Documents Properly

    The role of a document loader is to convert data from different sources and formats into a unified document object. A standard document object typically includes:

    • The main text content;
    • The document name;
    • Page or section location;
    • Creation and last-modified timestamps;
    • The document version;
    • The owning department or business scope;
    • Access level;
    • A source URL or unique identifier.

    For PDFs, for example, loading should not extract plain text only; it should also preserve page numbers, heading hierarchy, and table structure as much as possible. Answers later need to give the user a source. If only the text survives and position information is lost, even a correct answer becomes hard to trace.

    Illustrative code:

    1. document = load_document(
    2. path="docs/internal_policy.pdf",
    3. metadata={
    4. "department": "Operations",
    5. "version": "2024-01",
    6. "access_level": "internal"
    7. }
    8. )
    9. pages = document.split_by_page()

    In practice, the parsing output also needs cleaning:

    • Remove repeated headers, footers, and page numbers;
    • Fix incorrect line breaks;
    • Merge sentences broken across pages;
    • Identify headings, lists, and tables;
    • Remove meaningless whitespace;
    • Run OCR on scanned documents;
    • Log pages that failed to parse, so content is never silently lost.

    If the raw document is parsed incorrectly, both vector retrieval and answer generation downstream are affected. Document quality is the foundation of knowledge base reliability.

    Step 2: Chunk Text Sensibly to Avoid Context Distortion

    Converting an entire document into a single vector usually makes retrieval granularity too coarse. When a user asks about a specific clause, the system may only be able to recall the whole document, forcing the model to search for the answer inside a large amount of irrelevant content.

    Conversely, chunking too finely breaks semantic completeness. For example:

    • A heading gets separated from the body text it belongs to;
    • A condition gets separated from its conclusion;
    • Table headers get separated from the data;
    • An exception gets cut into a different chunk;
    • A negation gets separated from the key action.

    Chunking strategy should therefore not be driven by a fixed character count alone. Document structure should take priority. The recommended principles are:

    1. Use sections, paragraphs, lists, and tables as basic boundaries;
    2. Each chunk should express one complete topic where possible;
    3. Set a maximum length for long paragraphs;
    4. Keep a small amount of overlap between adjacent chunks;
    5. Prepend headings and section paths to each chunk;
    6. Handle tables separately, preserving field names with their corresponding values.

    Example:

    1. chunks = split_text(
    2. text=document.text,
    3. max_length=800,
    4. overlap=100,
    5. preserve_headers=True,
    6. preserve_tables=True
    7. )

    The resulting chunks can be organized like this:

    1. Section: Expense Reimbursement / Travel Accommodation
    2. Body: Travelers should claim accommodation expenses according to the standard for their city and job level...
    3. Source: internal_policy.pdf
    4. Page: 12
    5. Version: 2024-01

    This not only helps retrieval but also lets the model understand the context of the current chunk while generating an answer.

    Step 3: Build a Vector Index and Improve Recall Quality

    The basic idea of vector retrieval is to convert document chunks and user questions into vectors, then find relevant content based on how similar those vectors are.

    Vector similarity alone, however, does not solve every problem. Enterprise knowledge bases are full of keywords, IDs, product model numbers, policy clause numbers, and dates, which are often better handled by keyword search. Production systems therefore typically use hybrid retrieval:

    • Semantic retrieval: suited to questions phrased differently but similar in meaning;
    • Keyword retrieval: suited to IDs, proper nouns, and exact phrases;
    • Metadata filtering: narrowing by department, version, access level, and time range;
    • Reranking: reordering candidates with a more fine-grained relevance model.

    A reasonable retrieval flow looks like this:

    1. candidates = semantic_search(query, top_k=20)
    2. candidates += keyword_search(query, top_k=20)
    3. filtered = filter_by_metadata(
    4. candidates,
    5. user_role=current_user.role,
    6. document_status="active"
    7. )
    8. evidence = rerank(query, filtered, top_k=5)

    top_k should not be set arbitrarily high. Too many candidates increase the model’s processing load and can pull mutually conflicting, weakly relevant fragments into the context. More importantly, the system should record the score, source, and filter reason for every retrieval, so issues can be diagnosed later.

    Step 4: Define Answer Boundaries with the Prompt

    The purpose of a prompt is not to make the model “smarter” but to define its responsibilities and limits for the current task. These rules are worth putting in the system prompt:

    1. Answer only from the provided reference material;
    2. Do not speculate when the reference material does not state something;
    3. Do not substitute training memory for knowledge base evidence;
    4. Explicitly state uncertainty where content is uncertain;
    5. For questions involving time, version, or permissions, prefer the latest valid document;
    6. Attach the corresponding source after each conclusion;
    7. If multiple documents conflict, list the conflict and flag it for human confirmation;
    8. Use a consistent refusal message when an answer is not possible.

    An illustrative prompt template:

    1. You are an enterprise knowledge base Q&A assistant.
    2. Follow these rules strictly:
    3. 1. Answer questions using only the "Reference Material";
    4. 2. If the reference material does not contain the answer, reply "The current knowledge base does not contain enough evidence to answer this question";
    5. 3. Do not add conclusions based on common sense, guesses, or external information that was not provided;
    6. 4. Do not alter the dates, conditions, figures, or scope of applicability in the source text;
    7. 5. Cite the source for every key conclusion;
    8. 6. If the reference material conflicts with itself, state the conflict explicitly instead of choosing one version.
    9. User question:
    10. {question}
    11. Reference material:
    12. {context}

    Note that prompt constraints are not absolutely reliable. If the retrieval module passes in a large amount of irrelevant content, or the reference material contradicts itself, the model can still be wrong. Prompts must therefore be combined with retrieval filtering and answer validation.

    Step 5: Build an Explicit Refusal Mechanism

    A truly reliable knowledge base system does not “answer every question”; it “answers only when there is evidence.” Multiple levels of refusal conditions can be defined:

    1. Retrieval scores are too low

    If the highest relevance score falls below a preset threshold, the knowledge base probably contains no matching content. The system should return a clarification or a refusal directly instead of continuing to generate.

    2. Multiple candidates contradict each other

    For example, one document sets a reimbursement deadline of thirty days while an older document sets sixty days. The system should flag a version conflict and show the document timestamps rather than deciding on its own.

    3. The evidence does not cover the question

    A user question may contain several sub-questions while the retrieval results cover only some of them. The system should decompose the question and judge evidence coverage for each part, rather than answering everything just because one part has support.

    4. The user lacks permission

    Access control must be applied at the recall stage, not after retrieval by asking the model to hide sensitive content. Otherwise, information the user should not access has already entered the model’s context, which is a data leakage risk.

    Step 6: Make Answers Verifiable

    The output of a knowledge base Q&A system should not be a paragraph of natural language alone; it should include evidence sources. A recommended structure:

    1. Conclusion:
    2. ...
    3. Evidence:
    4. - Document: Internal Policy
    5. - Section: Expense Reimbursement / Travel Accommodation
    6. - Page: 12
    7. - Version: 2024-01
    8. Note:
    9. The conclusion above applies only to the current version and the stated scope.

    For important business scenarios, structured output can also be required:

    1. {
    2. "answer": "...",
    3. "confidence": "medium",
    4. "evidence": [
    5. {
    6. "document": "Internal Policy",
    7. "page": 12,
    8. "quote": "..."
    9. }
    10. ],
    11. "need_human_review": false
    12. }

    Structured output makes frontend rendering, log recording, and automated evaluation easier, and reduces cases where the model omits citations.

    Common Mistakes and How to Fix Them

    Mistake 1: Relying on the LLM alone

    No matter how capable the model is, it cannot automatically obtain an enterprise’s latest internal data. The fix is a stable data synchronization and version management process.

    Mistake 2: Using vector retrieval only

    Vector retrieval is not necessarily stable for exact IDs, numbers, and domain-specific terms. Combine it with keyword retrieval, metadata filtering, and reranking.

    Mistake 3: Chunks that are too long or too short

    Overly long chunks reduce matching precision; overly short ones lose meaning. Define different strategies per document type and validate them with a real set of user questions.

    Mistake 4: Passing every retrieval result to the model

    Irrelevant and conflicting content degrades answer quality. Filter, deduplicate, and rerank first, then build a bounded context.

    Mistake 5: No evaluation system

    Judging quality by human impression alone makes problems hard to localize. Build an evaluation set that covers recall accuracy, answer correctness, citation completeness, refusal accuracy, and access isolation.

    Recommended Rollout Path

    The following order works well:

    1. Curate high-quality documents and define an owner, version, and validity period for each;
    2. Establish a unified document object and metadata specification;
    3. Implement structured parsing, chunking, and indexing;
    4. Adopt hybrid retrieval combining semantic and keyword search;
    5. Add permission, version, and status filtering;
    6. Use prompts to restrict the model to evidence-based answers;
    7. Add low-confidence refusal and conflict detection;
    8. Output document sources, page numbers, and quoted fragments;
    9. Build an offline test set and a production feedback loop;
    10. Periodically remove outdated documents and re-evaluate retrieval quality.

    Summary

    Reducing hallucination in knowledge base Q&A is not simply a matter of adding a prompt. It requires building a complete evidence chain: accurate document parsing, chunking that preserves meaning, retrieval results that are relevant and permission-aware, model answers under explicit constraints, and final output that is traceable and verifiable.

    The most important design principle is: when the knowledge base has no evidence, the system must be allowed to say “I don’t know.” Only when “answer with evidence, refuse without evidence, and escalate conflicts to a human” is implemented in the system flow can the model be prevented from freely answering around the knowledge base, taking a retrieval-augmented generation solution from a demo-grade application to a stable, controllable production system.