Build a Unified Knowledge Preparation Layer for RAG Systems

When an enterprise connects a large language model to internal knowledge, what usually limits quality is not the model itself but data that is not ready. Retrieval-augmented generation (RAG) depends on knowledge units that can be retrieved, cited, and updated. If source material is messy, chunked arbitrarily, or poorly versioned, even a strong retriever will send the wrong passages into the context. A dedicated knowledge preparation layer cleans multi-source material, chunks it sensibly, attaches metadata, and supplies it in one consistent form to applications such as chat, coding assistance, customer support, and internal search.

Why a separate data layer is needed

Putting cleaning logic inside every application creates three problems. First, the same policy may be chunked differently for a support bot and a developer assistant, so their answers conflict. Second, source updates cannot be synchronized incrementally, and outdated clauses remain in the vector store. Third, permissions, sensitivity, and expiry are not carried with each passage, so the retrieval stage cannot filter them.

The data layer should be limited to ingestion, parsing, normalization, chunking, enrichment, quality checks, publishing, and retirement. Answer generation, tool calls, and permission decisions stay with the application and the gateway. When the vector store or chat entry point changes, the parsing pipeline does not have to be rebuilt.

Which materials the input side should cover

Common enterprise material falls into a few categories rather than one fixed format:

• Structured references: API specifications, data dictionaries, and configuration examples, which fit chunking by resource, field, and error code.
• Long documents: policies, manuals, slide decks, e-books, and office files, which need heading hierarchy plus page or slide numbers.
• Code and notes: README files, comment-heavy modules, and interactive notebooks, which should keep path, language, and symbol names.
• Dynamic content: documentation sites, feeds, enterprise knowledge-base pages, and collaboration-space exports, where links and update times matter most.
• Non-text: meeting recordings and training videos should be transcribed and aligned with timestamps so an entire subtitle track does not become one huge chunk.
• Conversation logs: chat exports contain greetings and outdated conclusions. They should not enter the formal store by default; summarize, deduplicate, and confirm them by rules or review first.

Ingestion should land on one intermediate representation: plain text, heading path, source identifier, time, author or responsible team, visibility, and an original locator (page, line, timestamp, or anchor). Later chunking and retrieval depend only on this layer, so each format does not need its own retrieval logic.

Cleaning: remove noise that should not be cited

Cleaning is not rewriting prose to sound better. It removes content that would pollute retrieval.

1. Drop navigation, headers and footers, copyright notices, repeated tables of contents, and “previous/next” links.
2. Join sentences broken by line wraps, while keeping the boundaries of code blocks, tables, and lists so example commands are not mixed into body text.
3. Mark low-confidence transcription from scans for human review instead of embedding it directly.
4. For tables, keep the mapping between headers and cells. Short “field: value” sentences can help both keyword and vector search.
5. Redact before storage: national ID numbers, secrets, private network addresses, and personal contact details should be replaced or removed. The data layer should reject obvious secret patterns by default rather than relying on each application to remember.

Cleaning rules must be versioned. After a rule change, affected documents should be reprocessed, not only new documents.

Chunking: follow semantic boundaries, not a fixed length

Fixed-length chunks are simple, but they split a premise from its conclusion or glue unrelated sections together. A layered approach is more reliable:

• First form parent chunks from headings, API resources, functions, or slides. A parent keeps full context and is not sent wholesale into the prompt.
• Then split into child chunks by paragraph, step, table row, or code symbol. Keep child length within what the model context can accept—often a few hundred to a little over a thousand words—and tune it by language and domain through retrieval tests rather than adopting one “optimal” number.
• Each child carries its parent identifier. After a child hit, the parent summary or neighboring steps can be filled in so the passage is not taken out of context.
• Add overlap only on boundary sentences of narrative paragraphs. Do not overlap code or tables; duplicate hits waste ranking slots.

The shape below is a data example, not an interface for any particular platform:

{
  "doc_id": "policy-leave-2024",
  "parent_id": "sec-3",
  "chunk_id": "sec-3-p2",
  "text": "Annual leave must be submitted in the system at least three business days in advance...",
  "heading_path": ["HR policy", "Leave", "Annual leave"],
  "source_uri": "internal://hr/leave",
  "updated_at": "2024-11-01",
  "acl": ["hr", "all-employees"],
  "locator": "page:12"
}

Enrichment and indexing

After chunking and before indexing, use light enrichment. Do not let a model freely rewrite the source:

• Generate a one-sentence title and keywords for each child chunk to support sparse retrieval.
• Extract entities such as policy names, API names, error codes, and product modules. Leave a field empty if extraction fails; do not invent values.
• Store a content hash and source version. If the hash is unchanged, skip embedding and avoid repeat computation.
• Keep embeddings, the keyword index, and the original text separate. Store originals in object storage or a document store, and store vectors plus filter fields in the vector store. The embedding model can then change while the source text remains.

Filter fields should at least include tenant, sensitivity, validity period, language, document type, and update time. Without them, later queries cannot be limited to current policy or to content a given team may see.

When publishing to several applications, do not copy the source text for each chat entry point. Publish the same chunks with different views: a coding assistant may emphasize APIs and code symbols, while a support assistant emphasizes public explanations and hides internal notes. A view is a filter and a field projection, not a second cleaning pipeline.

Quality checks

Before release, regress against a small set of real questions instead of only counting chunks:

• Whether hits land in the correct section and whether the citation locator opens the source.
• Whether a newer version outranks an expired one, and whether conflicting clauses are flagged rather than silently merged.
• The share of empty, very short, and duplicate chunks, and whether code is cut in the middle of a syntactic unit.
• Whether queries from unauthorized users are filtered before retrieval rather than masked after generation.

A batch that fails quality checks should be blocked as a whole. Partial success leaves old and new rules online together and makes troubleshooting more expensive.

Updates, deletion, and cost

The preparation layer must support incremental updates. Use source modification time and content hash to detect changes. Deletion or permission revocation must delete or hide the corresponding vectors; deleting only the original text is not enough. Conversation-style sources should have a shorter retention period and stay isolated from the formal document store so a casual reply does not override policy text.

Cost is mainly parsing, embedding, and storage. Cache parse results and re-embed only changed chunks. For long video, extract key segments before transcription. Monitor failure rate, queue delay, abnormal chunk counts per document, and empty-retrieval rate—not merely whether a job finished.

Object storage for source text, a message queue for parsing jobs, a container platform for workers, and logs plus monitoring alerts for failures are general capabilities. Choose them according to data classification and access control rather than hard-coding one vendor’s console steps into the workflow. A cloud platform such as Baidu AI Cloud can host storage, compute, and observability components, but whether the knowledge is trustworthy still depends on chunking, permissions, and version policy being enforced in the data layer.

A practical rollout order

1. Start with one high-frequency document type that has a clear owner, such as a public manual or an API reference, and connect ingestion through citation locators.
2. Add hash, validity, and permission fields, then bring in a second source type.
3. Leave conversation logs and video until later, and send them to a candidate store by default rather than the formal store.
4. Compare hit quality before and after rule changes on a fixed question set, and version the rules in the repository.

The ceiling of a RAG system is set by knowledge units that can be cited. Make cleaning, chunking, labeling, checking, and publishing an independent data layer so applications can change models and entry points without repeating the dirty work each time.