Local Knowledge Bases Should Retrieve on Demand, Not Flood the Context Window

Putting several megabytes of spreadsheets or tens of thousands of records into a conversation may appear to give the model access to every fact. In practice, it is one of the most expensive ways to perform a search.

Every conversation has a limited amount of available information, commonly called the context window. The window must hold the question, constraints, previous conclusions, and supporting evidence. Loading an entire file reserves most of that capacity for a single document. If the material is large, the input may be truncated or rejected with a response such as “There is too much content to process.” Even when the model manages to read everything, earlier sections can lose attention as later content fills the window. The final answer may look complete while being based on incomplete evidence.

This is not necessarily a problem with the model’s intelligence. It is a retrieval strategy applied at the wrong level.

People rarely research a document by reading every page from beginning to end. They check the table of contents, locate the relevant section, and then verify the original text. An assistant that knows only how to read files, but not what exists in the knowledge base, can do little more than scan the entire collection by brute force. The result is slow, disorganized, and prone to omissions. As the number of files grows, finding information becomes a matter of chance.

File-format handling creates another source of failure. Incorrect encoding, tables split into empty lines, and lost layout information can all turn extraction results into corrupted text. Many retrieval failures happen at the first step: the system did not fail to find the file; it failed to read the file correctly.

Three Common Misconceptions

1. “The entire knowledge base is the context”

This approach treats the knowledge base as one extremely long prompt and expects a single conversation to remember every policy, metric, and historical record. Once the context window is full, the request may time out or later content may be discarded. The user then receives incomplete reasoning.

2. “Without an index, just keep opening folders and files”

When a collection lacks navigation information such as titles, field descriptions, date ranges, and document types, retrieval degenerates into a repeated cycle of listing directories, opening files, and listing more directories.

The scanning cost grows linearly with the number of files, or worse. Missed results are difficult to detect because the system does not know what should have matched in the first place.

3. “A heavier toolchain is always better”

For semantic similarity, some teams immediately adopt heavy runtimes or require the original text to be sent to a remote service for embedding. For teams that only need to process Chinese policies, spreadsheets, and meeting notes locally, installation effort, configuration complexity, and data-transfer concerns may become obstacles before recall does.

For this type of use case, three conditions matter: lightweight operation, local processing, and reliable handling of Chinese text. If a solution cannot satisfy them, an advanced architecture may still be impractical.

Replace “Memorization” with “Browsing”

A sound architecture treats the knowledge base as queryable external storage rather than as part of the prompt.

The model should understand the question, plan what to retrieve, read a small amount of evidence, and produce the answer. The index should narrow the search space. The context for a single question should contain only:

• A rewritten or structured version of the question
• A small number of matching passages
• Necessary metadata, such as the source path, page or row number, and update time

Full text that did not match the query should not enter the context window.

A practical design can use three layers instead of introducing a vector database immediately.

Layer 1: Catalog and metadata

For each file, record its path, title, document type, approximate time range, key fields, and a short summary.

The summary can be generated through rules or one-time offline processing. Keep it to roughly a hundred Chinese characters or an equivalent short description. It is used to answer “What exists?” rather than to provide verbatim citations.

Human-maintained catalog information, such as a topic hierarchy, responsible department, or effective status, is often more reliable than automatically generated summaries. The system should therefore support manual tags.

Layer 2: Block-level indexing

Split long documents by headings, paragraphs, or logical table rows rather than cutting them into fixed-size byte ranges. Each block should retain its source location so that users can cite and verify it.

Retrieval can begin with high-recall filtering based on keywords, field conditions, and simple Chinese tokenization. A more detailed relevance ranking can then be applied to the candidate set.

For smaller datasets, an inverted index or in-memory term matching may be sufficient. Local vector search should be considered only as an additional ranking method when the candidate set remains too large or the semantic gap is significant. It should not be the only entry point.

Layer 3: Reading the source text

The model or program should open only the matching blocks and the necessary surrounding context. It can then extract figures, clauses, and exception conditions.

The context window should contain evidence, not the entire repository.

The control flow can be represented with the following pseudocode. It describes the logic rather than the interface of any particular product:

question -> identify entities, time ranges, and document types
catalog  -> use metadata to filter down to a small set of files
blocks   -> retrieve the top-K blocks by keyword or field within candidate files
read     -> read only those blocks and the necessary neighboring context
answer   -> answer from the evidence and return source locations

If Top-K is empty, the system should explicitly say that the current catalog does not cover the question and suggest adding tags or expanding the search scope. It should not pretend to remember an answer from an empty context.

Normalize Formats Before Applying Retrieval Algorithms

The principle of “garbage in, garbage out” is especially unforgiving in retrieval systems.

Tables should preserve headers and row boundaries. Merged cells should not be expanded into meaningless empty rows. Text should use a consistent encoding, and control characters used only for layout should be removed.

If a scanned document contains only images, it should not be treated as readable text. It must first undergo text recognition and should be stored with a recognition-confidence indicator.

For Chinese documents, preserve proper names, identifiers, and units. When splitting text into blocks, do not separate a clause number such as “Article 3.2” from the text that follows it.

For each source type, define a normalized representation, such as:

• Plain-text blocks
• Rows and columns with table headers
• An outline that preserves heading levels

Retrieval should operate on the normalized representation, while the original file remains available for traceability. This allows ranking strategies to change without redoing parsing. It also makes it easier to identify which type of format caused an extraction error.

Cost, Privacy, and Failure Handling

A local-first design means that original documents should remain in the local environment by default. Catalogs, blocks, and keyword indexes can be generated offline and updated incrementally as files change.

Only when there is a clear need and an approved policy should a system send a de-identified short query or short passage to an external capability. A remote vector service should not be treated as the default path.

From a cost perspective, reading entire documents makes expenses grow with document size and the number of conversation turns. Catalog filtering ties the cost to the size of the matching set instead.

In engineering practice, record how many blocks and characters each question actually loads. This is a useful signal for deciding whether the catalog needs more detail. It is more actionable than pursuing an accuracy percentage that cannot be independently verified.

Fallback behavior should be explicit:

1. If metadata filtering fails, relax the document-type conditions.
2. If keyword search returns no results, try synonyms and identifiers.
3. If there is still no match, report the coverage gap instead of expanding to the entire knowledge base.
4. Use full-database reading only for manual troubleshooting, with a strict size limit.

How to Evaluate a Local Retrieval Design

A qualified local retrieval system should meet at least four requirements.

First, every source passage cited in an answer must be traceable to a file and a specific location.

Second, documents that never enter the candidate set must not consume context-window capacity.

Third, after normalization, Chinese long-form documents and tables must be searchable by fields or clauses rather than only by filename.

Fourth, a small team should be able to maintain the catalog and perform routine question answering without installing a heavy runtime or requiring remote vectorization.

If these requirements are not met, adding a larger model or pasting more files into the conversation will only repeat the same failure: the context becomes occupied by the repository while the actual question remains unanswered.

Helping an assistant work with a knowledge base is fundamentally about separating responsibilities. The catalog answers what exists, retrieval decides which passages to read, and the model determines how to answer from the evidence. When these roles are separated, the context window becomes a resource for reasoning rather than a warehouse filled with documents.