Local Knowledge Base: Selection, Deployment, and Common Pitfalls
This article explores how to select, deploy, and optimize a local knowledge base. It covers core components such as document parsing, embedding models, vector databases, and LLM inference engines. Furthermore, it provides a minimal runnable example, discusses RAG workflows and fallback strategies, and outlines best practices for security, performance testing, and cost estimation.
1. What Problems Does a Local Knowledge Base Solve?
A local knowledge base is a system that transforms enterprise documents, web pages, and database records into knowledge accessible by Large Language Models (LLMs). It consists of four stages: text processing, semantic representation, vector retrieval, and model inference. Compared to cloud solutions, the main benefits of local deployment are keeping data within the internal network, the ability to adjust each stage according to business needs, and continuous operation during network outages.
Implementing locally usually presents three types of challenges:
- Data Scale and Retrieval Accuracy: As the document volume grows, relying solely on keyword retrieval (like BM25) can easily miss content with similar semantics but different wording, necessitating the addition of vector retrieval.
- Hardware Resources: Both embedding models and LLMs require significant memory or VRAM, requiring a trade-off between performance and cost.
- Knowledge Updates: Documents frequently change, requiring incremental index updates while ensuring the Q&A service remains uninterrupted.
2. Core Components and Selection Strategy
A typical local knowledge base includes five modules: document parser, text chunker, embedding model, vector database, and LLM inference engine. Below are the considerations for selecting each module.
2.1 Document Parsing and Chunking
The document parser must cover PDF, Word, Markdown, HTML, and scanned files. Scanned files require an OCR engine (such as Tesseract) and should preserve layout information as much as possible. When parsing, be careful with tables and code blocks; splitting them in the middle renders the retrieved chunks meaningless.
Chunking quality directly affects retrieval effectiveness. There are two common approaches:
- Fixed-length chunking: Splits by character or token count. It is easy to implement but can easily cut a sentence in half.
- Structure-aware chunking: Splits along paragraph, heading, and code block boundaries, recursively subdividing overly long paragraphs. Keeping a certain overlap between chunks, such as about 512 tokens per chunk with an overlap of about 128 tokens, helps retain context. The specific values should be determined by testing with your own documents.
2.2 Embedding Models
Embedding models convert text into vectors, forming the basis of semantic retrieval. For local deployment, you can start with a lightweight model, such as all-MiniLM-L6-v2 based on the Sentence-BERT architecture, which has an output dimension of 384 and a small model size. If higher performance for Chinese is required, you can try Chinese or multilingual models like the bge series, but you must evaluate VRAM and inference speed.
There is no strict compatibility requirement between the embedding model and the LLM’s tokenizer; they can be selected independently. What must remain consistent is:
- Indexing and querying must use the same embedding model; otherwise, the vector spaces will be inconsistent, making retrieval results meaningless.
- Vector dimensions and distance functions (cosine, Euclidean, dot product) must be determined when building the database. Changing them midway requires rebuilding the index.
- The knowledge chunks concatenated into the LLM prompt must have a token budget based on the LLM’s context window; excess parts should be truncated or compressed.
2.3 Vector Databases
The vector database is responsible for storing vectors and performing similarity retrieval. When selecting, you can refer to the table below. The scale assessments are only empirical references; final decisions should be based on testing with your own data.
| Solution | Suitable Scale | Operations Cost | Filtering & Hybrid Retrieval | Typical Scenario |
|---|---|---|---|---|
| Milvus | Large scale, shardable clusters | Relatively high, many components | Supports scalar filtering, dense and sparse vector hybrid retrieval | Large data volumes, dedicated ops team |
| Qdrant | Medium to large scale, clusterable | Medium | Strong metadata filtering, supports sparse vectors | Online services requiring complex filtering |
| pgvector | Small to medium scale | Low, even lower if PostgreSQL exists | Direct SQL conditional filtering, combinable with full-text search | Teams already using relational databases |
| Chroma | Prototyping to small scale | Very low | Basic metadata filtering | Prototyping, personal tools |
| FAISS | Limited by memory | Requires manual implementation of persistence and filtering | Provides only vector index, filtering must be custom written | Embedded in existing programs, offline batch processing |
When selecting, focus on four metrics: P95 query latency, index building speed, support for required distance functions, and support for hybrid retrieval.
2.4 LLM Inference Engine
Locally available inference methods include Ollama, vLLM, and llama.cpp. When resources are tight, quantized models like 4-bit can be used to reduce memory footprint, but quantization introduces some quality loss, which needs to be compared on your own question set. The inference engine should ideally provide an OpenAI-compatible interface so that upper-layer applications can be called uniformly, making it easier to swap models later.
3. RAG Workflow and Fallback Strategies
Retrieval-Augmented Generation (RAG) is the most common pattern for local knowledge bases, divided into two phases:
- Offline indexing: Parse documents, chunk them, compute embeddings, and write to the vector database. Simultaneously save the original text and metadata, such as source file, update time, tags, and document version.
- Online Q&A: Convert the user’s question into a vector, retrieve Top-K candidates, optionally fuse with BM25 results (e.g., using RRF), rerank using a cross-encoder, take the top few to concatenate into the prompt, and finally call the LLM to generate an answer.
Fallback strategies determine system behavior when retrieval fails, and it is recommended to design them before going live.
- No recall: If the highest similarity of Top-K is below a threshold, directly return “No relevant content found in the knowledge base,” and do not let the LLM answer out of thin air.
- Low confidence: If the reranking score is below a threshold, provide a list of relevant documents or prompt the user to clarify the question.
- Missing citations: If conclusions in the answer cannot correspond to knowledge chunk IDs, treat them as unreliable; regenerate or refuse to answer.
Thresholds must be determined using a set of annotated questions, not set by guesswork. The system prompt can instruct the model to “answer only based on the provided knowledge chunks and annotate citation IDs,” reducing hallucinated content.
4. Deployment Architecture and Resource Estimation
Deployment methods depend on user scale and operations capability, and can be divided into three tiers:
- Prototyping: A standard workstation is sufficient, using CPU inference, maintained by one person. The focus is on running the workflow and validating document quality and Q&A effectiveness.
- Department-level: One to two servers, optionally equipped with a consumer-grade GPU, with scheduled backups and basic monitoring. Requires part-time maintenance.
- Enterprise-level: Parsing, embedding, vector retrieval, and LLM inference are deployed separately, decoupled by message queues. The vector database uses a cluster, accompanied by monitoring alerts, log collection, and security auditing. Requires dedicated operations staff.
Memory usage must be tested empirically. A lightweight prototype can usually run on a machine with 8GB of RAM, but actual usage depends on the model version, quantization method, concurrency, and index size. Before going live, record memory and VRAM data under typical load using top or ps, and document the hardware configuration and test conditions together.
5. Minimal Runnable Example
The following example applies to a Linux environment with Python 3.10 and can run without a GPU. Dependency versions should be those that pass your local tests and must be pinned in requirements.txt. The interfaces for Chroma and sentence-transformers change frequently; without version pinning, the same code may fail to run.
Step 1: Install dependencies and confirm versions:
pip install sentence-transformers chromadb fastapi uvicornpip freeze | grep -iE "sentence|chroma|fastapi"
Step 2: Build the database and write chunks. Three checks are performed here: the number of chunks must match the number of metadata entries, the vector dimension must match the model output, and stable IDs are generated using “source ID plus sequence” to prevent duplicate data on repeated runs.
import chromadbfrom sentence_transformers import SentenceTransformermodel = SentenceTransformer("all-MiniLM-L6-v2")client = chromadb.PersistentClient(path="./kb_store")collection = client.get_or_create_collection("docs", metadata={"hnsw:space": "cosine"})def index_slices(slices, metas, source_id):# Each item in metas should include source_id for bulk deletion of old slices during updatesif len(slices) != len(metas):raise ValueError("The number of slices does not match the number of metadata entries")if not slices:returnvectors = model.encode(slices).tolist()if len(vectors[0]) != model.get_sentence_embedding_dimension():raise ValueError("Vector dimension does not match model output")ids = [f"{source_id}-{i}" for i in range(len(slices))]collection.delete(where={"source_id": source_id})collection.upsert(ids=ids, embeddings=vectors, documents=slices, metadatas=metas)
Step 3: Retrieve and apply fallback strategies. This assumes that a smaller distance return value indicates higher similarity, and the threshold must be calibrated with your own data.
def retrieve(query, top_k=5, max_distance=0.5):result = collection.query(query_embeddings=model.encode([query]).tolist(),n_results=top_k,)docs = result["documents"][0]dists = result["distances"][0]hits = [d for d, dist in zip(docs, dists) if dist <= max_distance]if not hits:return None # The caller uses this to return "No relevant content found"return hits
Step 4: Run a lightweight model (e.g., qwen2.5:1.5b) using Ollama, and send the retrieved results and question combined into a prompt to the model via the HTTP interface. Step 5: Wrap a /query interface using FastAPI, internally calling retrieval and generation sequentially. The code for these two steps is consistent with the previous ones; just integrate the fallback logic for when retrieval is empty.
6. Security, Backup, and Updates
Security measures should not be limited to network isolation and authentication; they must cover the entire data lifecycle.
- Least privilege: Application accounts should only have the permissions necessary to read and write to the knowledge base. Administrative interfaces should be separate from Q&A interfaces, and model APIs should restrict source IPs.
- Desensitization rules: Use regex to match phone numbers, ID numbers, and other sensitive information before insertion. Rules will inevitably cause false positives (e.g., misidentifying a sequence number as a phone number), so periodic sampling reviews are necessary, and an encrypted copy of the original text should be kept for recovery.
- Deletion audit: When deleting a document, synchronously delete the corresponding vectors and log the operator, time, and document ID. Use
source_idin the metadata for batch deletion to prevent residual expired chunks from being retrieved. - Backup: Regularly back up the vector database directory and original documents, and conduct periodic recovery drills to ensure backup files are functional.
There are two ways to update knowledge. Full reconstruction is suitable for scenarios with small data volumes and centralized changes, simply rerunning during off-peak hours daily. Incremental updates identify changes based on file hashes or version numbers, only reprocessing newly added or modified documents. Before updating, delete the old chunks to prevent the coexistence of old and new content.
7. How to Measure Performance and Quality
Before performance tuning, you must define the measurement criteria; otherwise, optimization results cannot be compared. It is recommended to record at least the following metrics:
- P95 query latency: Measured under fixed concurrency and a fixed question set, while recording the hardware configuration.
- Recall@K: For each question, manually annotate the correct document and calculate the proportion of times it appears in the Top-K results.
- Human relevance score: Business personnel rate answers on a scale of 1 to 5, documenting the scoring criteria.
- Hardware resources: Peak and average usage of memory, VRAM, and CPU.
The question set can start with a few dozen items, covering common questions, long-tail questions, and questions with no answers. Questions with no answers are used to verify whether the fallback strategy works.
Common tuning directions include: for slow retrieval, check the index type (HNSW is usually faster than Flat but consumes more memory) or reduce vector dimensions; for poor answer quality, adjust chunk overlap, reranking thresholds, and prompts; for tight resources, switch to a quantized embedding model or reduce the inference batch size. Change only one variable at a time and compare using the same question set.
8. How to Estimate Costs
Costs mainly come from three parts: hardware, maintenance labor, and model inference time. In the prototyping phase, hardware costs are low, primarily involving the time of the validating personnel. Department-level setups need to consider backup, maintenance, and related resource costs.
