Building a Personal Knowledge Base with Local Markdown Notes and AI Agents: A Practical Guide

5 hr ago0 views

This guide explains how to turn a folder of local Markdown notes into a personal knowledge base that both you and an AI agent can read and write. It covers a four-layer architecture with a rebuildable index, vault structure and metadata conventions, connecting chat and embedding models, chunking and incremental indexing with vector search, evaluation sets, everyday workflows such as cited Q&A and periodic cleanup, and the security and permission boundaries to set before you connect any model.

1. Start With the Why: Local Markdown Plus an AI Agent

Most note collections stall at the “write but never read” stage. Four pain points come up again and again:

Notes are scattered across note apps, chat histories, and screenshots, and recalling keywords gets harder over time;
The collection keeps growing, the same topics get recorded repeatedly, and earlier and later opinions contradict each other;
When a topic spans many notes, piecing the whole picture together quickly is difficult;
If a note platform shuts down, or its export format locks you in, years of accumulated content become a migration risk.

The combination of local Markdown and an AI agent targets exactly these problems:

You control the data. Everything is plain text on local disk. Where it lives, whether it syncs, and how it is backed up are all your decisions.
The format is open. Markdown is a general-purpose text format that mainstream editors can open, so material accumulated over a decade stays readable.
It is machine-readable. Plain text is naturally suited to program parsing, full-text search, and semantic vectorization.
Tools stay decoupled. Editor, version control, search programs, and model services each handle one part; any of them can be swapped, so you are never locked into a single application.

One clarification on terminology: “AI coding tool” here means any assistant or agent that can understand instructions and read and write files, search a codebase, and run commands. These usually ship as editor plug-ins or command-line programs. Once embedded in your local note repository, the knowledge base upgrades from “a document collection people browse” to “a knowledge foundation that humans and AI read and write together.”

2. Overall Architecture: Four Decoupled Layers, an Index You Can Always Rebuild

The whole setup splits into four layers, bottom to top:

┌───────────────────────────────────────────────┐
│ Interaction: note editor + agent (Q&A /        │
│              organizing / writing)             │
├───────────────────────────────────────────────┤
│ Retrieval: keyword search + vector search →    │
│            fused ranking → citation tracing    │
├───────────────────────────────────────────────┤
│ Indexing: parse → chunk → embed →              │
│           incremental update                     │
├───────────────────────────────────────────────┤
│ Storage: local Markdown vault (notes,          │
│          attachments, version snapshots)       │
└───────────────────────────────────────────────┘

A few design principles are worth settling before you start building:

The raw files are the single source of truth. Indexes, vector stores, and summaries are derived data and can be deleted and rebuilt at any time.
Each layer is independently replaceable. Switching editors or model services leaves the note vault untouched.
Citations must be traceable. Every conclusion the AI produces should point back to a specific file and section.
Least privilege. Programs can only access authorized directories by default, and sensitive content is isolated from the start.

3. Step 1: Turn the Vault Into a Clean Data Foundation

3.1 Directory Layout: Numeric Prefixes Instead of Deep Nesting

my-vault/
├─ 00_inbox/          # quick capture, to be sorted later
├─ 10_notes/          # atomic permanent notes (one concept per file)
├─ 20_literature/     # papers, articles, web clippings
├─ 30_projects/       # project materials and process records
├─ 40_daily/          # journals and weekly reviews
├─ 50_moc/            # maps of content (index notes)
└─ 90_assets/         # images, PDFs, and other attachments

Numeric prefixes keep ordering stable and consistent across platforms. Two levels of depth is about right: search programs scan directories faster, and mistakes are less likely.

Each directory has a distinct job:

00_inbox is the buffer. Fragments captured on the fly land here and get organized periodically.
10_notes holds distilled permanent notes. One file explains one concept, which makes it easy for other notes to reference repeatedly.
50_moc holds maps of content, one per topic, listing links to every note under that topic plus a one-line summary. It is both your table of contents and the agent’s “routing table” during retrieval.

3.2 Naming and Metadata: Fast Lookup for Humans and Machines

Name files YYYYMMDD-topic-keywords.md. That sorts naturally by date and avoids collisions. At the top of every note, describe its attributes with a YAML front matter block:

---
title: Practical recall tuning for vector retrieval
type: permanent          # inbox / permanent / literature / daily / project
tags: [retrieval, vectorization, evaluation]
aliases: [vector recall, semantic retrieval]
source:                  # article URL, book chapter, or conversation source
created: 2026-03-12
updated: 2026-05-08
status: growing          # seed (draft) / growing / evergreen
---

The metadata looks like overhead, but the payoff is substantial:

tags and type let the search program filter by kind or label — search technical notes only, for instance, or exclude journals;
aliases merge the several names people use for one concept, which matters whenever terminology drifts (for example, “large language model” vs. “LLM”);
created and updated give periodic audits something to work with, such as “notes not touched in over a year”;
source records where content came from, which prevents quoting things out of context later.

Actively keep certain content out of the vault: empty files titled “Untitled” or “New Note”, duplicate notes with identical or near-identical content, and temporary information that is long expired. They crowd out search results and lower the credibility of AI answers.

3.3 Bidirectional Links: Weaving Knowledge Into a Network

Use [[title of another note]] inside note bodies to create bidirectional links, and the notes form a directed graph:

Orphan notes (nothing links to them) are usually the least valuable, or the ones that most need more connections;
Frequently referenced notes are usually worth promoting to entry points in a map of content;
An agent that traverses the link graph can start from one concept and find the supporting arguments and examples.

3.4 Version Control and Backups: Every Change Stays Reversible

Put the vault under version control. Three benefits follow: every change has a commit record you can diff line by line; content that gets deleted or mangled can be rolled back; and sync conflicts across devices can be traced clearly.

Write short, descriptive commit messages, for example “Add: practical recall tuning for vector retrieval”. Large attachments such as images and PDFs are big and binary, and committing them often bloats repository history fast. Keep them in a separate attachment directory (or cloud object storage) and link to them from notes.

Back up with the 3-2-1 rule: at least 3 copies of the data, on 2 different media, with 1 kept off-site. Beyond the local vault, sync periodically to cloud object storage and enable its versioning feature.

Example ignore rules (put them in the ignore file at the repository root):

.trash/          # editor trash
.DS_Store        # auto-generated by some operating systems
*.tmp
.env             # secrets and sensitive config; never commit

4. Step 2: Let the LLM Live Inside Your Vault

4.1 Two Common Ways to Connect

Option 1: An in-editor agent plug-in. Many local Markdown note apps have a plug-in system that lets third-party extensions read and write repository files. Once installed, the vault becomes the agent’s working directory: it can read the current note directly, search the whole collection for keywords, and write or modify files as instructed. The advantage is what-you-see-is-what-you-get, which suits asking questions while you write.

Option 2: A command-line AI coding tool. Many AI coding tools run from a terminal and can be pointed at any model service that is compatible with the OpenAI API specification. Launch one inside the vault directory and that directory becomes its workspace — a good fit for batch cleanup, scheduled jobs, and other automation.

Security note: before installing any third-party plug-in, verify that its code repository comes from a trustworthy source and check which permissions it declares. When you enable switches such as “external script access” or “confirmation for dangerous commands”, be clear about what you are authorizing.

4.2 Model Access: Route by Privacy Level

Mainstream model services generally expose two kinds of endpoints:

a chat endpoint, which handles understanding the question and composing the answer;
an embedding endpoint, which converts text into vectors for semantic retrieval.

Route by how sensitive the notes are:

Note type	Recommended access	Rationale
Public technical notes, reading notes	Cloud API	Ample capability, no operations burden, pay as you go
Internal company material, personal sensitive data	Self-hosted open-source model	Data never leaves the internal network
High-volume, lower-quality-bar tasks	Small local model plus caching	Lower per-call cost

Whichever provider you use, read its data-usage terms — for example, whether your input data may be used for model training — and decide from that which content you are willing to upload.

On the calling convention: the OpenAI API specification has become the de facto common standard, and most mainstream model services are compatible with it. So as long as the tool layer is configured with an endpoint and a key, you can switch between models without being tied to a single vendor.

Store keys in environment variables rather than hard-coding them into a file:

# Configure in a way compatible with the OpenAI API spec (illustrative)
export API_KEY="your-api-key"
export BASE_URL="https://your-model-endpoint"   # usually https://service-domain/v1
export MODEL="model-name"

4.3 Context Management: Retrieve on Demand, Don’t Dump Everything

The first instinct after connecting AI is often “hand it all my notes at once”, and that is a common mistake. For one thing, large volumes of text are billed per token, so every conversation pays for the same content again. For another, a model can only use a limited amount of context effectively per pass: the longer the context, the more easily information in the middle gets ignored — the phenomenon the industry calls “lost in the middle”.

The more sensible approach is to let the agent read on demand:

Locate candidate files from metadata, titles, and maps of content;
Read only the few most relevant notes, drilling into specific sections only when needed;
Attach source citations to the answer.

4.4 Pin the Rules Down in a Collaboration Convention File

Put a Markdown collaboration convention file at the vault root (you might call it AI_GUIDE.md) and tell the agent the rules of this repository. For example:

# Vault Collaboration Convention
## Directory meanings
- 00_inbox: unsorted, writable; 10_notes: permanent notes, confirm before editing;
- 90_assets: attachments, read-only.
## Writing rules
- One concept per file; titles use a "noun + topic" format;
- Every file must start with front matter; see `templates/note.md` for fields;
- Use [[bidirectional links]] when referencing other notes.
## Answer rules
- Every conclusion must cite the source file and section path;
- If no evidence exists, state "not found in the notes" — do not speculate;
- For modifications to files outside 00_inbox, give a change plan first and wait
  for confirmation before executing.

Most AI coding tools support convention files like this, though the exact filename differs between tools, so check the documentation for the one you use.

5. Step 3: Help the AI Actually Understand the Notes — Parsing, Chunking, and Indexing

5.1 Parsing and Chunking

Long notes cannot be sent to the model whole; they need to be split into chunks of a reasonable size:

Chunk along the Markdown structure. Use headings (#, ##, and so on) as the primary boundaries. One section is one chunk: semantically complete, and much less likely to slice a table or code block in half.
Control chunk size. Aim for 300–800 tokens per chunk, with 10%–20% overlap between neighboring chunks so sentences crossing a boundary do not lose their context.
Inherit context. Give every chunk its file path, full heading hierarchy (for example “Retrieval Optimization > Evaluation Metrics”), and front matter, so retrieval can filter by them and display the source.
Handle special content. Convert tables into “field: value” text; keep code blocks verbatim and label the programming language.

5.2 Incremental Indexing

The vault changes daily, and rebuilding the whole index every time is both slow and expensive. Use version control to identify which files changed, and process only those:

from pathlib import Path
def incremental_index(vault_dir, index_store):
    """Index only new or modified Markdown files."""
    for md_file in Path(vault_dir).rglob("*.md"):
        if is_ignored(md_file):          # skip templates, asset notes, etc.
            continue
        file_hash = sha256_of(md_file)   # hash the file contents
        # Skip if unchanged; otherwise clear the old index entries and rebuild
        if index_store.get_hash(md_file) == file_hash:
            continue
        index_store.remove_blocks_of(md_file)
        blocks = split_by_heading(md_file)   # chunk along the heading structure
        for block in blocks:
            meta = extract_front_matter(md_file)  # title, tags, and other metadata
            embedding = embed_text(               # call the embedding endpoint
                f"{meta['title']}\n{block.heading_path}\n{block.text}"
            )
            index_store.add(block, meta, embedding, file_hash)

Two points matter in practice:

Indexing must be idempotent. For each file, clear its old entries before rebuilding. Then even if a run is interrupted and restarted, you get no duplicates.
Deletes must cascade. When a file is deleted or moved, clean up its index entries too; otherwise the AI cites content that no longer exists.

5.3 Vector Index: Speed Up Retrieval With Approximate Nearest Neighbors

Once chunks are embedded, retrieval becomes “find the chunks whose vectors are most similar to the question vector”. Beyond a few tens of thousands of notes, comparing one by one is very slow, so vector search typically relies on an approximate nearest neighbor (ANN) algorithm — for example a hierarchical navigable small world (HNSW) index, or an inverted file (IVF) index that clusters first and searches within clusters. Both trade a tiny loss of accuracy for a large gain in speed.

Personal vaults are usually small enough that a local embedded vector store is sufficient. If you need to share across devices, or use the knowledge base as a team, a general-purpose vector database service from a cloud provider is worth considering.

5.4 Retrieval Optimization for Languages Without Word Boundaries

Chinese has no spaces between words, and pure vector retrieval tends to struggle in these situations: abbreviations of proper nouns, personal names, product model numbers, and the different names people use for the same concept. Three measures help:

Keep a keyword retrieval channel. An inverted index with ranking algorithms such as BM25 matches terms and identifiers exactly;
Build a synonym and alias table. Map “large model / large language model / LLM” and similar variants onto one concept;
Enable query rewriting. Before retrieval, have the model generate several related phrasings grounded in the vault. “How do I make search more accurate” can be rewritten as “recall tuning” or “reranking” to widen the recall window.

5.5 Evaluation Sets: No Measurement, No Optimization

Prepare 30–50 question-answer pairs in a 99_eval/ directory, each recording a question, the reference answer, and the file paths that should be cited. After changing the chunking strategy, swapping the embedding model, or editing prompts, run the evaluation again:

---
qid: eval-017
question: Which method did I use to evaluate recall in vector retrieval?
expected_files: [20260312-practical-recall-tuning-for-vector-retrieval.md]
keywords: [Recall@k, evaluation set, reranking]
---

Record at least three metrics per run: Recall@5 (the share of cases where the correct answer appears in the top 5 results), citation accuracy, and correct refusal rate (whether the system says “I don’t know” when it should). With those numbers, optimization stops being guesswork.

6. Step 4: Put the Knowledge Base to Work in Your Daily Workflow

6.1 Cited Q&A With Source Tracing

Add enough constraints when you ask. Instead of “summarize this for me”, try “summarize all notes in 10_notes about vector retrieval written after 2025, grouped by method, with the core conclusion of each”. The more precise the conditions, the more precise the retrieval scope, and the more reliable the answer.

Every answer should carry citations with file paths and section titles. Before you adopt a conclusion, open the original and check it — that is the most effective defense against model hallucination.

6.2 Quick Capture and Periodic Cleanup

When something comes to mind, drop it into 00_inbox. A sentence or two is enough, and it does not interrupt what you are doing. Then set aside one session a week for cleanup, driven by a prompt template like this:

Read the files in 00_inbox created before today and, for each one:
1. Distill the core point into a single sentence;
2. Assign it to a suitable topic (refer to existing topics under 50_moc), and
   suggest a new one if none fits;
3. Rewrite it as a standalone permanent note in 10_notes and complete the
   front matter;
4. Find related notes already in 10_notes and add bidirectional links;
5. List anything you are unsure about — do not guess.

AI-generated output needs human confirmation. Judgments such as “which topic does this belong to” and “which notes is it related to” deserve a line-by-line review before you apply them. A wrong association, once it enters the note network, gets amplified in later retrieval.

6.3 Periodic Knowledge Audits

Use an AI coding tool to give the vault a regular checkup:

Find orphans: list notes with no inbound links and decide whether to connect or delete them;
Find duplicates: compare title and summary similarity, then merge or clearly differentiate near-duplicates;
Find stale notes: list notes untouched for over a year and confirm whether their conclusions still hold;
Find broken links: check for links pointing to renamed or deleted files.

Once the vault reaches a few thousand notes, wire these checks into a CI pipeline so they run on every commit. Prompt alerts on orphans and broken links keep small problems from piling up.

6.4 Cost and Performance Control

Cache embedding results. Use the text content hash as the cache key so identical content never hits the embedding endpoint twice;
Batch requests. Prefer batch embedding endpoints to cut call count and unit cost;
Tiered questioning. Answer simple questions with keyword retrieval only; bring in vector retrieval and reranking for harder ones;
Watch usage. Log daily call volume and spend, set threshold alerts, and keep a runaway job from blowing up the bill.

7. Security and Permissions: The More It Knows, the Tighter the Boundaries

Notes typically mix work material, personal identity information, and financial data. Once AI is connected, draw the security boundary in advance:

Classify the data. Sort notes into public, internal, and sensitive. Content involving keys, ID numbers, or customer privacy never enters any retrieval path that makes outbound network requests;
Isolate at the directory level. Keep sensitive notes in a directory the agent cannot access, or in a separate repository entirely;
Tier operation permissions. Search and read are open by default; modify, delete, and run commands require explicit confirmation each time;
Manage keys. Keys live only in environment variables or the system keychain and never appear in notes or repository history. Run a secret-scanning tool before pushing;
Audit operations. Use a cloud provider’s logging service to record model calls and index updates, so abnormal usage and unauthorized access are easy to spot;
Hold the line when you move to the cloud. If the knowledge base extends into the cloud — a managed vector database, scheduled jobs running in cloud functions, monitoring and alerting services tracking index status — make sure data is encrypted in transit and at rest, and grant access credentials under the least-privilege principle.

8. Troubleshooting

Symptom	Likely cause	Fix
Generic answers with no citations	Nothing relevant found, or the prompt never asked for sources	Check keywords and tags; require source citations in the prompt
Wrong files cited	Chunks too large, mixing several topics	Re-chunk by heading level and shrink chunk size
Proper nouns not found	No keyword retrieval channel	Add BM25 keyword search and extend the synonym table
Answers invent notes that don’t exist	Model hallucination	Require “say you don’t know without evidence”; verify citations
API cost suddenly spikes	Full index rebuild, or a call loop	Switch to incremental indexing; add caching and usage alerts

9. Getting Started Checklist and Closing Thoughts

The whole plan compresses into five steps:

Create a Markdown repository with the suggested directory layout and put it under version control;
Install a plug-in with AI chat and repository read/write in your local Markdown note app, or launch a command-line agent in the vault directory;
Configure the model endpoint and key, choosing a cloud or local model by privacy level;
Write the collaboration convention file, and start by granting read and search access only;
Build an evaluation set and begin with one high-frequency scenario, such as clearing 00_inbox or summarizing notes by topic.

The value of a personal knowledge base does not come from how many tools are installed, but from whether knowledge keeps accumulating and can be retrieved accurately when you need it. Local Markdown keeps the data open and under your control; retrieval and indexing let the AI understand the content; collaboration conventions, citation tracing, and permission boundaries make the whole process reliable and safe.

Tools will keep changing, but this approach — open storage, retrieval augmentation, and human-AI collaboration — holds up over the long term. If you use Baidu AI Cloud’s model services, or plan to extend the knowledge base to a team collaboration scenario, starting with the retrieval evaluation and permission tiering sections above will get the fundamentals right before you expand further.