Building and Using a Personal Knowledge Base with Knowledge Graphs: A Complete Guide

    A practical, end-to-end guide to building a personal knowledge base that is findable, connectable, and portable. It walks through a four-layer architecture (data, structure, efficiency, intelligence), Markdown-based file organization and naming conventions, YAML metadata, bidirectional links and tags, templates and search, Python scripts for health checks and bulk renaming, version control and multi-device sync, and an optional cloud knowledge graph plus RAG layer with AI-assisted curation. It includes concrete directory layouts, working code samples, prompt-design tips, risk controls, and a phased rollout roadmap.

    1. Why You Need Your Own Knowledge Base

    Information developers handle daily falls into three broad categories:

    • Short-term information: error messages, temporary code snippets—used once and discarded.
    • Project material: requirements documents, design drafts, API contracts—bound to a specific project.
    • Long-term knowledge: technical principles, architecture experience, troubleshooting approaches—reused across many projects.

    The first category is fine to search for on the fly. If the second and third are scattered across chat logs, browser bookmarks, and assorted documents, three problems tend to appear over time:

    1. Can’t find it: you remember “seeing this before” but can’t recall where it’s stored.
    2. Can’t connect it: the same topic is spread across multiple files, with no way to see how they relate.
    3. Can’t take it with you: data is locked inside a particular platform, in a closed format, with high migration costs.

    So the goal of a personal knowledge base comes down to three things: findable, connectable, portable.

    In recent years, bidirectional-link notes (backlink notes) have become a mainstream paradigm in knowledge management. They emphasize “atomic notes + bidirectional links”: each note covers a single concept, and notes form a network through links rather than nested folders. On top of this, scattered notes can be organized into a knowledge graph—notes as nodes, links as edges—making the structural relationships between concepts visible at a glance.

    On the technical side, a local-first architecture is recommended:

    • Data is stored as plain text in a local directory, with ownership fully in the user’s hands;
    • Core editing works without a network connection, so it remains fully usable offline;
    • Cloud capabilities are brought in only when needed—for collaboration, backup, or intelligent retrieval.

    As a developer, treating your personal knowledge base as a “small software system” to design and maintain pays off far more than simply hunting for a “good note-taking app.”

    2. Overall Architecture of a Knowledge Base

    A complete knowledge base can be divided into four layers:

    Layer Main responsibility Typical implementation
    Data layer Store note bodies and attachments Markdown files + an attachments folder for images, PDFs, etc.
    Structure layer Define organization and metadata conventions Directory layout, naming rules, YAML metadata, bidirectional links, tags
    Efficiency layer Lower the cost of recording and maintenance Templates, daily notes, full-text search, batch scripts
    Intelligence layer Enhanced retrieval, association discovery, and Q&A Version control, cloud indexing, vector search, AI-assisted processing

    The data layer uses the industry-standard Markdown format. There are two main reasons:

    • It’s plain text—compact in size, with clear version diffs, making it easy to compare and version-control;
    • It’s supported by many cross-platform editors and parsing libraries, so it won’t lose usability just because one piece of software stops being maintained.

    The structure layer determines the long-term usability of the knowledge base. A common mistake beginners make is “pick a tool first, then think about structure.” The result: after switching apps three times, the notes are still a mess. The correct order is the opposite: define the conventions first, then choose the tool.

    3. Data Layer: File Organization and Naming Conventions

    3.1 Comparing Three Organization Paradigms

    Approach Advantages Limitations Suitable scenarios
    Hierarchical folders Intuitive structure, easy to understand A note can belong to only one folder; cross-topic content is hard to file Project material, textbook-style notes organized by discipline
    Link-first Flexible, can express multiple relationships, new ideas can grow at any time Easily gets out of control without structure; needs some maintenance discipline Technical concepts, reading notes, accumulating inspiration
    Hybrid Folders handle “storage,” links handle “association” Requires agreeing in advance on how to divide the two Most personal knowledge bases (recommended)

    The hybrid approach is analogous to how a code repository is organized: folders act like module divisions, and links act like the dependencies between modules. Many-to-many relationships that a single folder can’t express are handled by links.

    1. my-kb/
    2. ├── 00-inbox/ # Inbox: unprocessed quick notes, clippings, and ideas
    3. ├── 10-daily/ # Daily notes and work logs
    4. ├── 20-notes/ # Long-term notes: atomic knowledge cards
    5. ├── 30-projects/ # Project material: requirements, designs, meeting notes
    6. ├── 40-assets/ # Attachments: images, PDFs, audio, video
    7. ├── 50-templates/ # Templates
    8. ├── 60-index/ # Index pages (MOC) and navigation
    9. └── 90-archive/ # Archive: finished or outdated content worth keeping

    Numeric prefixes between folders make them sort naturally in order of the workflow: information first enters the inbox, is processed into long-term notes or used directly in a project, and is archived after the project ends. This naming style is common in file-management methods based on a numbering system.

    Two points deserve emphasis:

    • The inbox and long-term notes must be physically separated. If quick notes are written directly into the long-term notes folder, the knowledge structure gets polluted by scattered content very quickly.
    • Keep folder depth to no more than three levels. Excessive nesting noticeably increases the cost of finding things; relationships should be expressed more through links.

    3.3 Managing Attachments

    Attachments tend to run into two kinds of problems:

    • Scattered everywhere, with duplicate filenames overwriting each other;
    • Notes using absolute paths, so if the whole library is moved, all images break.

    The recommended approach:

    • Store all attachments in 40-assets/, with subfolders by year and month;
    • Name files with a uniform “date + short description” format, such as 20250312-call-stack.png;
    • In the note editor, set the default attachment save location to this folder and reference files using relative paths;
    • Periodically run a script to scan for attachments “not referenced by any note” and clean them up.

    3.4 Naming Conventions

    Note titles should meet four requirements:

    • Readable: the topic is clear at a glance—avoid names like “Untitled” or “New Note”;
    • Unique: avoid using dates as titles as much as possible. Daily notes are the exception and can be uniformly named YYYY-MM-DD;
    • Linkable: no OS-reserved characters such as /\:*?"<>|;
    • Stable: once another note links to it, renaming requires updating all references, so weigh the cost against the benefit.

    Filenames may be in Chinese. As long as the system consistently uses UTF-8 encoding and you watch for case-sensitivity differences across devices during multi-device sync, Chinese filenames themselves are not a problem.

    4.1 Describing Note Attributes with YAML Metadata

    YAML metadata (front matter) is a set of key-value pairs written at the top of a Markdown file, wrapped between two --- lines. It’s like attaching a “structured attribute table” to each note:

    1. ---
    2. title: Sharding Strategy for the Index Service
    3. type: note
    4. status: growing
    5. tags:
    6. - Knowledge Management/Retrieval
    7. - Distributed Systems/Data Sharding
    8. created: 2025-03-12
    9. updated: 2025-06-01
    10. aliases:
    11. - Sharding strategy
    12. ---

    When designing metadata fields, you can refer to the “entity–attribute–value” model. Each field is essentially a triple describing the note:

    • Entity: the current note;
    • Attribute: the field name, such as status or created;
    • Value: the field’s value, such as growing or 2025-03-12.

    This is exactly the basic building block of a knowledge graph, and the later graph-building and structured queries all rest on it.

    Field design follows three principles:

    1. Few and stable: keep only five fields at first—title, type, status, tags, created—and expand gradually during use;
    2. Controlled values: fields like status and type should use only a small, pre-agreed set of values, avoiding a mix of “done,” “completed,” and “closed,” which makes later statistics difficult;
    3. Automated dates: let templates or scripts fill in created and updated timestamps automatically, rather than relying on manual upkeep.

    Most Markdown editors that support bidirectional links use the [[target note]] syntax. The main usages:

    1. # Sharding Strategy for the Index Service
    2. The core goal of sharding is to distribute write load evenly across multiple storage nodes.
    3. Related notes:
    4. - Background: [[Read-Write Hotspots and Cold-Data Tiering]]
    5. - Core concept: [[Consistent Hashing]]
    6. - Comparison: [[Range Sharding vs. Hash Sharding]]
    7. - Practice: the "Migration Throttling" section of [[Data Migration for Shard Resharding]]
    Syntax Purpose Usage tip
    [[Note Name]] Link to an entire note Most common; keep titles concise so they’re easy to type
    `[[Note Name\ Display Text]]` Customize the display text Good for embedding in a sentence to keep prose natural
    [[Note Name#Heading]] Link to a specific section of a note Point precisely to the paragraph holding the conclusion
    [[#Heading]] Link to a section within the current note Build an internal table of contents for a long note

    The key to bidirectional links is the “backlink”:

    • When A links to B, opening B shows “which notes reference it”;
    • This relationship is maintained automatically by the editor based on file contents—no manual record-keeping needed.

    If you view the whole library as a directed graph, each note is a node and each link is an edge running from the “referencer” to the “referenced.”

    • Outgoing links are “who I referenced”;
    • Backlinks are “who referenced me.”

    As nodes and edges accumulate, the knowledge base evolves from “files in a folder” into a “network of concept relationships.”

    A few tips worth keeping in mind when using bidirectional links:

    1. Atomicity: a note should carry a single standalone idea, making it reusable in multiple places.
    2. Link with context: write a sentence before or after the link explaining “why they relate,” rather than listing a string of bare links. For example: “Range sharding is friendly to range queries but prone to data skew; see [[Range Sharding vs. Hash Sharding]] for a comparison of the two.” Only edges that explain the reason for the association carry real value in the graph.
    3. Link to not-yet-created notes: bidirectional links can point to notes that don’t exist yet. These “to-be-created notes” can be seen as a topic list generated by the knowledge base—the more a to-be-created note is referenced, the more worthwhile it is to fill in first.
    4. Index pages (MOC): create an index navigation note for each topic area, gathering that domain’s core notes and reading order—essentially adding a few “hub nodes” to the knowledge graph.

    4.3 Tags: Fast Filtering Across Dimensions

    Links express “the specific relationship between notes,” while tags express “an attribute shared by notes.” For example, all notes related to retrieval can be tagged #KnowledgeManagement/Retrieval, regardless of which folder they’re stored in.

    A hierarchical “category/subcategory” style is recommended. The hierarchical prefix keeps namespaces from colliding—for instance, #KnowledgeManagement/Retrieval and #DistributedSystems/Retrieval—and lets you progressively narrow the scope when filtering.

    Tag management should stay “few and refined”:

    • Before introducing a new tag, search existing tags and reuse them where possible;
    • Use a script to count tag frequency, and merge or clean up once a month.

    The division of labor among tags, folders, and links can be understood like this:

    • Folders answer “where it’s stored”;
    • Tags answer “which category it belongs to”;
    • Links answer “what it’s related to.”

    Each does its own job without replacing the others—only then can the knowledge base’s structure stay stable over the long term.

    5. Efficiency Layer: Templates, Daily Notes, and Retrieval

    5.1 Templates: Standardizing the Recording Process

    The role of a template is to lock down repetitive structure. Below is a note template suited to technical learning:

    1. ---
    2. title: {{title}}
    3. type: note
    4. status: growing
    5. tags: []
    6. created: {{date}}
    7. ---
    8. ## One-sentence summary
    9. ## Background and problem
    10. ## Core content
    11. ## My understanding and practice
    12. ## References
    13. -
    14. ## Related notes
    15. -

    Here {{title}} and {{date}} are placeholders. Most template extensions support this style, or offer equivalent date/time variables; check your tool’s documentation for the exact syntax.

    Combined with a daily note (e.g., 10-daily/YYYY-MM-DD.md), this forms a “capture → process → distill” workflow:

    • During the day, quickly record problems, commands, and ideas in the daily note;
    • At a fixed time each week, break valuable content into atomic notes and add links and tags;
    • The daily note itself serves only as a timeline; long-term knowledge is consistently distilled into 20-notes/.

    5.2 Retrieval: The First Step to Getting Knowledge Back

    Retrieval capability directly determines how well the knowledge base lives up to “findable.” Mature note editors usually support search syntax with qualifiers. Common ones:

    Qualifier Meaning Example
    path: Limit to a folder path:20-notes/ sharding
    file: Limit to a filename file:sharding
    tag: Limit to a tag tag:#KnowledgeManagement/Retrieval
    content: Limit to body content content:"data migration"
    line: / block: Limit to a line or block line:(migration throttling)

    In addition, - means exclude and * matches any characters—for example, -tag:#status/archive excludes archived notes.

    It’s a good idea to save high-frequency search conditions as “saved searches,” effectively bookmarking common queries for the knowledge base.

    When the library reaches several thousand notes, you can go further with an extension plugin that supports structured queries—similar to a database’s SELECT statement—to filter and summarize notes from YAML metadata. The following pseudo-code uses a SQL-like syntax:

    1. TABLE file.ctime AS "Created", status, tags
    2. FROM "20-notes"
    3. WHERE status = "growing"
    4. AND contains(tags, "#KnowledgeManagement/Retrieval")
    5. SORT file.ctime DESC

    It’s equivalent to running: SELECT Created, status, tags FROM 20-notes WHERE status = 'growing' AND tags contains '#KnowledgeManagement/Retrieval' ORDER BY Created DESC.

    Results of such queries can be embedded into index pages and automatically rendered as a “notes to be organized” list—no manual upkeep required.

    6. Automation: Maintaining the Knowledge Base with Scripts

    One of the biggest advantages of plain text is that the knowledge base can be operated on directly with scripts. All maintenance tasks can be handled with a general-purpose programming language, without depending on any specific tool. Below, Python is used as an example (it runs with the standard library alone).

    As notes grow, manually checking “whether links are broken” and “which notes are never referenced” becomes increasingly costly. The following script does this check:

    1. import re
    2. from pathlib import Path
    3. VAULT = Path("./my-kb") # Knowledge base root
    4. EXCLUDE_DIRS = {"40-assets", "50-templates", "10-daily"}
    5. PATTERN = re.compile(r"\[\[([^\]|#]+)")
    6. notes = {
    7. p.stem: p
    8. for p in VAULT.rglob("*.md")
    9. if not EXCLUDE_DIRS.intersection(p.relative_to(VAULT).parts)
    10. }
    11. incoming = {name: set() for name in notes}
    12. broken_links = []
    13. for path in notes.values():
    14. for raw in PATTERN.findall(path.read_text(encoding="utf-8")):
    15. target = raw.strip()
    16. if target in notes:
    17. if target != path.stem: # Skip self-references
    18. incoming[target].add(path.stem)
    19. else:
    20. broken_links.append((str(path), target))
    21. for src, target in broken_links:
    22. print(f"[Broken] {src} -> [[{target}]]")
    23. for name, sources in sorted(incoming.items()):
    24. if not sources:
    25. print(f"[Orphan] {name}")
    26. print(f"Notes: {len(notes)}, broken links: {len(broken_links)}, "
    27. f"orphans: {sum(not v for v in incoming.values())}")

    The script’s logic runs in three steps:

    1. Recursively scan all Markdown files, excluding the attachments, templates, and daily-note folders;
    2. Use a regular expression to extract target note names from [[...]] and build a reverse index of “who referenced whom”;
    3. Output lists of broken links and orphaned notes.

    These results are themselves important maintenance signals:

    • Broken links: create the missing notes or fix the links;
    • Orphaned notes: either add associations, or archive or delete them.

    When a batch of note titles needs to be standardized (for example, uniformly adding a date prefix), changing filenames directly breaks all references. The correct order is “update links first, rename files after,” following a two-step strategy: do a dry run first, confirm it’s correct, then execute for real.

    1. import re
    2. from pathlib import Path
    3. VAULT = Path("./my-kb")
    4. PATTERN = re.compile(r"(\[\[)([^\[\]|#]+)([\]\[\]|#][^\]]*\](?!\]))?")
    5. renames = {
    6. "Consistent Hashing": "20250312-Consistent Hashing",
    7. "Range Sharding vs. Hash Sharding": "20250315-Range Sharding vs. Hash Sharding",
    8. }
    9. dry_run = True # Set to False after confirming everything is correct
    10. if not dry_run:
    11. for path in VAULT.rglob("*.md"):
    12. text = path.read_text(encoding="utf-8")
    13. new_text = PATTERN.sub(
    14. lambda m: m.group(1)
    15. + renames.get(m.group(2), m.group(2))
    16. + (m.group(3) or ""),
    17. text,
    18. )
    19. if new_text != text:
    20. path.write_text(new_text, encoding="utf-8")
    21. print(f"[Update link] {path}")
    22. for old, new in renames.items():
    23. for path in VAULT.rglob(f"{old}.md"):
    24. dst = path.with_name(f"{new}.md")
    25. print(f"[{'DRY RUN' if dry_run else 'EXECUTE'}] {path.name} -> {dst.name}")
    26. if not dry_run and not dst.exists():
    27. path.rename(dst)

    This script’s regex has three capture groups, corresponding to:

    • The left delimiter [[;
    • The target note name;
    • The optional display text or section anchor (e.g., |alias, #heading).

    When replacing, only the target note name changes—the display text and anchor are preserved as-is. For a naming system that uses dates as note names, this script can also write a created field into the file contents and update it automatically.

    6.3 Three Rules for Using Scripts

    • Back up before executing: combined with version control, any batch operation can be rolled back.
    • Dry run before executing: default to dry-run, and write for real only after confirming the output matches expectations.
    • Incremental, not wholesale: run on a regular schedule to catch and address problems early.

    7. Persistence: Version Control and Multi-Device Sync

    7.1 Version Control

    Plain text is inherently suited to version control. Using a distributed version control system to manage the note library brings these benefits:

    • Traceability: every change to every note is recorded and can be compared or rolled back at any time;
    • Multi-device collaboration: multiple devices exchange changes through a remote repository;
    • Scheduled snapshots: scheduled tasks can commit automatically, forming an archive of daily or hourly history.

    Given the nature of a personal knowledge base, a “trunk-based development + scheduled snapshots” strategy is recommended:

    • Keep daily changes on the trunk branch to preserve continuity of your main line;
    • For higher-risk operations like bulk refactoring and structural changes, create a separate branch first, and merge back to the trunk only after verifying it’s correct;
    • Large attachments generally don’t belong in a version repository; handle them separately with a sync tool, or ignore them in the repository configuration.

    7.2 Comparing Multi-Device Sync Options

    Option Principle Advantages Notes
    Sync-drive folder Automatically syncs an entire folder between the cloud and your computer Simple to configure, good mobile experience Slower with many attachments; conflicts may produce duplicate files
    Version repository Distributes changes through a remote repository Complete history, explicitly controllable conflicts Requires manual or scheduled pull/push; attachments handled separately
    Self-hosted sync service Deploy your own sync server for device-to-device sync Fully autonomous data, fast on a LAN Requires deploying and maintaining the service yourself

    Whichever option you choose, follow these principles:

    1. Separate attachments from notes: notes go through the version repository, attachments through the sync drive.
    2. Make conflicts explicit: avoid editing the same note on two devices at once; when a conflict file appears, merge it manually and promptly—don’t just delete it.
    3. Follow the 3-2-1 backup rule: keep at least 3 copies of your data, on 2 different media types, with 1 stored off-site.

    8. Intelligence Layer (Part 1): From a Note Library to a Cloud Knowledge Graph

    As the library grows to thousands of notes and multiple people need to share it, purely local full-text search gradually becomes inadequate:

    • It only matches keywords, struggling with content that is “semantically similar but worded differently”;
    • It can’t perform unified association and reasoning across documents;
    • It can’t provide a unified retrieval entry point for collaboration.

    At this point, while preserving local data sovereignty, you can upgrade the knowledge base to a cloud knowledge graph + retrieval-augmented generation (RAG) architecture.

    8.1 The Data Pipeline

    1. Local Markdown note library
    2. │ Change detection (scheduled scan / event trigger)
    3. ▼
    4. Document parsing: body extraction, metadata parsing, on-demand OCR
    5. │
    6. ├──────────────► Full-text inverted index (keyword retrieval)
    7. │
    8. └──► Embedding ──► Vector index (semantic retrieval)
    9. │
    10. ▼
    11. Knowledge graph: entity extraction, relation extraction, entity disambiguation
    12. │
    13. ▼
    14. Retrieval fusion → LLM generation → answers with cited sources

    Each stage’s responsibilities:

    1. Change capture: periodically scan the library, or listen for file-change events, computing a content digest of each file to identify additions and modifications.
    2. Document parsing: extract the body, read the YAML metadata, and perform OCR on scanned PDFs or image attachments as needed.
    3. Index building:
      • The full-text index handles exact matching of keywords, terms, and code identifiers;
      • The vector index maps text to high-dimensional vectors, using similarity search to recall semantically similar content.
    4. Graph building: extract entities and relations from notes, links, tags, and metadata; disambiguate identically named concepts; form a queryable knowledge network.
    5. Retrieval and generation: recall candidates by combining keyword and vector retrieval, hand them to an LLM to generate an answer, and attach links to the source notes.

    8.2 Common Cloud Capabilities

    Most of the above stages can be handled by general-purpose products from cloud providers. In Baidu AI Cloud, for example, you can typically combine:

    • Object storage: centrally store raw notes, attachments, and parsing results;
    • Document parsing and OCR: handle unstructured content like PDFs, images, and scans;
    • Vector retrieval / vector databases: host embeddings and similarity search;
    • Knowledge base products: package the flow of document ingestion, chunking, retrieval, and Q&A orchestration;
    • Content moderation: run compliance checks before content is shared externally.

    It’s worth emphasizing that the goal of this architecture is to augment the local knowledge base, not replace it: the local files remain the single source of truth, and cloud data is just a derived index that can be rebuilt at any time.

    8.3 Engineering Essentials for Incremental Indexing

    • Idempotency: processing the same content repeatedly yields a consistent result, avoiding duplicate writes;
    • Traceability: each index record is associated with the source file’s path and content digest, making it easy to locate stale indexes;
    • Retry on failure: when parsing or embedding fails, the item enters a retry queue so a single failure doesn’t block the whole pipeline;
    • Permission alignment: the access permissions of the cloud index should match the sensitivity level of the source files; confidential content should not go to the cloud.

    9. Intelligence Layer (Part 2): Using AI to Organize Materials Efficiently

    9.1 Treating AI as a “Knowledge-Processing Pipeline”

    Knowledge processing breaks into roughly four stages, and AI can take on much of the repetitive work in each:

    Stage Human responsibility Work AI can take on
    Capture Judge whether information is valuable Summarize web pages and PDFs, and extract key points
    Organize Define the taxonomy and tagging conventions Recommend tags, create bidirectional links, file scattered quick notes
    Distill Form your own understanding and judgment Compare multiple notes, generate topic overviews and indexes
    Output Verify conclusions, add practical details Draft initial content, format it into structured documents

    The core principle: AI handles the “processing”; humans handle the “judgment.” Classification standards, what knowledge to keep, and final conclusions always remain in human hands.

    9.2 Prompt Design: Role, Task, Format, Constraints

    A practical prompt usually has four parts:

    1. Role: tell the model what identity to think from;
    2. Task: clearly state the specific work to be done;
    3. Format: specify the structure of the output;
    4. Constraints: bound the scope and require honest labeling of uncertain information.

    For example, when organizing a scattered technical quick note, you can use:

    1. You are a knowledge management assistant. Read the note I provide and do the following:
    2. 1. Summarize the core point in one sentence;
    3. 2. Extract 3–5 key concepts, each with a brief explanation;
    4. 3. Recommend 2–3 hierarchical tags in the format #category/subcategory;
    5. 4. List the technical terms mentioned in the text;
    6. 5. Output 3 follow-up questions for further thought.
    7. Constraints: answer only based on the original text; for information not mentioned in the original, explicitly mark it "not mentioned in the source."

    9.3 Batch-Generating Summaries and Tags with a Script

    When the inbox has piled up a lot of text to process, you can batch-process it through a model service’s compatible API. The example below assumes the service is compatible with a common chat-completion interface format; MODEL_API_BASE and MODEL_API_KEY must be replaced with the actual model service address and key:

    1. import json
    2. import urllib.request
    3. from pathlib import Path
    4. MODEL_API_BASE = "https://<model-service-endpoint>/v1/chat/completions"
    5. MODEL_API_KEY = "<api-key>"
    6. MODEL_NAME = "<model-name>"
    7. INBOX = Path("./my-kb/00-inbox")
    8. PROMPT = (
    9. "You are a knowledge management assistant. Read the note below, first output a "
    10. "summary of no more than 200 characters, then give 3–5 hierarchical tags "
    11. "(format: #category/subcategory). "
    12. "Keep the order fixed as 'Summary:' and 'Tags:'. "
    13. "Answer only based on the original text; do not add information not in the original."
    14. )
    15. def ask_model(text: str) -> str:
    16. payload = {
    17. "model": MODEL_NAME,
    18. "messages": [
    19. {"role": "system", "content": PROMPT},
    20. {"role": "user", "content": text},
    21. ],
    22. }
    23. req = urllib.request.Request(
    24. MODEL_API_BASE,
    25. data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
    26. headers={
    27. "Content-Type": "application/json",
    28. "Authorization": f"Bearer {MODEL_API_KEY}",
    29. },
    30. )
    31. with urllib.request.urlopen(req, timeout=60) as resp:
    32. data = json.loads(resp.read().decode("utf-8"))
    33. return data["choices"][0]["message"]["content"]
    34. for md in sorted(INBOX.glob("*.txt")):
    35. content = md.read_text(encoding="utf-8").strip()
    36. if not content:
    37. continue
    38. try:
    39. result = ask_model(content)
    40. except Exception as exc:
    41. print(f"[Failed] {md.name}: {exc}")
    42. continue
    43. print(f"===== {md.name} =====\n{result}\n")

    In practice, you also need to strengthen a few things:

    • Add rate limiting and timeout retries to avoid requests that are too dense;
    • Write model output to temporary files for manual review, and only write to the library after confirmation;
    • Ensure the API endpoint and key are managed through environment variables or a config file—don’t hardcode them in the script.

    9.4 Writing Results Back and Risk Controls

    AI output should be reviewed by a human before entering the knowledge base. A three-stage “draft area → review → formal area” flow is recommended:

    1. Draft area: model output is first written to a pending file in 00-inbox;
    2. Review: manually fix errors, add personal understanding, and complete bidirectional links;
    3. Formal area: after confirming it’s correct, move it to 20-notes and standardize the metadata.

    The main risks to guard against:

    • Hallucination: the model may generate content not present in the original. Technical details, data, and citations must be checked back against the source.
    • Privacy and compliance: sensitive information should be de-identified before being submitted to an external model service, or you can use a private deployment where data stays within the local environment.
    • Copyright: copyrighted content should be used only for personal study and research, not publicly distributed.
    • Service volatility: changes to free quotas, rate limiting, model version updates, or a service being retired can all break an existing processing flow.
    • Architectural redundancy: if full-text search already meets your needs, there’s no need to bring in a vector database and model services just to “use AI,” which only adds maintenance cost.

    In the long run, relying on free quotas isn’t dependable. A more robust approach:

    • Position AI as an “accelerator” rather than “infrastructure”;
    • Make model services switchable through configuration in scripts, keeping them replaceable;
    • Periodically evaluate the total cost of different paths, such as self-hosted models versus cloud services.

    10. Implementation Roadmap

    Knowledge base building can be divided into four phases:

    Phase Duration Core tasks Completion marker
    Foundation setup Weeks 1–2 Install a local Markdown note tool and common extensions, set up the directory structure and templates, configure the attachments folder Daily recording and retrieval work smoothly
    Habit formation Weeks 3–6 Stick to the “capture–process–distill” loop, organize the inbox weekly, actively create bidirectional links Bidirectional links and tags become a recording habit
    Automation boost Weeks 7–10 Introduce health-check and bulk-rename scripts, configure version control and automatic backups Library structure stays healthy and changes are traceable
    Intelligence upgrade Week 11 onward As needed, connect cloud indexing, vector retrieval, and AI-assisted processing You can ask questions in natural language and get answers with sources

    Before each phase ends, do a “knowledge base check-up”:

    • Is the inbox backlog more than a week old?
    • Are the numbers of broken links and orphaned notes steadily declining?
    • How many real problems did you actually solve with the knowledge base in the past month?

    If the answers aren’t satisfactory, return to the previous phase to adjust the workflow rather than rushing to bring in a new tool.

    Conclusion

    Building a knowledge base is essentially ongoing infrastructure development, not a one-time software selection:

    • The data layer determines who owns the data;
    • The structure layer determines whether knowledge can be reused;
    • The efficiency layer determines whether you can keep recording;
    • The intelligence layer determines how much value knowledge can create.

    Start with a clear directory structure and a YAML field, keep at it with atomic recording and bidirectional links, then supplement it with scripts and AI tools, and scattered information will gradually grow into a knowledge network that is searchable, connectable, and continuously growing. For developers, this process is itself a training in systems thinking: organize knowledge like code, and operate the note library like a system.