How AI Coding Assistants Understand Code: From Keyword Retrieval to a Project Knowledge System

1. The Starting Point: How Does It “Understand” Tens of Thousands of Lines of Code?

You type a requirement into an AI coding assistant, and it locates the relevant files, explains how the modules call each other, and proposes changes that match the project’s existing style. That looks like understanding, but in engineering terms it is an information retrieval problem: finding the few hundred most relevant lines of code inside a limited context window.

The intuitive answer is usually the classic retrieval-augmented generation (RAG) route: chunk code offline → compute embedding vectors → write them into a vector database → retrieve by similarity when the user asks → stitch the matching snippets into the prompt. This route works well for document Q&A and support knowledge bases, but many teams hit a gap when they transplant it to code repositories.

One observation is worth noting: both publicly shared technical talks and observable product behavior point in the same direction — the core retrieval capability of mainstream AI coding tools relies more on model-driven, multi-round keyword search than on prebuilt vector indexes. When two competing implementations make the same choice, that usually signals not a coincidental preference but a rational outcome under the constraints of the scenario.

Two questions therefore deserve serious discussion:

1. Why is keyword retrieval good enough for code search — and sometimes better than vector retrieval on key metrics?
2. If tools understand a project mainly by searching it, what should our repository prepare in advance so they search more accurately?

2. Breaking Down the Retrieval Mechanism: A Model-Driven Search Loop

The core of these tools is not “one retrieval” but an agent loop in which the model makes the decisions:

analyze the problem → choose a retrieval tool and invoke it → observe results → judge if information is sufficient
        ↑                                                                      │
        └────────────────── if not sufficient, keep searching ←──────────────────┘

The basic tools available are usually only a few categories, but they combine powerfully:

Capability	Typical role	Suitable scenarios
Regex content search	Match identifiers, strings, and comments across all text files	Finding symbol definitions, error codes, log keys, configuration items
Filename pattern matching	Locate files by path, extension, or naming convention	Finding tests, migration scripts, sibling modules
Exact reads	Read a specific line range of a specific file	Getting full context instead of isolated fragments
Sub-agent exploration	Spawn sub-agents to run independent multi-step investigations in parallel	Large-scope research, checking multiple candidate paths
Language service capabilities	Go to definition, find references, type information	Resolving naming ambiguity in dynamically typed languages

A real retrieval is often four to six rounds: the first casts a wide net to confirm candidate files; the second reads the matching snippets with context; the third reads the full file to confirm the call chain; the fourth follows references to fill in upstream and downstream pieces; the fifth reads tests and configuration to validate behavioral assumptions.

The fundamental difference from vector retrieval is that the retrieval strategy is decided at runtime rather than fixed offline. After seeing the first round of results, the model revises its assumptions — for example, discovering that the keyword was actually a legacy name and the real entry point lives in another directory — then searches again with a different set of terms. Vector retrieval gets only one shot: both query and documents are compressed into a fixed-dimension dense representation, and once offline chunking splits a function from its callers into different chunks, that cross-file causal relationship is lost permanently.

3. Why Exact Matching Fits the Code Scenario Better

3.1 Code Is Highly Identifiable Text

In natural language, “cancel order” and “rescind order” mean the same thing, but in code they most likely map to two different symbol names. The semantics of code are almost entirely anchored to precise tokens: identifiers, string constants, configuration keys, and error codes. The value of embedding models lies in handling “different phrasing, same meaning,” while code needs the opposite: precisely locating that one unique name. Using a fuzzy tool for a precise problem only introduces noise.

3.2 Chunking Semantics Conflicts with Code Structure

The full meaning of a function depends on its callers, its callees, type definitions, constants in the same file, and adjacent handling branches. Chunking by fixed token length cuts that context apart; chunking by syntax tree makes granularity hard to decide — too fine loses causality, too coarse raises noise. An operation like “read a specified line range” lets the model pull context on demand, so granularity stays dynamic.

3.3 Index Freshness and Maintenance Cost

A code repository is a high-churn object: a branch switch, a bulk rename, or a dependency upgrade can all invalidate an index. Vector indexes need full or incremental rebuilds, and retrieval quality is uncontrollable during a rebuild; with multiple branches and workspaces coexisting, you also have to maintain multiple index sets. Keyword retrieval requires zero preprocessing and takes effect with zero delay — as long as the file is on disk, the results are current. For development workflows where people switch branches several times a day, this difference is decisive.

3.4 Why Retrieval Performance Is Good Enough

A common objection is “how could a full-repository scan possibly be fast.” Several facts support it in practice:

• Source text in medium and large projects typically runs from tens to hundreds of MB, and modern SSD sequential read bandwidth is enough to sweep it in a few hundred milliseconds;
• Parallel directory traversal + multithreaded matching + memory-mapped files can push latency into an interactively acceptable range;
• Explicit ignore rules (build artifacts, dependency directories, binary assets) cut the actual scanned volume by another order of magnitude;
• Semantic work is delegated to language services, trading an index for precise reference lookup — and that index is usually maintained incrementally at controllable cost.

In other words, the “brute force” here is not blind full scanning but “cheap preprocessing + high parallelism + smart pruning.”

3.5 Boundaries You Must Acknowledge

Frankly, keyword retrieval is not superior across the board. It clearly struggles with:

• Poor naming and no shared token: functionality scattered across the codebase with unrelated names, impossible to link by literal text;
• A long distance between requirement description and implementation: no bridge between product language and code vocabulary;
• Implicit conventions and metaprogramming: reflective calls, dynamic registration, annotation-driven behavior, configuration-driven routing — static literal search cannot see the connections;
• Cross-repository and dependency internals: third-party dependency source code is outside the retrieval scope;
• Non-code information such as documentation, comments, and change history: requirement background and design decisions do not live in the code.

These are exactly the parts a “project knowledge system” should fill in. The capability boundary of the retrieval tool is a given, but the readability and searchability of the project are variables we can change.

4. Building a Project Knowledge System: Ten Practices That Make Tools Search Better

4.1 Provide an “Entry Map”

Place a file in the repository root meant to be read by both people and tools, focused on four things: what the system does, which top-level directories hold the code, one sentence on each directory’s responsibility, and how to build and test. It is not API documentation but a navigation layer. Example skeleton:

# Project Overview
## What This Is
Order fulfillment service: consumes order events and handles inventory reservation, payment verification, and shipment scheduling.
## Code Map
- cmd/        process entry points and startup wiring
- internal/order      order aggregate and state machine
- internal/inventory  inventory domain, including external channel adapters
- internal/gateway    outward-facing HTTP/gRPC interface layer
- pkg/events          domain event definitions
- migrations/         database changes, named by timestamp
## Build and Verify
- Build: make build
- Unit tests: make test
- Integration tests: make test-integration (requires local dependency containers)
## Key Conventions
- Domain events are always named `xxx.happened` / `xxx.requested`
- Cross-domain calls must go through the gateway layer; direct access to another domain's repository is forbidden

4.2 A Unified Glossary: One Concept, One Word

When the same business concept shows up as “user / customer / member,” retrieval will inevitably miss it. Maintaining a term mapping table that aligns business language with code symbols is the lowest-effort, highest-return item:

# docs/glossary.yaml
terms:
  - business: cancel order
    code: order.Cancel
    aliases: [rescind order, void order]
    files: [internal/order/cancel.go]
  - business: reserve inventory
    code: inventory.Reserve
    files: [internal/inventory/reserve.go]

This table works as an onboarding handbook for people and as a retrieval starting point for tools, quickly anchoring a vague requirement to definite symbols.

4.3 Make Implicit Conventions Explicit

Framework magic (auto-scanning, convention over configuration) is an efficiency gain for humans and a blind spot for retrieval. Leave an explicit anchor at every magic entry point: a centralized registry, a configuration sample listing all keys, or a comment explaining how routes map from directories.

// The single registration entry point for all scheduled jobs.
// New jobs must be registered here or the scheduler will not load them (there is no directory auto-scan).
var scheduledJobs = []Job{
    {Name: "settle_daily", Cron: "0 2 * * *", Handler: settle.Run},
    {Name: "expire_reservation", Cron: "*/10 * * * *", Handler: reserve.Expire},
}

The extra benefit of centralized registration: one retrieval yields the complete set, instead of gambling directory by directory.

4.4 Build a Searchable “Metadata Registry”

Requirement triage often starts from a log line, an error code, or a feature flag. Registering these scattered strings in one place significantly raises the retrieval hit rate:

Metadata type	Recommended practice
Error codes	Define centrally with a uniform prefix; ban scattered literals
Log keys / event names	Manage as central constants with an explanation of each one
Feature flags	One configuration center, plus a code location and owner noted for each flag
External interfaces	Keep contract files next to the implementation in the same directory tree

4.5 Align Directory Structure and Naming with the Domain Model

The first round of retrieval is often filename pattern matching. Directory names are the cheapest keywords. Organizing code by domain rather than by technical layer (instead of dumping every handler in one place) makes “which files does this requirement touch” almost visual. For naming, avoid pinyin abbreviations, single letters, and meaningless numeric suffixes — names that cannot be searched.

4.6 Provide Executable Specs: Tests as Documentation

Tests are documentation that never goes stale, and they are naturally named after functionality. Writing key business rules as table-driven tests whose names directly describe business behavior gives the tool a reliable path from requirement to code:

func TestCancelOrder(t *testing.T) {
    cases := []struct{
        name string   // "paid but not shipped: cancelable, triggers refund"
        ...
    }{...}
}

4.7 Maintain a Retrieval Ignore List

The more irrelevant files, the noisier retrieval gets. Explicitly exclude: build artifacts, dependency directories, lock files, compressed and binary assets, generated code, large snapshots and test fixtures, logs and local caches. Also, for generated code, put it in a separate directory with a prominent header comment so tools do not treat generated output as hand-maintained implementation and start editing it.

# Shared exclusions for retrieval and version control
vendor/
node_modules/
dist/
bin/
**/*_generated.go
**/testdata/large/
*.pb.go
coverage/

4.8 Layer Local Conventions

Keep global conventions in the root directory and put subsystem-specific notes in each subsystem directory (module boundaries, dependency direction, prohibitions). Once retrieval lands in a directory, it picks up the local rules on the spot, avoiding the “global document is comprehensive but the actual details can’t be found” problem.

4.9 Use Automation to Keep Structure from Decaying

Documentation drifting away from code is an inevitable trend, and self-discipline cannot sustain it. Put the checkable parts into the pipeline:

• Dependency direction checks: architecture constraint tests that forbid cross-layer direct access;
• Terminology consistency checks: warn when one concept appears with multiple spellings;
• Entry map validation: every directory listed in the overview file must actually exist, and vice versa;
• Dead code and orphan module scans, cleaned up on a schedule.

On Baidu AI Cloud, when building an R&D toolchain, you can combine general-purpose capabilities such as code hosting, continuous integration pipelines, log services, and monitoring and alerting: structure checks run in the pipeline, observability data about retrieval behavior lands in the log service, and quality trends are surfaced through monitoring alerts — closing the loop.

4.10 Measure Retrieval Effectiveness Instead of Going by Gut Feeling

Add observability to internal tools: average number of retrieval rounds, rounds needed to hit, first-round hit rate, and task abandonment rate. When a module’s average rounds sit significantly above baseline, the problem is usually not the tool but the naming or structure of that module needing treatment. Treating retrieval cost as a code quality metric is the part of this system that pays off over the long term.

5. Hybrid Strategy: When to Add Semantic and Structured Capabilities

Exact retrieval is the workhorse, but there is no need to reject other means — the key is filling gaps where the workhorse is weak:

• Symbol index and call graph: cover the static blind spots of reflection and dynamic registration with definite reference relationships;
• Language service capabilities: resolve naming ambiguity and type inference in dynamically typed languages;
• Commit history and change records: answer “why was it changed this way,” information the code itself does not contain;
• A small amount of semantic retrieval: only for coarse filtering from natural-language requirements to code, or for recall over non-code corpora such as docs, comments, and tickets, as a supplement that narrows scope before exact retrieval;
• Code knowledge graph: in very large multi-repository organizations, for cross-service dependency and impact analysis.

The principle: use deterministic means wherever they can solve the problem; when you introduce probabilistic means, they must be explainable and revertible.

6. Rollout Path and Summary

Three steps, each delivering value on its own:

1. Within two weeks: write the root entry map, add the ignore list, and clean up unsearchable naming (prioritizing high-traffic modules).
2. Within a month: build the glossary and metadata registry, centralize and make implicit conventions explicit, and add table-driven tests on critical paths.
3. Ongoing: wire in structure checks and retrieval effectiveness metrics, and drive governance with the data.

Back to the original question: AI coding assistants can “understand” tens of thousands of lines of code not because of a prebuilt vector store, but because of a model-driven retrieval loop that converges step by step. That mechanism hands a lot of control back to the model, and it also hands part of the responsibility to the project itself — the more your repository looks like a precisely searchable knowledge system, the closer the tool’s behavior comes to “it really understands this project.”

And this investment does not only serve AI. Unified terminology, clear directories, explicit conventions, executable specs, and automated structural guardrails are the baseline any engineering organization that wants to scale, hand over work easily, and keep communication costs low should already have.