How AI Coding Assistants Understand Codebases: RAG, Agentic Search, and Structured Retrieval

1. Bottom line: not an either-or choice, but three coexisting paradigms

Give an AI coding assistant a natural-language request, and it can locate relevant code, map dependencies, and generate an implementation that matches the project’s style. But how does it find the right piece of the puzzle inside a codebase with tens of thousands or even hundreds of thousands of lines?

The conclusion first:

• Codebase retrieval is not a simple binary choice between vector RAG and agentic search. In engineering practice, the common pattern is multi-route retrieval plus on-demand reading: indexing is responsible for “quickly narrowing the search space,” while agentic search handles “precise drill-down and verification.”
• The biggest variable affecting final results is not the name of the algorithm, but context quality: whether names are semantically clear, whether boundaries are well defined, and whether there is a project guide written for AI assistants.
• There is no universal optimal solution. The right choice depends on repository size, whether there is a reliable build system, model reasoning capability, and latency/token budget.

Understanding this point helps us work backward and improve how code is organized, how comments are written, and how rule files are configured.

2. The intuitive answer: a vector-centered RAG pipeline

Most people’s first instinct is RAG, or retrieval-augmented generation. The classic pipeline has four steps:

1. Chunking: split the codebase into fragments by file, function, or fixed length.
2. Embedding: encode each fragment into a high-dimensional vector with an embedding model.
3. Indexing: write the vectors and metadata into a vector database.
4. Retrieval: encode the question into a vector, use approximate nearest neighbor search (ANN) to recall the most relevant fragments, and pass them to a large language model for answer generation.

This approach has been widely validated in document Q&A and knowledge-base retrieval. It is clear, engineering-mature, and appears to be a natural fit for codebases.

3. The surprising step: model-driven agentic search

Publicly reported claims include:

• In November 2024, a co-founder of Cursor, a mainstream AI coding assistant team, mentioned in a public podcast interview that the team had experimented with vector indexing for codebases early on, but later shifted toward an approach where the model autonomously invokes search tools.
• The same team also explained in public interviews and official documentation that, for large codebases, letting the model “search—read—search again” often works better than building a semantic index in advance.

It is important to be precise: this is one team’s experience in its own testing scenario. It depends on the team’s model reasoning capability and tool design, and it is not a universally reproduced conclusion. For contrast, another type of approach is Claude Code, a mainstream command-line coding assistant, which states in its official documentation that it does not build an index for the codebase by default and relies mainly on search and file-reading tools. Meanwhile, code search at code hosting platforms—such as Sourcegraph’s Zoekt trigram index—still follows the prebuilt-index route. In other words, both approaches coexist in real products, and the tradeoff point is the scenario, not superiority.

Agentic search works by letting the model explore iteratively, like an engineer:

• Use regex or keyword search, such as grep, to locate possible entry points.
• Use filename patterns, such as glob, to narrow the scope.
• Read the actual content of matched files and judge whether it fits expectations.
• Based on newly discovered symbols and import relationships, refine the next search terms until the target is locked down.

Search is no longer a “one-shot retrieval.” It becomes a multi-step exploration led by the model.

4. The overlooked middle layer: keyword search, hybrid retrieval, and structured indexes

Beyond vector RAG and agentic search, codebase retrieval also has two common paradigms, and they are the real shape of many tools.

Keyword and hybrid retrieval. Sparse retrieval methods such as BM25 are strong at exact symbol-name matching; trigram indexes, such as Zoekt, are suitable for large-scale code search. Mature solutions usually use sparse plus dense dual-route recall → rerank → hand results to the model, balancing exact symbol hits with semantic similarity.

Structured indexing. Indexes based on AST, LSP, ctags, tree-sitter, or compilation databases can provide relationship information such as “definition—reference—call chain,” which pure text search cannot obtain. Aider’s repo map, for example, uses tree-sitter to parse symbols and generate a repository overview. It is a typical “structured summary plus on-demand reading” combination.

Evolution timeline (with verifiable representative milestones):

• 2020–2022: dominated by keyword/regex search and trigram indexes; code search at code hosting platforms was a representative example.
• 2023: vector RAG became popular with the retrieval-augmented ecosystem and became the default imagination for “AI reading code.”
• 2024: hybrid retrieval plus reranking matured; agentic search entered mainstream coding assistants, with the November 2024 public interview being a landmark moment.
• 2025 onward: the combination of agentic search and structured indexing became the mainstream engineering form, and project rule files such as AGENTS.md formed a de facto convention.

This timeline explains the word “evolution” in the title: it is not replacement, but a move from single-route indexing to multi-route reuse and on-demand reading.

5. Side-by-side comparison: how to view four retrieval methods

Retrieval method	Advantages	Bottlenecks	Suitable scenarios
Keyword/BM25, trigram indexes	Exact hits for symbol names, types, and constants; simple to implement; low latency	Misses synonyms, abbreviations, or poorly named symbols; regex metacharacters and dynamic string concatenation can fail	Entry-point location in large single-repo codebases; fallback recall for other approaches
Vector RAG	Can handle paraphrased requests; friendly to natural-language requirements, including Chinese requirements	Chunking can break cross-file call chains; embedding models are insensitive to exact symbols; index needs continuous rebuilding	Ambiguous requirement descriptions, document-heavy semantics, cross-language search
Hybrid retrieval + rerank	Sparse and dense methods complement each other; first-round hit rate is significantly higher than single-route retrieval	Longer engineering pipeline; requires labeled data to tune weights; reranking adds latency	Production-grade code Q&A and enterprise knowledge bases
Agentic search	Reads complete, uncut source code; no index maintenance; code changes take effect immediately	Depends on model reasoning and context window; multi-round search consumes tokens and time; tool-permission risks	Frequently changing codebases; context reading and verification; cross-file refactoring and troubleshooting
AST/LSP/compilation database	Provides structural relationships such as definitions, references, and call graphs	Requires a runnable build environment; generation and maintenance have costs	Projects with build systems; cross-module impact analysis

Why the “dumb approach” is often better: bottlenecks of RAG in code scenarios

• Chunking breaks semantic completeness. The value of code often lies in cross-file call chains. Mechanically splitting by length can sever relationships, producing recall results that are “locally correct but globally disconnected.”
• Embeddings and code semantics are mismatched. Synonym abbreviations such as cnt/count, multiple similar implementations of the same function, and comments disconnected from implementation can all cause semantically similar but actually irrelevant fragments to crowd out recall slots. Meanwhile, retrieval truly needs precise symbol names, types, and scopes.
• Index maintenance costs. A codebase evolves every day. Any change requires incremental rebuilding; otherwise, results quickly become stale. Full indexing for large multi-repo projects can be costly in both time and storage.

Agentic search is not without costs

It depends heavily on model reasoning capability and the context window, and multi-round search consumes more tokens and time. Its failure scenarios are worth noting:

• grep may miss dynamically constructed calls such as obj[method]() or getattr.
• Unescaped regex metacharacters can cause false matches.
• Once the context window is exceeded, the model is forced to truncate.
• Overly broad tool permissions create a risk of unintended edits.

More precisely, agentic search means no index to maintain and no rebuild required for code changes to be discovered. It does not mean “zero cost.”

Should a codebase choose RAG or agentic search?

• Large repositories, latency-sensitive scenarios, and a reliable build system → prioritize structured indexing plus keyword recall.
• Clear naming conventions, frequent changes, and a need to read context for judgment → agentic search is more flexible.
• Many Chinese natural-language requirements, document-heavy semantics, and cross-language search → use hybrid retrieval as a fallback.
• Production deployment → most teams’ answer is a combination of “multi-route recall + rerank + agentic drill-down.”

Why is the hit rate still not high?

First, define the metric: first-round hit rate = number of questions whose target files are hit while reading no more than max_k files ÷ total number of questions (the target file is the set of files manually annotated as “necessary to answer the question”). Recommended accompanying metrics: average search rounds, average token consumption, and P95 latency.

Common problems, ordered by impact:

1. Poor naming semantics (utils2.js, helper.js)
2. No project-level documentation
3. Chunk size too large or too small
4. Lack of dependency graph and build artifacts
5. Only retrieval without a verification loop

6. Engineering implementation: turning principles into an executable checklist

1. Write a project rule file for AI. Place AGENTS.md in the repository root; equivalent conventions include .cursor/rules/, CLAUDE.md, and .github/copilot-instructions.md. Suggested sections:

• Tech stack and versions
• Directory map
• Build and test commands
• Key interfaces and data models
• Forbidden items, such as directories that must not be changed or patterns that must not be used
• Code style and commit conventions

Keep it readable within one screen, and link to separate documents for details.

2. Use naming and boundaries to reduce retrieval difficulty. Rename utils2.js to date_formatters.js. This is kind to both humans and AI. Clear file names, function names, and single responsibilities improve hit rates for both grep and vector retrieval.

3. Use existing artifacts to provide structural information; do not hand-write call graphs. Artifacts such as compile_commands.json for C/C++, go.mod for Go, package.json for front-end/Node projects, pom.xml for Java, and ctags or tree-sitter symbol indexes can cheaply add static relationships between modules for AI assistants.

4. Make retrieval and verification a closed loop. Encourage the assistant, after locating code, to continue reading context and run tests to verify assumptions, rather than stopping at “found it.”

5. Use layered retrieval for large repositories. A two-layer example:

# Layer 0: repository overview (title-level information only, within 200 lines)
- AGENTS.md: tech stack / build and test commands / forbidden items
- Top-level directory tree (depth=2) + first paragraph of each subdirectory README
- Build configuration: package.json scripts, Makefile targets
# Layer 1: module drill-down (on demand, expand one directory at a time)
src/billing/
  ├── __init__.py          # public exports
  ├── invoice.py           # Invoice generation and validation
  ├── tax_rules.py         # tax-rate parsing
  └── tests/test_invoice.py
# After a hit: read(invoice.py, around=matched_lines) → then grep definitions of called symbols

Pseudocode (illustrative, not runnable code):

while rounds < max_rounds and tokens < token_budget:   # stopping condition
    hits = grep(pattern) or glob(name_glob) or index_lookup(symbol)
    for f in select(hits, max_k):            # maximum files read per round; recommended 3–5
        ctx = read(f, around=matched_lines)
        if matches_intent(ctx):
            return ctx, rounds
    pattern = refine(pattern, based_on=what_we_learned)   # search terms evolve with understanding
return summarize_partial(visited)             # fallback: return visited clues instead of forcing an answer

Recommended settings:

• max_k: 3–5
• Maximum search rounds: 5–10
• Token budget per task: set at 20%–30% of the model’s context window; exceeding it should trigger fallback summarization.

7. Risks and boundaries

• Private source code being sent externally: before using third-party model services, confirm permission controls, log retention, and redaction strategies. Sensitive modules can consider local models or private deployment, and compliance requirements and data-residency clauses should be checked.
• Tool permissions and sandboxing: agentic mode allows the model to execute commands and modify files. Limit writable paths, disable dangerous commands, and run in a container or feature branch.
• Human review is authoritative: code located or generated by AI must be reviewed by humans, and test results should be the final criterion. “Looks correct” is not enough.

8. Next step

First, add a rule file (AGENTS.md) and a two-layer directory overview to the repository. Then select 10–20 real requirements for baseline testing: record first-round hit rate, average rounds, token consumption, and P95 latency. With that data, combine indexing and real-time search according to repository size and latency budget. Indexes are reusable, real-time search is more flexible, and combining them by repository size and latency budget is more effective than arguing about which paradigm is more advanced.