Unified Retrieval Architecture for Agents: Design and Practice
Agents often stall not because the model is too small, but because each action lacks complete, fresh, citable context. This article explains why agent retrieval must collapse many source-specific APIs into one governed entry point, covering paradigm shifts, long-tail coverage, a five-layer architecture, evaluation, scenarios, and common engineering pitfalls.
Agents can already carry tasks such as writing code, assembling a research brief, or checking public dependency risk to a fairly complete finish. What usually separates strong systems from weak ones is not swapping in a larger model, but whether each action starts with context that is complete enough, fresh enough, and checkable against sources.
The traditional pattern is one retrieval interface per information type: web, documents, code, market data, and community discussion each live on their own. As interfaces multiply, auth, quotas, fields, and failure semantics diverge. The planner then hard-codes “call this first, fall back to that.” The mash-up may work in a demo; in multi-step work it shows up as coverage gaps, broken citations, and runaway tail latency. Agent-oriented retrieval needs to converge from many interfaces into one governable entry point.
Why retrieval is changing again
Treat retrieval as four stages, not a single product launch.
First: directories and keywords. People rewrote queries as terms; systems returned link lists; judgment stayed with the human.
Second: vertical engines and link analysis. Sites and genres got separate indexes. Ranking used link structure and click feedback, but results were still built for people to click.
Third: retrieval-augmented generation. Fragments are recalled, then a model composes an answer. Citations matter, but queries are still mostly single-turn and single-source.
Fourth: agent retrieval. The model plans sub-questions, calls retrieval repeatedly, and uses evidence to decide whether to search again, switch sources, or drop claims that cannot be verified. If the entry point is still a pile of incompatible APIs, the planner spends time on protocol glue instead of evidence.
A unified entry is not dumping every dataset into one store. It means a stable tool contract for the agent: structured query intent in; a result set with sources, timestamps, confidence cues, and citable snippets out. Multiple indexes and connectors can sit behind it; callers see one semantics.
Why long-tail information never appears in “one search”
Public web indexes cover pages that can be crawled, normalized, and refreshed on a relatively stable cadence. Much of what actually helps a task sits at the edge of those indexes: dynamic pages that need interaction, fast-moving notices and market data, issues and commit messages in code repos, forum experience posts, and internal documents. These are not a mysterious second internet. They are high crawl cost, unstable structure, and clear permission boundaries.
Saying “X percent cannot be found” is not verifiable and does not guide engineering. A better question is whether the current task is missing public web pages, structured data, or private documents inside an authorization boundary.
Unified retrieval should distinguish three scopes explicitly.
Public, citable material may enter the final answer with a link or bibliographic identifier.
Authorized material is recalled only when the caller’s credentials are valid, and originals should not be forwarded by default.
Unavailable material—out-of-scope pages and unverifiable paraphrases—should return “no citable evidence found,” not a model-invented source.
Security-related work follows the same boundary: retrieve dependency lists, licenses, and public advisories and summarize them for comparison. Do not turn the retrieval system into a channel that delivers exploit steps.
Layered architecture
A practical entry point usually has five layers. Each can be replaced independently.
Query understanding. Split natural language into retrieval units: entities, time range, genre (web, code, notice, discussion), whether sources are required, and acceptable freshness. One user question often contains several independently searchable sub-questions. This layer splits them and marks dependencies so a vague query is not fired at every index.
Source routing. Choose connectors by genre and permission, not by brand. Code questions prefer code and docs indexes; time-sensitive numbers prefer structured data APIs; background synthesis can add public web. Routing needs degradation: if the primary source times out, return partial results marked incomplete instead of retrying until the whole agent chain dies.
Recall and rerank. Recall optimizes coverage; rerank optimizes alignment with the current sub-question. Features can include semantic similarity, publish time, source stability, and whether a snippet is internally consistent. Do not pretend there is a universal mystery score. Cross-source dedup should keep independent sources for the same fact and drop republished copies, not mutually corroborating reports.
Evidence packaging. Return short snippets, locators, crawl or publish time, and whether citation is allowed—not a full HTML page. Snippets that are too long crowd the context; snippets that are too short drop qualifications. Pair claims with evidence so later generation can only state sentences the evidence supports.
Observability and budget. Log routing decisions, hit counts, truncation reasons, and end-to-end latency on every call. Agents are multi-step; P95 and failure rate matter more than average latency. Put latency budgets in the tool description so the planner can decide whether another hop is worth it.
A vendor-neutral tool contract can look like this:
search(query, constraints) -> {query: string,constraints: {freshness: "any" | "day" | "week",genres: ["web", "code", "doc", "structured"],must_cite: boolean,max_latency_ms: number}}returns:{results: [{text: string,source: { title, locator, published_at },genre: string,incomplete: boolean}],coverage_note: string}
Callers depend only on these fields. Caching, object storage for snapshots, and message queues for async index refresh are generic cloud capabilities. If a team already has object storage, queues, and monitoring on Baidu AI Cloud, the same layers can be assembled there without baking retrieval logic into a single connector.
How to evaluate, not how to praise
“Richer results” is not a metric. Richness can be duplicate stacking or the addition of a critical qualifier. On your own task set, lock four measurements and do not import unexplained leaderboard numbers.
Completeness: whether every point the gold answer depends on is covered by some citable evidence. Extra unrelated paragraphs do not count.
Accuracy: whether statements can be checked sentence by sentence against the matching snippet, without swapped times, numbers, or subjects.
Traceability: whether each key conclusion has a locator. Model-added “according to reports” is not a source.
Latency and stability: timeout rate, empty-result rate, and tail latency under the same concurrency—not one fast demo.
When comparing retrieval stacks, freeze the same model, same prompt, and same task set, and change only the retrieval layer. Otherwise you cannot tell model skill from evidence quality. Source choice is on the same order of importance as model capability: the same model guesses when evidence is thin and behaves more like constrained execution when evidence is dense. That is observable in evals—hallucination falls as “must cite and snippets must be verifiable” tightens, while refusal rises. Refusal is the correct behavior when evidence is insufficient.
How it shows up in scenarios
Coding. Split queries into interface semantics, version differences, and known-defect notices—not a single “what does this error mean.” Code snippets need a repo path or docs anchor. When the version is uncertain, mark it so old docs are not treated as current behavior.
Research briefs. List propositions to verify (scale, time, subject, definition), then retrieve one by one. Do not force inconsistent figures into one conclusion; present them side by side and state the statistical scope. Forum experience can be a lead, not a standalone quantitative source.
Compliance-oriented dependency audit. Input is package names and version ranges. Retrieve license text and public advisories. Output is what was found, what was not found, and the date evidence stops. No advisory is not “no risk”; that negation must be written out. Do not auto-generate exploit steps or attack scripts from retrieval results.
Easy engineering holes
First, caches that amplify error. Time-sensitive data needs short TTLs and origin times on results. Static docs can cache longer, but version changes must invalidate.
Second, inconsistent failure semantics. Some connectors use an empty list for no hits; others throw on no permission. The unified entry must separate no results, no permission, timeout, and partial success. Otherwise the planner treats “no access” as “this fact does not exist.”
Third, optimizing the first-page look. Agents consume snippets, not clickbait titles. Rerank for citable fragments, not title CTR.
Fourth, ignoring permissions. When private-document connectors share a result list with public indexes, mark visibility so a summary is not written into an answer visible to external users.
A practical rollout order: define the tool contract and eval set first; wire one or two painful sources; get citation and failure semantics right; then add connectors. Connector count is not the advantage. Verifiable coverage and stable latency are. If a team still writes a separate agent plugin per data source, it is still on the tail of the third paradigm. When the planner faces one constrained retrieval entry and can decide whether to search again from incompleteness flags, retrieval becomes agent infrastructure rather than another set of interfaces someone has to babysit.
