Search Infrastructure for AI Agents: Design and Practice

Why Search Is the Hidden Bottleneck for AI Agents

Over the past year, AI agents powered by large language models have advanced rapidly. Writing code, generating research reports, executing security audits, and mapping business processes—tasks that once required close human guidance—can now be completed by agents through most stages. However, when a task shifts from “generating content” to “obtaining facts,” a seemingly minor problem becomes critical: search.

When asked to check a company’s ownership structure, an agent may only return the company’s official self-introduction. When asked to find code suitable for production use, it may return an introductory tutorial. When asked to analyze threat intelligence for a suspicious IP address, it may only retrieve general explanatory articles. The issue is not insufficient model reasoning, nor slow response speed. The real issue is that search is “blind”: a large amount of high-value information never enters the agent’s field of view.

The core judgment is this: in the agent era, search has evolved from an auxiliary tool for humans into foundational infrastructure for machines. The importance of retrieval quality and source selection is gradually becoming equal to model capability itself. With the same model, stronger retrieval and better context can produce an order-of-magnitude difference in usability.

Why Traditional Search Fails Agents

To understand why a new paradigm is needed, it helps to examine the structural limitations of traditional search engines. They were designed to serve humans, not to be consumed by machines.

1. Optimization is for human reading, not machine consumption

Traditional search returns a list of web links. It implicitly assumes that a human will click through the results one by one, filter them, judge their credibility, and piece together the information. Agents cannot perform this manual post-processing. They need structured content that can be fed directly into the reasoning chain, not a collection of URLs.

2. Coverage is biased toward the surface web

Search engine crawlers can reliably access mainly static, public, and well-structured pages. Much high-value information is located in the so-called deep web: content that requires login, dynamically rendered pages, structured databases, forum and community discussions, specific implementations inside code repositories, real-time market data, and more. These sources are either difficult to crawl or difficult to parse effectively after retrieval.

3. Limited semantic understanding and result integration

Traditional retrieval is often based on keyword-based inverted indexes, such as classic BM25-style algorithms. It matches literal text rather than intent. It does not perform deduplication, body-text extraction, cross-source verification, or summary compression for the user—yet these capabilities are essential for agent decision-making.

4. Mismatched invocation patterns

Agents make high-frequency, batched calls. A single task may trigger dozens or even hundreds of retrieval requests. Interfaces designed for low-frequency human browsing often struggle to meet machine-side requirements for throughput, rate limiting, and latency stability.

Required Capabilities for Agent-Oriented Search Infrastructure

Taken together, a qualified search infrastructure for AI agents should cover at least the following capability dimensions:

• Unified multi-source access: Integrate heterogeneous information such as web pages, community discussions, code repositories, structured data sources, and real-time market data through a unified entry point, while hiding underlying differences.
• Deep content extraction: Do not merely return links. Extract main content, remove noise, and deliver ready-to-use content fragments.
• Semantic retrieval: Support natural-language intent matching, not just literal matching.
• Result integration and ranking: Perform cross-source deduplication, credibility-based ranking, relevant-fragment aggregation, and necessary summarization.
• Stable machine-facing interfaces: Provide unified APIs, clear authentication and rate limiting, and predictable low latency with high concurrency support.

Reference Architecture: A Layered Design

The following is a vendor-neutral layered reference architecture. It can serve as an evaluation framework for building an in-house system or selecting an existing solution.

┌─────────────────────────────────────────────┐
│              API Gateway Layer              │
│  Unified authentication / rate limiting /   │
│  quotas / multi-tenant isolation            │
├─────────────────────────────────────────────┤
│            Result Consolidation Layer       │
│  Body extraction · deduplication ·          │
│  reranking · summary compression            │
├─────────────────────────────────────────────┤
│               Retrieval Layer               │
│  Hybrid retrieval: keyword inverted index   │
│  (BM25) + vector semantic retrieval         │
├─────────────────────────────────────────────┤
│             Index Construction Layer        │
│  Collection → Cleaning → Chunking →         │
│  Embedding                                  │
├─────────────────────────────────────────────┤
│            Multi-Source Ingestion Layer     │
│  Web crawling · data-source adapters ·      │
│  real-time data streams                     │
└─────────────────────────────────────────────┘

When implementing the infrastructure, each layer can map to common cloud capabilities. Raw collected data and index snapshots are well suited for object storage. Collection and embedding pipelines can use message queues for asynchronous decoupling and traffic peak shaving. Retrieval and consolidation services are suitable for container platforms to support elastic scaling. Logging, monitoring, and alerting should also be integrated to establish observability for retrieval latency, recall rate, and failure rate. This combination is supported by general-purpose product categories on major cloud platforms, such as Baidu AI Cloud, and can be matched according to actual requirements during selection.

Key Implementation Details

1. Hybrid Search: Keyword + Vector

Each retrieval method has limitations. Keyword search is precise but weak with synonymous expressions. Vector search is strong in semantics but may miss proper nouns. Production environments usually adopt hybrid retrieval and fuse the scores:

def hybrid_search(query, top_k=10):
    # Keyword recall: good for proper nouns and exact matches
    lexical_hits = bm25_index.search(query, top_k=top_k * 2)
    # Vector recall: good for semantics and synonym rewrites
    vector_hits = vector_store.search(
        embed(query), top_k=top_k * 2
    )
    # Fuse after score normalization (illustrative: weighted summation)
    fused = reciprocal_rank_fusion(lexical_hits, vector_hits)
    return fused[:top_k]

2. Chunking and Embedding Strategy

After body-text extraction, content should be split into chunks according to semantic boundaries. If chunks are too large, relevance is diluted. If they are too small, context is lost. In practice, content is often split by paragraphs or heading levels, with a certain amount of overlap retained. Each chunk is then embedded and written into a vector store.

3. Reranking

The recall stage usually returns many candidates with coarse scores. Before entering the agent context, a reranking step should be performed. A stronger cross-encoder is used to score query-document pairs and move the most relevant fragments to the top. This step often has a more significant impact on final answer quality than simply expanding the recall set.

4. Content Extraction and Cleaning

For crawled pages, navigation, advertisements, scripts, and other noise should be removed, leaving only the main content. The quality of this step directly determines whether the agent sees real content or template text. It is also key to improving the visibility of deep information.

Practical Recommendations and Best Practices

1. Prioritize machine consumability: Output structured content rather than link lists so agents can enter the reasoning process directly.
2. Evaluate recall and reranking separately: Monitor recall rate and reranking effectiveness independently, and avoid conflating the two types of issues.
3. Model source credibility: Sources differ significantly in authority. Credibility weighting should consider source type, timeliness, and cross-source verification.
4. Optimize for high-frequency calls: Cache hot results, use batching, and apply asynchronous prefetching to control cost and latency.
5. Build end-to-end observability: Establish metrics and alerts for latency, failure rate, and retrieval quality, and iterate continuously.

Conclusion: The ceiling of agent capability increasingly depends on how much high-quality information it can “see.” Designing search as infrastructure—with multi-source access, deep extraction, hybrid retrieval, result consolidation, and stable interfaces—is the key path to overcoming the current retrieval bottleneck for agents. This approach does not depend on any specific product. Developers can use it to build their own systems or evaluate existing solutions, extending an agent’s field of view from the surface web to the full extent of the internet.