Why Search Becomes a Bottleneck for AI Agents

In the past, the main goal of a search system was to help users find web pages. A user entered keywords, the system returned links, and the user read, judged, and synthesized the information. In that model, search results only needed to be relevant. The final conclusion was produced by a human.

AI agents work differently. They must not only find material but also read, filter, summarize, reason, and act. For example:

• Query the equity relationships of a company and distinguish official website descriptions from public registry information;
• Find production-grade code implementations rather than conceptual tutorials;
• Determine whether a network address is associated with malicious activity, exploitation, or anomalous behavior;
• Aggregate market data from multiple sources and indicate time, definitions, and credibility;
• Generate runnable API examples based on the latest technical documentation.

A common requirement across these tasks is that results cannot merely look plausible. They must carry sufficient evidence, context, and traceability.

Conventional search often has four limitations.

1. Limited Data Coverage

The public web is only part of the internet. Much valuable information is distributed across code repositories, forum discussions, technical documentation, announcement pages, data APIs, industry databases, and dynamically rendered pages. Some content is not fully covered by conventional web indexes, and some pages require authentication, pagination, or specific request methods.

As a result, even an agent with strong reasoning capabilities may fail to obtain key evidence because its retrieval entry points are limited.

2. Results Are Designed for Humans, Not Machine Processing

Search results usually consist of titles, snippets, and links. This format is intuitive for humans, but an agent must perform additional work:

1. Determine whether the page actually answers the question;
2. Identify entities, time, values, and conditions on the page;
3. Distinguish main content from comments, ads, and navigation;
4. Extract structured fields for downstream reasoning;
5. Preserve the original source for review and citation.

If all this work is delegated to a large model, the system incurs higher latency and less controllable cost, and page noise can easily lead to incorrect judgments.

3. Insufficient Freshness

Technical documentation, vulnerability information, market data, and policy content are all time-sensitive. A conclusion that was correct several months ago may be invalid today. Static knowledge cannot replace real-time retrieval. A search system must explicitly record when data was retrieved, when it was published, and what scope it applies to.

4. Search Results Lack Explicit Credibility

The same question may produce multiple conflicting answers. If a system returns only a synthesized paragraph without showing sources, evidence snippets, and conflicts, users cannot easily judge whether the conclusion is reliable.

For agent-oriented search, the core objective should therefore shift from “returning web pages” to “providing verifiable data evidence.”

A Paradigm Shift in Search Infrastructure

A new search system designed for agents can be understood as an infrastructure layer between large models and internet data. It is not just a search box. It is an intermediate layer responsible for connecting, cleaning, understanding, organizing, and delivering information.

Its basic pipeline can be abstracted as follows:

User task
   ↓
Intent recognition and query decomposition
   ↓
Multi-source retrieval and page access
   ↓
Content cleaning, deduplication, and structured extraction
   ↓
Source ranking, evidence aggregation, and conflict detection
   ↓
Structured result delivery
   ↓
Agent reasoning, citation, and execution

The biggest difference from conventional search is that “search results” are designed as data objects that agents can continue to process, rather than page lists intended only for human readers.

A qualified result object should usually contain at least the following:

{
  "title": "Information title",
  "url": "Source URL",
  "snippet": "Evidence snippet directly relevant to the question",
  "source_time": "Content publication or update time",
  "retrieved_at": "Retrieval time for this request",
  "entities": ["Entity A", "Entity B"],
  "facts": [
    {
      "field": "Field name",
      "value": "Field value",
      "evidence": "Corresponding source text"
    }
  ],
  "confidence": "high"
}

The key is not the exact set of fields. The key is that every conclusion can be traced back to specific evidence.

Multi-Source Access Expands Visibility

Agent-oriented search cannot rely on a single web index. It should select different data sources based on the task type.

1. Web Pages and Documents

This source type is suitable for news, product descriptions, technical articles, policy documents, and public reports. The system needs to handle main-content extraction, pagination merging, duplicate-content filtering, and dynamic content loading.

2. Code and Technical Materials

Code retrieval cannot rely only on keyword matching. It must also understand language, framework, version, dependencies, and runtime environment. For questions such as “how to implement a feature,” the system should prioritize complete context, including function definitions, configuration files, invocation methods, and license information, rather than isolated lines of code.

3. Community Discussions and Q&A Content

Community content often contains real troubleshooting experience and edge cases, but its credibility varies widely. A retrieval system must distinguish questions, answers, comments, and personal speculation. It must not treat unverified opinions as facts.

4. Structured Data Sources

Domains such as stocks, weather, geography, business registration, vulnerabilities, and statistics often provide structured APIs or standardized data files. Such data is better passed directly to agents, but field definitions, units, time ranges, and data versions must be preserved.

5. Pages Requiring Special Access Methods

Some content requires handling authentication, pagination, script rendering, or rate limits. System design should define permission boundaries clearly and comply with the access rules of target sites. “Accessible” must not be misunderstood as “freely collectible.”

Query Planning Determines Search Quality

A user question is usually not a simple keyword but a composite task. For example:

Analyze whether a network address poses a security risk and provide the most recent related record.

This question can be decomposed into at least the following sub-questions:

• What is the basic attribution information for the address?
• Does it appear in public threat intelligence?
• Is it associated with a specific malicious sample, domain, or event?
• What is the timestamp of the related records?
• Do different sources conflict with one another?
• What is the confidence level of the conclusion?

A search system should therefore support query planning rather than sending the original question directly to the retrieval module.

A practical planning process includes:

1. Identify the task type: fact lookup, comparative analysis, code retrieval, risk assessment, or open-ended research;
2. Extract entities: people, organizations, addresses, technical components, time ranges, and so on;
3. Generate multiple retrieval expressions: synonyms, abbreviations, field names, and constraints;
4. Select data sources for different sub-questions;
5. Set stopping conditions: sufficient evidence, conflicting sources, or the need for human confirmation;
6. Merge results into a citable evidence set.

Query planning can also reduce hallucination risk in large models, because the model no longer answers directly from memory. It must organize its answer around the evidence it has obtained.

Structured Output Matters More Than Long Summaries

Many so-called “intelligent search” systems ultimately generate only a long summary. Summaries are readable, but they are not suitable for automated workflows.

In an enterprise research scenario, for example, an agent may actually need:

• Organization name;
• Registration date;
• Related entities;
• Shareholding percentage;
• Data source;
• Information update time;
• Uncertain fields.

If the system returns only a natural-language paragraph, downstream programs must call a model again to parse it, which can easily lead to missing fields and inconsistent formats.

A more reasonable approach is to provide three layers of results at the same time:

1. Raw evidence layer: retain web snippets, document paragraphs, or API responses;
2. Structured fact layer: extract entities, attributes, values, time, and relationships;
3. Natural-language explanation layer: generate readable conclusions for users.

These three layers make the result easier for humans to understand and for programs to continue processing.

Sources, Time, and Conflicts Must Be Explicitly Managed

The credibility of a search system is not determined by how fluent its language is. It is determined by its evidence chain.

Each fact should record at least the following information:

• Source URL or source identifier;
• Page title;
• Retrieval time;
• Original publication time;
• Location of the evidence;
• Extraction method;
• Source type;
• Confidence;
• Whether a conflict exists.

When multiple sources provide different results, the system should not simply choose one answer. It should present the differences. For example:

Source A: record time is March 2025
Source B: record time is April 2025
Possible reason: different update times or different statistical definitions
Recommendation: prefer the source with the more recent publication time and clearer field definitions

This kind of expression is more suitable than “the system automatically determined a certain value” for high-risk scenarios such as finance, law, security, and enterprise research.

Designing a Search Interface for Agents

Interface design should center on tasks, not pages. A general request can include:

{
  "query": "Question to research",
  "task_type": "research",
  "time_range": {
    "start": "2025-01-01",
    "end": "2025-12-31"
  },
  "required_fields": ["conclusion", "evidence", "source_time"],
  "max_results": 10,
  "need_citations": true
}

The response should include:

• Search results;
• Evidence snippets;
• Structured fields;
• Source information;
• Relevance scores;
• Credibility notes;
• Unresolved questions;
• Suggested next queries.

In engineering implementation, the following issues should also be considered.

Caching

For queries repeated within a short period, raw results and structured results can be cached. Real-time data must have a short validity period, and the cache time must be clearly marked.

Timeouts and Degradation

Multi-source retrieval may encounter slow responses from individual sources. The system should support partial success: even if one data source is unavailable, it should return results from other sources and describe what is missing.

Rate Limiting and Retries

Different sources have different access limits. Exponential backoff, request queues, and concurrency control must be used to avoid instability caused by high concurrency.

Observability and Auditing

The system should record query latency, source hit rate, parsing failure rate, duplication rate, citation coverage, and user feedback. These metrics are the foundation for continuously improving search quality.

Evaluating Whether a Search System Is Effective

Evaluation should not focus only on search speed or the number of returned results. It should be based on the final quality of completed agent tasks.

The following metrics are recommended:

1. Recall: Are key facts retrieved?
2. Precision: How much of the returned content is directly relevant to the question?
3. Evidence coverage: Does the final conclusion have supporting sources?
4. Freshness: Do the results meet time requirements?
5. Structured accuracy: Are fields and values extracted correctly?
6. Conflict detection rate: Can the system proactively flag inconsistencies across sources?
7. End-to-end success rate: Can the agent complete follow-up tasks based on the results?
8. Latency and cost: Does the system respond within acceptable resource consumption?

The test set should cover fact-based Q&A, code retrieval, security analysis, enterprise research, and multi-hop reasoning. It should also retain human-annotated reference answers and evidence locations.

Practical Recommendations for Developers

If a team plans to provide unified search capabilities for agents, it can proceed in the following order.

Step 1: Define Task Boundaries First

Do not attempt to cover the entire internet from the beginning. Prioritize high-frequency scenarios that can be evaluated, such as technical material retrieval, internal knowledge lookup, or security event analysis.

Step 2: Define a Unified Result Protocol

First determine how sources, evidence, time, structured fields, and confidence should be expressed in results. Without a unified protocol, multi-source integration will quickly lead to format chaos.

Step 3: Establish Evidence-First Prompts and Workflows

Require agents to list evidence before forming conclusions. For fields without evidence, explicitly return “cannot confirm” rather than filling in guesses.

Step 4: Add Human Review Mechanisms

For high-risk conclusions such as security response, financial judgment, and compliance recommendations, search results should be used as supporting material, with a human confirmation step retained.

Step 5: Continuously Build Evaluation Sets

Each time an error occurs, record the problem type: not found, found but misjudged, outdated source, field parsing error, or missing citation. Over time, these cases will become the most valuable regression test set.

Conclusion

The ceiling of agent capability depends not only on model scale but also on whether the agent can obtain timely, complete, structured, and verifiable information. Search is shifting from a tool that “helps people find web pages” to infrastructure that “helps agents connect to the real world.”

A more mature search system in the future should not only answer “what was found.” It should also answer:

• Where did the information come from?
• When was it produced?
• Why should it be trusted?
• Which conclusions remain uncertain?
• What can the agent do next?

When multi-source data access, query planning, structured extraction, evidence management, and task evaluation form a closed loop, agents will truly have the foundation to complete complex work reliably. For developers, building a high-quality search layer is no longer merely adding a retrieval plugin. It is establishing a reliable information entry point for the entire intelligent application.