Search Infrastructure for the Agent Era: Accessing the Invisible Web

1. What Really Blocks Agents: The Ability to See Relevant Facts

Over the past two years, agents have significantly reduced manual involvement in tasks such as code generation, document writing, log analysis, and vulnerability auditing. However, when these capabilities are applied to real business workflows, failures often occur earlier than the reasoning stage: the agent cannot obtain the correct facts.

Several high-frequency failure scenarios look very similar:

• To investigate a company’s ownership structure and affiliated relationships, the agent only retrieves the company’s website introduction.
• To find a production-ready implementation, the agent retrieves a beginner-level tutorial.
• To analyze threat intelligence for a suspicious IP address, the agent can only find a few general introductory articles.

These failures are not because the model is “not smart enough.” They happen because the input corpus itself is incomplete. A widely cited observation in the industry is that the same model, when paired with stronger and more complete retrieval context, often shows a much larger usability difference than when the model itself is replaced.

The reason is straightforward: probabilistic generation in language models cannot compensate for missing facts. If information is absent, the model cannot recover it. It can only fill the gap more confidently with fabricated content.

1.1 Why Traditional API Integration Fails in Agent Scenarios

An early approach was to create one tool for each data source: one for business registration information, one for code hosting, one for academic literature, and one for public sentiment monitoring. The longer the agent runs, the more obvious the problems become:

• Tool proliferation causes context bloat. Tool descriptions consume reasoning budget, reducing the probability that the agent selects the correct tool.
• Each interface has different authentication methods, pagination rules, field naming conventions, and error codes. The agent has to learn multiple “dialects.”
• If any source is rate-limited or fails, the entire chain fails. There is no unified degradation or retry semantics.
• Returned results have different formats, making it difficult for downstream components to perform unified citation, deduplication, and provenance tracking.

The conclusion is not that APIs should not exist. The conclusion is that APIs should not be directly exposed to the model. A better architecture inserts a unified retrieval layer between data sources and the agent, converting multiple heterogeneous APIs into one semantically stable tool.

1.2 Four Shifts in Retrieval Paradigm

Stage	Interaction Pattern	Main Limitation
Directory navigation	Manual category browsing	Limited coverage and dependent on manual maintenance
Keyword search	User enters keywords and receives a list of links	Users must filter, open, and compare results themselves
Generative question answering	The system directly outputs an answer summary	Prone to hallucination and difficult to trace sources
Agent-native retrieval	Multi-source parallel retrieval, structured evidence, verifiable citations	Requires strong engineering capabilities, including quality and compliance systems

The key difference in the fourth stage is that retrieval results are no longer “links for humans to read.” They are “structured evidence for models to consume.” This changes the design goal of the entire pipeline from “how good is the ranking?” to “is the evidence complete, verifiable, and delivered in time?”

2. What Is the Invisible Web?

The invisible web does not mean that search technology has regressed. It means that certain content is naturally outside the coverage of general-purpose crawlers. It mainly comes from six categories:

1. Content behind login walls or paywalls. Crawlers have no credentials to access it.
2. Pages that require JavaScript rendering or interaction to generate content. Static crawling can only obtain an empty shell.
3. Non-HTML resources, such as code repositories, issue discussions, configuration examples, and dataset files.
4. Semi-public community content, which often has deep threads, rapid updates, and complex URL structures.
5. Structured data sources whose fields can only be obtained through APIs or forms, rather than by traversing links.
6. Highly time-sensitive short-lived content, where indexing cycles cannot keep up with publishing frequency.

If we map these categories to three typical tasks, the coverage gaps become clear:

• Entity relationship information often falls into categories 4 and 5.
• Production-grade code implementations are concentrated in category 3.
• Threat intelligence, market data, and breaking news are concentrated in category 6.

A general-purpose search engine naturally has a low hit rate for these three types of needs.

3. Four Core Capabilities of an Agent Retrieval Layer

To fill these gaps, the retrieval layer must meet four requirements at the same time. If any one of them is missing, the system will degrade into “just another search box.”

Coverage breadth and multi-source access. Federated search should be the backbone. One query is dispatched in parallel to multiple heterogeneous backends, and their results are fused in a unified layer. General search provides broad recall, while vertical sources add depth.

Structured output. The retrieval layer should not return a plain summary. It should return standardized evidence objects with fields such as title, source type, snippet, publish time, fetch time, original link, and confidence score. The degree of structuring determines whether downstream components can perform secondary reasoning and precise citation.

Trust and verifiability. Every conclusion must be traceable to specific evidence and a specific location. This requires sentence-level or paragraph-level citation granularity, rather than document-level citation. It also requires preserving fetch time and content fingerprints so that conclusions can still be reviewed if the source changes later.

Engineering usability. The retrieval layer needs unified authentication, per-source quotas and rate limiting, timeout budgets, circuit breaking and degradation, result caching, and observability. Agents often involve long multi-turn call chains, and tail latency in the retrieval layer can directly amplify overall unavailability.

4. Reference Architecture and Layer Responsibilities

+-------------------------------------------------------------+
| Agent side: unified search tool (single schema, one entry)  |
+-------------------------------+-----------------------------+
                                |
                                v
+-------------------------------------------------------------+
| 1. Query understanding layer:                                |
|    intent classification / query rewriting / slot extraction |
+-------------------------------+-----------------------------+
                                |
                                v
+-------------------------------------------------------------+
| 2. Routing and scheduling layer:                             |
|    source selection, parallel dispatch, timeout, circuit breaker |
+-------+---------------+---------------+-----------------+
        |               |               |
        v               v               v
  +-----------+   +-----------+   +-----------+
  | General   |   | Code /    |   | Structured|
  | web       |   | docs      |   | source    |
  | adapter   |   | adapter   |   | adapter   |
  +-----+-----+   +-----+-----+   +-----+-----+
        |               |               |
        +---------------+---------------+
                        |
                        v
+-------------------------------------------------------------+
| 3. Normalization and fusion layer:                           |
|    field mapping / deduplication / fusion / reranking        |
+-------------------------------+-----------------------------+
                                |
                                v
+-------------------------------------------------------------+
| 4. Evidence assembly layer:                                  |
|    evidence package + citation IDs + content fingerprints    |
+-------------------------------+-----------------------------+
                                |
                                v
+-------------------------------------------------------------+
| Cache and observability                                      |
+-------------------------------------------------------------+

A common misunderstanding is to ask a large language model to perform “intelligent routing” for layer 1. In practice, a more stable approach is to use lightweight rules and keyword-based classification for intent detection, and only invoke model-based rewriting for ambiguous queries. This reduces latency and prevents the model from changing the user’s original constraints, such as time range, region, or version number.

5. Key Implementation: From Abstractions to Code

5.1 Unified Evidence Model and Adapter Protocol

Mapping all data sources to the same structure is the foundation of the entire pipeline. Once the fields are unified, fusion, reranking, citation, and caching can reuse the same logic.

from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Protocol, Any, List
@dataclass
class Evidence:
    title: str
    url: Optional[str]                 # Leave empty if there is no original link; do not fabricate one.
    snippet: str
    source_type: str                   # web / code / doc / forum / structured_api
    score: float = 0.0
    published_at: Optional[datetime] = None
    fetched_at: datetime = field(
        default_factory=lambda: datetime.now(timezone.utc)
    )
    content_hash: Optional[str] = None  # Normalized content fingerprint for deduplication and verification
class SearchAdapter(Protocol):
    name: str
    timeout_budget: float               # Timeout budget for a single call, in seconds
    async def search(
        self,
        query: str,
        *,
        k: int = 10,
        filters: Optional[dict[str, Any]] = None,
    ) -> List[Evidence]:
        ...

For content_hash, it is recommended to hash the normalized body content after removing whitespace and template navigation. This is a key technique for handling cases where the same content is reposted across dozens of sites. It is much more effective than URL-based deduplication.

5.2 Query Understanding: Turning Natural Language into Executable Slots

The agent often passes in a complete intent, for example: “Find the recent investors and affiliated parties of a company.” Before routing, the system needs to extract slots.

import re
from typing import Optional, Any, List
from dataclasses import dataclass, field
from datetime import date, timedelta
SOURCE_RULES = [
    (("equity", "affiliate", "corporate registry", "funding"), ("structured_api", "web")),
    (("error", "implementation", "issue", "sample code"), ("code", "web")),
    (("CVE", "IOC", "threat intelligence", "sample"), ("security", "web")),
]
@dataclass
class QueryPlan:
    original: str
    rewrites: List[str] = field(default_factory=list)
    entity: Optional[str] = None
    date_from: Optional[date] = None
    sources: List[str] = field(default_factory=lambda: ["web"])
def build_plan(q: str, today: Optional[date] = None) -> QueryPlan:
    today = today or date.today()
    plan = QueryPlan(original=q)
    # 1) Recency intent: if no explicit date is provided, default to the last year.
    if any(w in q.lower() for w in ("latest", "recent", "this year", "recently")):
        plan.date_from = today - timedelta(days=365)
    m = re.search(r"\b(\d{4})-(\d{1,2})\b", q)
    if m:
        plan.date_from = date(int(m.group(1)), int(m.group(2)), 1)
    # 2) Source selection: take the union of matched rules, otherwise fall back to general web search.
    picked = set()
    for keywords, srcs in SOURCE_RULES:
        if any(w.lower() in q.lower() for w in keywords):
            picked.update(srcs)
    if picked:
        plan.sources = sorted(picked)
    # 3) Rewriting: keep the original query, add constraint terms, and do not modify user constraints.
    plan.rewrites = [q]
    if plan.date_from:
        plan.rewrites.append(f"{q} {plan.date_from.year}")
    return plan

5.3 Multi-Source Fusion: Use RRF Instead of Weighted Scores

Scores from different sources are not comparable. A search engine provides relevance scores, code search provides text-matching scores, and an API may provide business confidence scores. Directly normalizing and weighting these scores often allows one source to dominate the final result.

Reciprocal Rank Fusion (RRF) depends only on rank and is a stable choice for heterogeneous fusion:

def rrf_fuse(result_lists: List[List[Evidence]], k: int = 60,
             top_n: int = 20) -> List[Evidence]:
    scores: dict[str, float] = {}
    keep: dict[str, Evidence] = {}
    for results in result_lists:
        for rank, doc in enumerate(results, start=1):
            key = doc.content_hash or doc.url or f"{doc.source_type}:{doc.title}"
            scores[key] = scores.get(key, 0.0) + 1.0 / (k + rank)
            if key not in keep:
                keep[key] = doc
    ranked_keys = sorted(scores, key=lambda x: scores[x], reverse=True)[:top_n]
    out: List[Evidence] = []
    for key in ranked_keys:
        doc = keep[key]
        doc.score = scores[key]
        out.append(doc)
    return out

k=60 is a commonly used empirical value in the industry. Its role is to reduce the absolute advantage of top-ranked results and allow multi-source consensus to emerge. After fusion, adding a cross-encoder reranking stage is often more cost-effective than optimizing recall alone. The reranker sends the query and each snippet as a pair for fine-grained scoring.

5.4 Evidence Package: Making the Model State Only What It Can Cite

The final step of the retrieval layer is not to return a list of documents. It is to return an evidence package with clear constraints and to enforce behavior at the prompt level.

def build_context(evidence: List[Evidence], max_chars: int = 12000) -> str:
    blocks, used = [], 0
    for i, e in enumerate(evidence, start=1):
        meta = (
            f"[E{i}] source_type={e.source_type} "
            f"published_at={e.published_at or 'unknown'} "
            f"fetched_at={e.fetched_at:%Y-%m-%d} "
            f"link={e.url or 'no public link'}"
        )
        body = e.snippet[:1200]
        block = f"{meta}\n{body}"
        if used + len(block) > max_chars:
            break
        blocks.append(block)
        used += len(block)
    return "\n\n".join(blocks)

The accompanying constraints should be fixed as three rules:

1. The model may only use information from the evidence package.
2. Every conclusion must be followed by a citation ID.
3. If the evidence is insufficient, the model must explicitly output “insufficient evidence” and list the fields that still need to be retrieved. Speculative completion is not allowed.

The third rule is especially important. It turns “silence” into a legitimate output mode and prevents the model from using common sense to fill factual gaps.

5.5 Tool Definition: A Stable Contract for the Agent

The search tool exposed to the agent should remain small and stable. The more parameters it has, the more likely the model is to fill them incorrectly.

SEARCH_TOOL = {
    "name": "unified_search",
    "description": "Search across general web pages, code repositories, technical documentation, "
                   "and structured data sources. Returns citable evidence with source, time, and link.",
    "parameters": {
        "type": "object",
        "properties": {
            "query": {"type": "string", "description": "Natural-language search intent"},
            "k": {"type": "integer", "minimum": 1, "maximum": 50, "default": 10},
            "freshness": {
                "type": "string",
                "enum": ["any", "day", "week", "month", "year"],
                "default": "any",
            },
            "need_sources": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Optional. Restricts the source types to be used.",
            },
        },
        "required": ["query"],
    },
}

5.6 Engineering Guardrails: Timeout, Degradation, and Caching

import asyncio, hashlib, json
async def gather_with_deadline(tasks, budget: float):
    """Isolate timeouts per source. A single-source failure does not break the overall response."""
    results = []
    for coro in asyncio.as_completed(tasks, timeout=budget):
        try:
            results.append(await coro)
        except (asyncio.TimeoutError, Exception):
            continue          # Record metrics, degrade gracefully, and do not raise upstream.
    return results
def cache_key(q: str, sources: list[str], k: int, freshness: str) -> str:
    raw = json.dumps([q, sorted(sources), k, freshness], ensure_ascii=False)
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

Cache TTL should differ by source type. Structured business registration data or market data should have a shorter TTL. Technical documentation and specification content can have a longer TTL.

All retrieval calls should be instrumented with metrics: QPS by source, timeout rate, empty-result rate, P95 latency, and cost per call. These metrics determine capacity planning and cost control for the retrieval layer.

6. Measuring Correctness: Build Your Own Evaluation Set

The quality of a retrieval layer cannot be judged by subjective feeling. It requires an offline evaluation set. A minimal usable evaluation set contains 100 to 300 real tasks. Each case records the query, the expected golden evidence, and the key points of the ideal answer.

from typing import Iterable
def evaluate(cases: Iterable[dict], retriever, k: int = 10) -> dict:
    hit, mrr_sum, empty = 0, 0.0, 0
    for c in cases:
        docs = retriever(c["query"], k=k)
        if not docs:
            empty += 1
        ranks = [i for i, d in enumerate(docs, 1)
                 if c["gold_match"](d)]        # Match by URL, fingerprint, or required fields.
        hit += 1 if ranks else 0
        mrr_sum += 1.0 / ranks[0] if ranks else 0.0
    n = max(len(cases), 1)
    return {
        "recall@k": round(hit / n, 4),
        "mrr@k": round(mrr_sum / n, 4),
        "empty_rate": round(empty / n, 4),
    }

Evaluation should cover four categories, and they must be reviewed together:

Category	Metrics	Focus
Retrieval quality	Recall@k, MRR, nDCG	Whether golden evidence appears in the top k
Answer quality	Factual accuracy, citation verifiability	Whether each conclusion can be traced back to evidence
Performance	P50 / P95 latency, timeout rate	Tail behavior in long-chain scenarios
Cost	Tokens and calls per query	Whether the system is sustainable in production

An evaluation that only reports accuracy is basically meaningless. Real user experience is determined jointly by “accuracy × latency × cost.” This is especially important in multi-turn agent scenarios, where retrieval is called repeatedly and tail-latency amplification becomes very obvious.

7. Compliance and Security: A Non-Negotiable Layer

Sending external content into a model context means putting untrusted input directly into the reasoning chain. This is a risk surface that agent retrieval must handle.

• Crawling compliance: Follow the target site’s terms of service and access-control requirements. Control request rates and avoid putting pressure on source sites. Paid or authorized content must be obtained through proper authorization channels.
• Prompt injection protection: Web pages, forum posts, and issue discussions may contain instructions aimed at the model. The principle is to treat retrieved content strictly as data, not as instructions. Wrap retrieved content with explicit delimiters and strip or alert on obviously overreaching instructions.
• Personal information and cross-border data: Data sources involving natural-person information require assessment of legal basis and data minimization. Cross-border scenarios should consider data residency requirements.
• Credential management: Multi-source access means multiple sets of credentials. Use a centralized secret-management service and least-privilege policies. Never write credentials into code or logs.
• Auditability: Retain audit logs for search queries, returned results, fetch time, and content fingerprints. This makes it possible to review the basis of a conclusion after the fact.

8. Rollout Path and Common Pitfalls

A practical rollout should proceed in three steps.

Step 1: Build a single-source POC. Choose the source with the highest business value, such as an internal document library or code repository. Build the “retrieval → evidence → citation” loop and create an evaluation set.

Step 2: Expand to multi-source federation. Focus on tuning RRF parameters and reranking models.

Step 3: Add engineering hardening. Bring caching, rate limiting, circuit breaking, and monitoring into production.

Skipping step 1 and moving directly to multi-source federation usually makes it impossible to determine whether quality problems come from recall or ranking.

There are four common pitfalls:

1. Treating the retrieval layer as an “enhanced question-answering system,” tuning only the model without improving corpus coverage. The problem will simply reappear.
2. Relying only on keyword recall without structured field extraction. Entity-relationship tasks will never perform well.
3. Ignoring freshness annotations. The model may treat an announcement from two years ago as a current fact.
4. Depending on a single source. General search, code search, and vertical data sources have completely different coverage boundaries. Missing one of them creates entire categories of blind spots.

In the fields of retrieval and natural language processing, providers such as Baidu have accumulated long-term technical experience. Capabilities such as multimodal search and semantic understanding can serve as reference points when building an in-house retrieval layer. However, regardless of the implementation chosen, the four core capabilities, evaluation system, and safety boundaries remain common engineering requirements.

Conclusion

The next stage of competition for agents is not about model parameters. It is about fact supply.

The decisive advantage belongs to those who can reliably give agents access to information scattered behind login walls, inside communities, across code repositories, and within structured data sources, and deliver that information to the model in a verifiable form.

The retrieval layer is not a replacement for a search box. It is the interface layer through which agents connect to the real world. It deserves to be built with infrastructure-level standards.