Enterprise AI Web Crawlers: Architecture and Production Practices

Define the Problem Before Building

An “AI crawler” is not simply a system that sends page text to a large language model. Its real purpose is to continuously and controllably transform web pages into searchable, traceable, and reviewable structured data within clearly authorized and public boundaries. Typical use cases include retrieval-augmented generation, knowledge base updates, public competitor monitoring, and internal research.

Many teams begin with the web-reading capability built into an AI coding tool. This approach is useful for validating an idea: provide a few URLs, retrieve the main text, and generate a summary. In production, however, critical gaps quickly appear. There may be no queue or priority system, no incremental crawling or deduplication, no failure classification, no rate or politeness strategy, and no evidence chain for extracted data. The correct approach is to treat “reading one page” as a single stage in a pipeline, rather than tying the entire system to one conversational call.

Before implementation, define four things:

• Business questions: What questions must the system answer? Which fields are required? Should data be updated every few minutes, hours, or days?
• Data boundaries: Are only public pages included, or is authorized authenticated content also allowed? Does the data contain personal information?
• Success criteria: Define coverage, field completeness, freshness, duplicate rate, and the pass rate for manual sampling.
• Stop conditions: Specify which websites, status codes, legal signals, or terms-of-service signals require collection to stop immediately.

Without these constraints, concurrency, models, and storage will only amplify mistakes.

Overall Architecture

Split the system into six layers rather than building one long script:

1. Seeding and discovery: Sources may include manually curated lists, sitemaps, internal links, search results, or URLs exported from business systems.
2. Scheduling: Handle deduplication, priorities, domain-level quotas, retries, and cooldown periods.
3. Fetching: Provide both static HTTP fetching and browser rendering when necessary. Both paths should produce the original bytes, final URL, status code, response headers, and crawl timestamp.
4. Normalization: Handle character encoding, main-content extraction, navigation and advertisement removal, language identification, and content hashing.
5. Intelligent extraction: Divide responsibilities among deterministic rules, lightweight models, and large language models. Output structured fields together with evidence snippets.
6. Persistence and quality control: Store raw content in object storage, structured fields in a database, and connect stages through a message queue. Monitoring and alerting should cover failures and data drift.

Object storage, message queues, container platforms, logging services, and monitoring systems are all general-purpose cloud capabilities. When deploying in an environment such as Baidu AI Cloud, select services by these functional categories instead of hard-coding crawler logic to a particular console workflow. Run compute tasks in horizontally scalable containers, store raw pages and snapshots in object storage, manage task state through queues, and send metrics to monitoring systems. This allows the runtime environment to change without requiring a rewrite of the core crawler.

A representative task object can remain stable:

Task
  url, normalized URL, domain, priority
  discovery source, scheduled time, attempt count
  rendering mode: direct | browser
  status: pending | succeeded | retryable | permanent failure | skipped

Downstream components should consume this object instead of privately parsing temporary files created by other stages.

Polite Crawling and Access Control

The first principle of a production crawler is that it must be stoppable, rate-limited, and explainable—not simply optimized to collect as much as possible.

• Read and follow the site’s crawler policies and page-level crawling metadata. If a path is disallowed, mark it as skipped and record the reason.
• Apply rate limits by domain rather than relying only on a global thread count. Configure per-domain concurrency, request intervals, and daily limits separately.
• Use an identifiable and contactable user agent. Retain request timestamps and URLs so that activity can be traced if the site operator asks for clarification.
• Crawl only the paths required by the business use case. Set separate budgets for listing pages, detail pages, and attachment pages to prevent unlimited pagination.
• Access authenticated content only with written authorization, an owned account, or an interface permitted by contract. Isolate credentials and never place passwords in logs or prompts.

When the system encounters access challenges, CAPTCHAs, or explicit refusal, downgrade to a manual process or use an official data interface. Do not investigate ways to bypass access controls. Such workarounds are unstable and can turn a technical system into a compliance incident.

Failures should be classified instead of retried indiscriminately:

• Retryable: Timeouts, connection resets, HTTP 429 responses, and some 5xx responses. Use exponential backoff and domain-level circuit breakers.
• Non-retryable: HTTP 404 or 410 responses, prohibited protocols, unsupported content types, and pages that explicitly prohibit collection.
• Requires manual review: Login walls, geographic restrictions, major structural changes, or consistently low extraction confidence.

After a circuit breaker opens, retain only lightweight probe requests to avoid overwhelming a failing site.

Fetching and Rendering

Prefer direct HTTP fetching. Most news, documentation, and listing pages do not require a browser. Escalate to browser rendering only when the target fields appear after client-side execution. Both channels must output the same “fetch envelope”:

• Final URL
• HTTP status code
• MIME type
• Response body or DOM snapshot
• Redirect chain
• Request duration

This allows parsers to remain independent of the fetching method.

Useful implementation details include:

• Normalize URLs: Standardize the protocol, hostname casing, default ports, tracking parameters, and fragments. Use the normalized URL for deduplication while retaining the original URL for replay.
• Use conditional requests: Store cache validators and skip parsing and model calls when content has not changed. This is one of the most important cost-control mechanisms.
• Define a content-type allowlist: Accept HTML, plain text, and public documents. Ignore image streams, audio, video, and unknown binary files unless the business case explicitly requires them.
• Set size limits: Truncate oversized responses and mark them accordingly so that abnormal content cannot exhaust the parsing process.
• Set rendering budgets: Browser pools need concurrency limits, per-page timeouts, and memory reclamation. If rendering fails, fall back to the direct-fetch result instead of reopening the page indefinitely.

For dynamic pages, distinguish between data available in the initial response and data revealed only through subsequent interaction. If the page exposes a public data interface that provides the same information, prefer that structured response. It must still comply with rate limits and authorization requirements. Do not simulate endless scrolling merely for convenience unless it is the only public source and complies with the site’s rules.

From Main Content to Structured Fields

Use three extraction tiers, ordered from lower to higher cost:

1. Deterministic rules: Stable elements such as titles, dates, authors, and breadcrumbs can be extracted with selectors or layout templates. Version templates so that changes can be rolled back.
2. Lightweight models or statistical methods: These are suitable for identifying the main content region, detecting language, removing duplicates, and classifying pages at high throughput.
3. Large language models: Use them when fields are inconsistent, tables and paragraphs are mixed, or cross-paragraph synthesis is required. Provide cleaned content segments rather than full pages containing navigation and scripts.

Large language model calls should use schema constraints that define field names, types, nullability, enumerations, and units. Every output must include evidence: an original sentence or table cell supporting the field. If no evidence exists, mark the field as unknown. Do not allow the model to invent numbers that do not appear on the page.

Extraction result
  field: Publication date
  value: 2024-03-18
  evidence: “Published on March 18, 2024”
  confidence: high
  extractor version: rules-v3 / model-batch-12

Prompts and schemas are code assets. Version-control them together with the crawler. After a change, run regression tests against a fixed labeled dataset rather than judging quality from a few live examples.

A practical division of labor is to keep stable fields with deterministic rules and use models only for long-tail templates. Sample results by domain and template, with special attention to new templates, low-confidence outputs, and business-critical fields. When systematic bias appears, revise the rules or schema before replacing the model.

Scheduling, Deduplication, and Incremental Crawling

URL deduplication only prevents repeated downloads. Content deduplication determines whether a page has actually changed. Both levels are necessary.

• URL level: Store normalized URLs in a persistent set. Use leases for tasks in progress so that multiple instances do not crawl the same URL concurrently.
• Content level: Compute a hash of the main content. If the hash is unchanged, update only the “last seen” timestamp.
• Business level: When the same news item is reposted in multiple sections, merge records using title, date, and content similarity while retaining every source URL.

Scheduling strategies include:

• Crawl seed pages and home pages frequently, while checking historical detail pages less often.
• Shorten the next interval for pages that change frequently. For pages that remain unchanged, extend the interval exponentially, subject to a maximum.
• Give newly discovered links a one-time high priority so that they are not starved by repeated crawling of historical pages.
• Use an independent token bucket for each domain and maintain a global model-call budget to prevent extraction costs from growing uncontrollably.

The correct incremental state is not simply “yesterday’s date.” It should include the last successful content hash, the last crawl time, and the next scheduled time. Clock changes, time zones, and changes in listing-page order should all be considered in the design.

Storage, Traceability, and Quality

At minimum, retain three categories of data, each with a defined retention period:

• Raw fetch envelopes: Used for dispute investigation and parser replay. These are large and should be stored in object storage, partitioned by date and domain.
• Normalized main content: Used for search and re-extraction.
• Structured records: Used for business queries and containing the extractor version, evidence, and quality-control status.

Every record should answer the following questions: When was it fetched? Which URL was fetched? What redirects occurred? Which rule or model version was used? Which source text supports each field? Was the record later corrected by a reviewer? Without an evidence chain, a knowledge base cannot distinguish a change on the source site from an extraction error.

Recommended daily, domain-level dashboard metrics include:

• Crawl success rate, skip rate, circuit-breaker count, and average latency
• Non-empty main-content rate, template match rate, field completeness, and low-confidence ratio
• Sampling accuracy, manual correction rate, duplicate rate, and freshness from publication to ingestion
• Model-call count, per-page cost, and cache hit rate
• Queue backlog and processing capacity

Alerts should be actionable. Examples include a domain’s success rate falling below a threshold, a template’s completeness declining continuously, a sudden increase in model cost, or a queue backlog exceeding processing capacity. Link alerts to representative failure samples instead of reporting only an aggregate percentage.

Security and Engineering Hygiene

A crawler actively accesses untrusted content, so it is also an attack surface.

• Run parsers and renderers in isolated environments, restrict outbound access, and prevent rendered pages from freely reaching internal network addresses.
• Check downloaded files for type and size before processing them. Do not open macros or executable content on crawler machines.
• Load keys, proxies, and accounts from a secrets manager. Keep them out of repositories, prompts, and publicly accessible snapshot directories.
• Redact logs by masking tokens, email addresses, and phone numbers in query parameters.
• Update dependencies and base images regularly. Security patches for rendering components should take priority over feature iteration.

If the results will be exposed through a public-facing service, add a publication gate. Only quality-approved versions should be visible to search, while raw crawl data should not be directly accessible to end users by default.

Recommended Implementation Sequence

Do not deploy a full browser cluster on the first day. A more reliable sequence is:

1. Choose one domain and one page type. Define the fields manually and prepare 20–50 labeled samples.
2. Connect direct fetching, content cleaning, rule-based extraction, and object storage. First ensure that every result can be replayed.
3. Add domain-level rate limits, deduplication, incremental crawling, and failure classification. Run continuously for several days and review the metrics.
4. Introduce models for fields that rules cannot cover. Require evidence and regression tests. Add browser rendering only when necessary.

Each stage should be measurable through field accuracy on the sample set, daily crawl volume, per-page cost, and the percentage of manual corrections. Stop expansion when the metrics do not meet expectations. Scaling improves throughput; it cannot solve unclear objectives or untrustworthy extraction.

Common Failures and Corrections

• Treating conversational “read this webpage” functionality as a production dependency: Replace it with queued tasks and retain the original content and version information.
• Crawling an entire site indiscriminately: Apply content-type budgets and policy checks. Prioritize legality and usefulness.
• Sending every page through a browser and a large language model: Prefer direct HTTP fetching and deterministic rules. Use models for long-tail cases.
• Storing only extracted results: Preserve source snippets and fetch envelopes so that outputs can be audited.
• Retrying every failure: Classify failures and use backoff and circuit breakers.
• Comparing models without a labeled dataset: Build a template-stratified regression set before evaluating replacements.

Conclusion

The professionalism of an AI crawler is not measured by how new its model is. It is measured by clear boundaries, respectful collection, reviewable extraction, governable failures, and predictable costs. Intelligent extraction is only one enhancement layer. Scheduling, storage, quality control, and compliance are what allow the system to run reliably over time.

With this layered architecture, teams can replace individual components—whether page reading is provided by in-house services or general-purpose cloud compute and storage—without rebuilding the entire system.