Document OCR in the LLM Era: Evaluation and Deployment

    Document OCR has moved beyond extracting characters from images to recovering layout, reading order, tables, and semantic structure. This article explains how to define the OCR task before comparing models, why closed-source services often lead public comparisons, what to measure beyond character accuracy, how decoding approaches differ, and how to build and validate a maintainable document-to-text pipeline while controlling hallucination, cost, and compliance risk.

    Document optical character recognition (OCR) has evolved from extracting characters out of images to recovering layout, reading order, tables, and semantic structure together. As models become larger and decoding shifts from per-character classification to generation or iterative reconstruction, a single leaderboard score is an increasingly poor guide to model selection. Closed-source services often lead on difficult layouts, while self-hosted models offer more control over data, cost, and customization. A responsible comparison must first state the task, sample distribution, and downstream use; only then should it argue about which system is stronger.

    Define the Task Before Comparing Models

    In engineering, OCR is often used as an umbrella term for at least four capability layers.

    The first is text detection and recognition: locate text regions in scans, photos, or screenshots and transcribe them into character sequences. Key failure conditions include blur, skew, stamp occlusion, and low resolution.

    The second is layout analysis: distinguish headings, body text, headers and footers, footnotes, captions, multiple columns, and reading order. In multi-column papers, contract clauses, and newspapers, incorrect reading order can cause downstream retrieval to splice unrelated passages even when character accuracy is high.

    The third is structured extraction: reconstruct table cells and merged cells, preserve formula symbols and structure, and align key-value fields. Returning only plain text discards the hardest structural information.

    The fourth is document-level consistency: handle continued tables across pages, correspondence between a table of contents and the body, and references between text in figures and the main text. The value of a 100-page PDF is not a single-page demo but that no pages are lost or duplicated across the whole document, and page numbers and block order remain traceable.

    Before evaluation, define the target output schema as an acceptance criterion: plain text, blocks with coordinates, a hierarchical heading tree, or a two-dimensional table structure. Different output schemas make metrics incomparable. Treating “can recognize characters” and “can feed a knowledge base” as one score is a common source of misleading comparisons.

    Why Closed-Source Systems Often Lead on Benchmarks

    In public comparisons, closed-source document-understanding services often have an overall advantage. This usually reflects a combination of data, training objectives, and systems engineering rather than one technique.

    Data coverage is broader. Real business documents include official documents with stamps, invoices, scanned medical records, engineering drawing notes, multilingual pages, and heavily compressed faxes. A closed-source service that has long seen this distribution is often more stable on rare layouts and noise. An open-source or self-trained model trained mainly on clean digital renders can lose noticeable accuracy when moved to mobile-phone photos.

    Training objectives are closer to the final artifact. Traditional pipelines optimize detection, recognition, layout, and tables separately, so errors compound at each stage. End-to-end document models learn directly from page images to structured text and are more sensitive to reading order and cross-block relationships. This is a key part of the recent paradigm shift: decoding is no longer just position-by-position classification over a vocabulary, but conditional generation from page semantics or iterative denoising and reconstruction of the whole page. Generative decoding can use longer context to correct locally blurred characters, but it may also “complete” words that were not in the source. Evaluation must therefore treat hallucination separately.

    The system side includes complete pre- and post-processing. Deskewing, shadow removal, super-resolution, page splitting, specialized heads for formulas and tables, confidence filtering, and human review interfaces all affect the final result, yet they are rarely documented in a model card. Comparing bare model weights with a service that includes full preprocessing will favor the service, but that does not mean the backbone network itself is stronger.

    Therefore, “closed source is far ahead” should be read as an empirical observation about current system-level capabilities, not a permanent law of a given architecture. Once a self-hosted approach closes the data loop and fills in post-processing, the gap can narrow. Conversely, merely swapping in a larger open-weight model often does not solve layout and table problems.

    Evaluation Dimensions: Accuracy Is Only One Signal

    Use a fixed test set for offline evaluation and keep reproducible preprocessing scripts. At minimum, measure the following dimensions.

    Character and word accuracy. Use edit-distance-normalized accuracy and report it separately for printed text, handwriting, artistic text, and small font sizes. For Chinese, distinguish simplified and traditional characters, full-width and half-width forms, and punctuation. Do not hide systematic header or footer errors behind a whole-page average.

    Layout and reading order. Match predicted blocks with annotated blocks and check misses, false positives, and order inversions. Multi-column documents should be a separate subset. Order errors appear as spliced answers in retrieval and can be more damaging than isolated character mistakes.

    Structural fidelity. For tables, check cell text, row and column counts, and merged cells. For formulas, check the symbol sequence or visual consistency after rendering. For headings, check whether the outline can be reconstructed. Plain-text metrics cannot replace these checks.

    Hallucination and rejection. For generative decoding, measure the proportion of text that is written but absent from the source and whether low-confidence regions are flagged. In business settings, it is better to reject a region and send it to human review than to silently rewrite an amount, dosage, or legal clause number.

    Efficiency and stability. Record per-page latency, end-to-end time for 100-page documents, queuing time under memory or concurrency limits, and retry rate. A claim of “seconds per long document” is meaningful only under fixed hardware, resolution, and concurrency; slogans without those conditions are not metrics.

    Cost and compliance. Estimate per-page or per-call cost, and examine whether data leaves the environment, whether logs contain source text, and whether the model can run inside a private network. Classified documents often cannot be sent to external APIs merely to improve a benchmark score.

    Publish the sample composition: what share is digital PDF, scanned PDF, photographed, or screenshot; what share is Chinese, English, mixed-language, or table-heavy. Changing the distribution can reverse the ranking. A single “overall first place” without subset reporting has limited reference value.

    Decoding Paradigms: Classification, Generation, and Iterative Reconstruction

    Understanding the decoding paradigm helps explain error types rather than chasing buzzwords.

    Detection-plus-recognition pipelines remain the base for many production systems. Detection boxes are stable, coordinates are interpretable, and local re-recognition is cheap, making this approach suitable for receipts and fixed layouts. The weakness is that if a box is wrong, later stages cannot recover, and multi-column order requires a separate sorting model.

    Autoregressive or one-shot document generation treats the page as a conditional generation problem and outputs marked text or structured fragments. It offers strong context and tolerance for touching characters. The drawbacks are that long pages may drop lines, repeat content, or invent sentences, and token-by-token decoding latency grows with page length. Engineering mitigations include pagination, sliding windows, and length constraints.

    Iterative or diffusion-based reconstruction views decoding as gradually recovering the page’s symbol layout from a noise or mask state. It emphasizes global consistency and can correct early local errors later, making it suitable for extremely complex layouts. The cost is that sampling steps, scheduling, and stopping conditions all affect speed and stability; poor tuning can lead to over-smoothing or repeated blocks. This approach is suitable as a research path or an enhancement route for hard pages. Before launch, set a latency budget at fixed steps; do not look only at the best visualizations from papers.

    Whatever the decoder, post-processing is mandatory: normalize punctuation and whitespace, merge line-break words, reconstruct reading order from coordinates, and validate key fields with dictionaries or business rules. The model proposes; the rules guard.

    Engineering Pipeline: From Files to Retrievable Text

    A maintainable parsing path can be split into stages so that any model can be replaced.

    Ingestion and normalization. Accept PDFs, images, or multi-page packages and determine whether a PDF has selectable text. Prefer text-layer extraction for digital PDFs; fall back to visual OCR only when text is missing, garbled, or scanned. This can significantly reduce cost and character errors. After page splitting, use a consistent rendering resolution: too high slows recognition, too low loses small text. Choose the inflection point on the validation set rather than blindly using maximum DPI.

    Image preprocessing. Deskewing, cropping, contrast adjustment, and moiré handling should be switchable by document type. Preprocessing must be replayable; otherwise online results cannot be aligned with offline evaluation.

    Parsing and structure. The output should include at least page number, block ID, block type, text, confidence, and bounding box. Text in tables and images should be separate blocks rather than mixed into body text. Mark tables that continue across pages so they are not split into two unrelated tables.

    Quality control. Send low-confidence pages, failed amount-field checks, and page-count or table-of-contents mismatches to a review queue. Set sampling rates according to business risk, not model marketing claims.

    Downstream integration. Chunk content according to headings and paragraph boundaries before building vector or keyword indexes. OCR coordinates can be written back into citations so users can jump to the source page. Store originals and parse results in object storage, use a message queue for asynchronous parsing, run elastic inference on a container platform, and monitor failure rate, latency percentiles, and queue backlog. These are general cloud capabilities and do not need to be tied to a proprietary API.

    If a team wants to reduce the maintenance burden of self-hosted inference and layout components, it can use a managed document-intelligence or OCR service such as Baidu AI Cloud’s offering as a cloud fallback after text-layer extraction fails, or as the primary path for public documents, where access and data-location requirements allow. Sensitive documents and layouts requiring deep customization should still remain in a private network. Compare candidates only on the validation set across structural fidelity, rejection strategy, data egress scope, and unit cost; do not change acceptance criteria for a brand narrative.

    A neutral result structure can remain independent of any specific vendor:

    1. {
    2. "page": 3,
    3. "blocks": [
    4. {
    5. "id": "p3-b2",
    6. "type": "paragraph",
    7. "text": "...",
    8. "bbox": [72, 140, 540, 210],
    9. "score": 0.97
    10. }
    11. ]
    12. }

    Callers depend only on page, block, type, text, bounding box, and confidence. When the underlying engine changes from a pipeline to a generative or iterative model, retrieval and review logic should not need to be rewritten.

    How to Compare Without Being Misled by Demos

    Lock the comparison to the same test set, the same rendering parameters, the same output schema, and the same ceiling for post-processing. Do not compare screenshots from each vendor’s curated samples. Test long documents as complete documents: missing pages, page order, cross-page tables, and total time matter more than a polished single-page demo.

    Grade errors by business consequence. A one-character error in a contract number, amount, or date has higher priority than an error in a function word. Report both key-field accuracy and full-text accuracy.

    Look at failure modes, not only means. One system may be near perfect on clean digital documents but collapse on stamps and handwritten annotations; another may have a slightly lower average but rarely changes numbers. For finance and government use cases, the latter is often more usable.

    Generative models need an explicit “do not add facts” test: cover half a line and check whether the model rejects the region or fabricates content. Iterative decoders should report the curve between sampling steps and latency, rather than showing only the best output at high step counts.

    Write conclusions as applicability boundaries: for fixed layouts, high concurrency, and coordinate backlinks, prefer a detection-and-recognition pipeline plus rules; for very irregular layouts with many columns and mixed image-text content, prefer document-level generation or iterative reconstruction with mandatory confidence checks and human sampling; when data cannot leave the environment, self-hosting or private-network deployment takes priority over any external high-scoring service. No decoding paradigm is universally superior for all documents.

    Common Pitfalls in Production

    • Optimizing recognition without optimizing chunking, which hurts knowledge-base recall.
    • Testing only Chinese printed text, then failing on English tables, vertical text, and other layouts.
    • Not retaining coordinates, so the system cannot prove which page a citation came from.
    • Not distinguishing the electronic text layer from the scanned layer, then re-OCR-ing correct text, which is both slower and introduces errors.
    • Lacking a versioned test set, so each model change is judged by feel.
    • Treating the 100-page latency from a demo as capacity planning; concurrency causes timeouts after launch.

    A safer setup is to build a gold set by document type; split metrics across key fields, layout order, tables, and hallucination; keep pipeline modules replaceable; short-circuit digital documents; reject low-confidence output; include page numbers and boxes in parse results; and monitor failure rates rather than a one-time demo. With this approach, teams can use the same acceptance language to decide whether to switch the underlying engine—traditional recognition, generative decoding, or iterative reconstruction—instead of being led by a single leaderboard.