Document OCR Model Evaluation and Decoding Paradigms: A Practical Guide

    This guide explains how to evaluate document OCR systems beyond raw character accuracy, covering text recognition quality, layout reconstruction, table and formula handling, latency, stability, robustness, and operational needs. It compares autoregressive decoding with diffusion-based decoding, outlines a reproducible evaluation workflow with layered test sets, unified output formats, and blind review, and provides practical guidance for RAG-oriented parsing, production deployment, and selecting the right decoding approach for business requirements.

    1. Why Document OCR Evaluation Is No Longer Just Character Accuracy

    In scenarios such as document digitization, knowledge base construction, intelligent question answering, and workflow automation, OCR is no longer limited to recognizing characters in images. Real-world documents often contain complex layouts, multi-column formats, tables, formulas, headers and footers, stamps, handwritten annotations, skewed scans, low-resolution images, and paragraphs that span multiple pages. If evaluation only compares character-level or word-level recognition accuracy, it can miss the most important parts of a document parsing pipeline: layout understanding, reading order restoration, table structure reconstruction, machine-readable formula representation, and structured output for downstream applications.

    For this reason, a document OCR model or parsing framework should be evaluated from the perspective of end-to-end document understanding, rather than through a single recognition metric in isolation. A model may achieve high accuracy on ordinary printed text while still performing poorly on multi-column papers, financial reports, contract clauses, scanned invoices, or complex tables. Conversely, some solutions may not have the highest character-level accuracy, but may be more suitable for real business scenarios because of better layout preservation, stronger structured output, higher throughput, or more stable engineering behavior.

    When selecting a solution, it is useful to divide the evaluation goals into three layers:

    1. Basic recognition capability: whether the text is accurate and whether the system supports multiple languages, numbers, symbols, superscripts, subscripts, and similar elements.
    2. Document understanding capability: whether the system can correctly identify titles, paragraphs, lists, tables, images, formulas, headers, footers, and other document elements.
    3. Business usability capability: whether the system can reliably produce structured results that can be used for retrieval, storage, question answering, and information extraction, while maintaining acceptable latency, cost, and operational complexity.

    2. Core Metrics for Evaluating Document OCR Systems

    2.1 Text Recognition Quality

    The most basic metrics include character accuracy, word accuracy, and edit distance. For Chinese content, additional details may matter, such as punctuation, spacing, full-width and half-width characters, traditional and simplified forms, and number formats. In practice, a misrecognized number in a financial report, a misrecognized amount in a contract, or a misrecognized date on an invoice is often more damaging than a few incorrectly recognized Chinese characters.

    For long documents, average accuracy alone is not enough. It is also important to check whether quality degrades noticeably in later sections. Some models perform well on short sentences but produce repeated content, missing lines, disordered output, or inconsistent context when processing full pages, cross-page paragraphs, or long continuous text.

    2.2 Layout Structure Reconstruction

    The biggest difference between document OCR and ordinary image text recognition is structure. A contract, paper, report, or manual is not merely a collection of characters; it also contains clear semantic hierarchy. Evaluation should focus on the following points:

    • Whether heading hierarchy is preserved.
    • Whether paragraphs are incorrectly split.
    • Whether list items maintain their original relationships.
    • Whether headers and footers are correctly removed or marked.
    • Whether multi-column layouts are output in the correct reading order.
    • Whether captions and figure descriptions are mixed with body text.
    • Whether tables are broken into meaningless text fragments.

    A good document parsing result should be close to human reading behavior and should be convenient for downstream chunking, indexing, retrieval, and question answering.

    2.3 Table and Formula Processing

    Tables and formulas are especially difficult areas in OCR evaluation. Plain text recognition cannot cover cell relationships, merged cells, header hierarchy, or cross-row and cross-column structures in tables. Formulas involve mathematical symbols, superscripts and subscripts, fractions, matrices, Greek letters, operators, and other complex expressions.

    When evaluating tables, it is not enough to check whether text inside cells is recognized. The table structure itself must also be reconstructed correctly. For example, evaluation should check whether headers are mapped to the correct columns, whether all columns are complete, whether numbers fall into the correct cells, and whether complex tables are incorrectly split into multiple tables.

    When evaluating formulas, the focus should be on whether the result can be used by downstream systems. If a formula is only recognized as plain text and is not converted into a structured mathematical expression, its value in knowledge question answering, computational reasoning, and paper retrieval may be greatly reduced.

    2.4 Latency, Throughput, and Stability

    Even the best model will struggle with large-scale deployment if engineering performance is insufficient. Evaluation should consider:

    • Average latency per page.
    • P95 and P99 latency during peak periods.
    • Concurrency handling capability.
    • Whether long-document parsing time grows linearly.
    • Whether large images or multi-page documents are likely to time out.
    • Whether batch tasks support queue-based processing.
    • Whether failed requests can be retried, traced, and recovered.

    In production environments, long-tail requests are often more important than average metrics. A system with low average latency but frequent timeouts on complex documents may provide a poor user experience.

    2.5 Robustness and Exception Handling

    Real documents vary greatly in quality, so an evaluation set should not contain only clean printed text. It should cover:

    • Documents photographed with mobile phones.
    • Low-resolution scans.
    • Images with skew, curvature, shadows, or reflections.
    • Multi-column newspapers or papers.
    • Handwritten annotations.
    • Text covered by stamps.
    • Invoices, receipts, and forms.
    • Tables spanning multiple pages.
    • Mixed image-and-text materials.
    • Small fonts and low-density layouts.

    A robust system usually forms a complete loop around preprocessing, layout analysis, confidence evaluation, and human review strategies.


    3. Autoregressive Decoding vs. Diffusion Decoding

    Inside a document OCR model, the recognition and decoding method can significantly affect output quality, speed, and stability. A common approach is autoregressive decoding. In this approach, the model generates text step by step based on visual features, outputs one segment at a time, and then predicts the next part based on the existing output. This is similar to human reading, where characters or words are read sequentially. It benefits from a strong language model prior and can use context to correct recognition results.

    However, autoregressive decoding also has inherent limitations. First, it is sequential. When generating long text, later content must wait for earlier content to be produced, making it difficult to fully use parallel computing resources. Second, errors can accumulate. If an earlier region is recognized incorrectly, later generation may continue around the wrong content, causing errors to be amplified in long documents. Third, it is not always friendly to complex structures. Tables, formulas, and multi-column layouts may require additional constraints if they are generated as linear text; otherwise, spatial relationships may be lost.

    Diffusion decoding offers a different approach. Instead of generating text token by token from beginning to end, it starts from noise or candidate states and gradually approaches the target text and layout through multiple denoising steps. If document understanding is viewed as a kind of “inverse rendering”—inferring text content, layout structure, and style parameters from the final image—then the diffusion process can be well suited to this kind of task, which involves multiple variables, parallel updates, and gradual convergence.

    The potential advantages of diffusion decoding include:

    1. Parallel potential: Multiple text segments or structural regions can be updated at the same time instead of waiting in strict sequence.
    2. Long-document stability: Combining global noise states with local constraints may reduce error accumulation in linear generation.
    3. Structural representation: The system can generate not only text but also structural information such as layout, reading order, and region type.
    4. Sampling controllability: Denoising steps, confidence thresholds, and local correction strategies can be used as engineering tuning mechanisms.
    5. Complex layout adaptation: Diffusion-based methods may be better suited to expressing spatial relationships in multi-column pages, tables, and mixed image-and-text layouts.

    That said, diffusion decoding is not automatically better. Its performance depends heavily on training objectives, noise scheduling, conditional encoding, sampling steps, and inference strategies. If the number of steps is too low, the structure may be incomplete or unstable. If the number of steps is too high, latency and compute cost increase. In engineering practice, diffusion models usually need to be combined with distillation, quantization, caching, batch inference, and routing strategies to balance quality and performance.


    4. How to Design a Reproducible OCR Evaluation Process

    To avoid drawing conclusions from isolated cases, it is recommended to use a layered test set, multiple dimensions of metrics, and blind human review.

    4.1 Build a Layered Test Set

    The test set should be organized by document type, for example:

    • Ordinary printed documents.
    • Scanned books or papers.
    • Financial reports.
    • Contract clauses.
    • Invoices and receipts.
    • Technical manuals.
    • Documents with handwritten annotations.
    • Mixed image-and-text materials.
    • Multi-column layouts.
    • Low-quality images.
    • Multilingual mixed documents.

    Each category should have clear human-annotated ground truth, including text content, paragraph structure, table structure, and reading order.

    4.2 Define a Unified Output Specification

    Different models may produce different output formats. For fair comparison, a unified evaluation protocol is needed. For example, all results can be converted into a structured intermediate format containing fields such as page number, region type, coordinates, text, table structure, formula representation, and confidence.

    An illustrative structure is shown below:

    1. {
    2. "page": 1,
    3. "blocks": [
    4. {
    5. "type": "title",
    6. "text": "Document title",
    7. "bbox": [100, 80, 900, 140]
    8. },
    9. {
    10. "type": "paragraph",
    11. "text": "Paragraph content",
    12. "bbox": [100, 160, 900, 260]
    13. },
    14. {
    15. "type": "table",
    16. "rows": 4,
    17. "cols": 3,
    18. "content": [
    19. ["Field A", "Field B", "Field C"],
    20. ["Value 1", "Value 2", "Value 3"]
    21. ]
    22. }
    23. ]
    24. }

    This avoids incomparable results caused by format differences across systems.

    4.3 Example Metric Calculation

    The evaluation logic can be expressed in pseudocode:

    1. def evaluate_model(model, dataset):
    2. results = []
    3. for doc in dataset:
    4. output = model.parse(doc.image)
    5. score = {
    6. "text_accuracy": calc_text_score(output, doc.gold_text),
    7. "layout_score": calc_layout_score(output, doc.gold_layout),
    8. "table_score": calc_table_score(output, doc.gold_tables),
    9. "formula_score": calc_formula_score(output, doc.gold_formulas),
    10. "latency": measure_latency(model, doc.image)
    11. }
    12. results.append(score)
    13. return aggregate(results)

    Text metrics may include character accuracy and edit distance. Layout metrics may include region detection and reading order. Table metrics may focus on structural reconstruction. Formula metrics may check mathematical expression consistency. Engineering metrics may include latency and success rate.

    4.4 Introduce Human Blind Review

    Automatic metrics cannot fully cover readability and business applicability. For important documents, human blind review is recommended. Evaluators receive only the input image and model output, without knowing the model source, and score the results according to a unified standard. Scoring dimensions may include:

    • Whether the text is readable.
    • Whether paragraphs are coherent.
    • Whether the reading order matches human expectations.
    • Whether tables can be reconstructed.
    • Whether formulas are understandable.
    • Whether the result can be stored directly.
    • Whether extensive post-processing is required.

    Blind review helps reduce subjective bias and is especially useful during final selection.


    5. Document Parsing Practices for RAG Scenarios

    In RAG scenarios such as knowledge base question answering, enterprise search, and intelligent document analysis, document parsing quality directly affects downstream performance. If the parsing stage loses heading hierarchy, table relationships, or paragraph boundaries, later retrieval and generation can hardly compensate for it. Therefore, the goal of a document parsing system should not be merely “recognition.” It should be to produce results that can be understood, chunked, and cited.

    5.1 Preserve Semantic Hierarchy

    The output should preserve structures such as titles, sections, lists, and paragraphs. This allows chunking to be based on semantic blocks rather than fixed character counts. Structured chunking is usually better for retrieval than fixed-length splitting.

    For example, if a title block maintains a parent-child relationship with its body paragraphs, retrieval hits can provide more accurate context. If a table preserves its row and column structure, it is easier for a question answering system to cite the correct information.

    5.2 Remove Low-Value Noise

    Headers, footers, page numbers, watermarks, and binding-margin text can pollute a knowledge base if they are not handled. The parsing system should support rule-based filtering, position-based filtering, duplicate detection, or model-based judgment to remove such content from the main text flow or mark it as low-priority fields.

    5.3 Apply Specialized Processing to Tables and Formulas

    Tables should be output as structured objects rather than plain text. Formulas should be converted into parseable mathematical expressions where possible, or at least retained as dual representations of both image and text to support later computation and display.

    5.4 Establish Confidence and Review Mechanisms

    For low-confidence results, a human review queue should be used. Risk regions can be identified through character-level confidence, layout anomaly detection, table structure anomaly detection, or model uncertainty estimation. This helps prevent incorrect content from entering the knowledge base.


    6. Engineering Deployment: From Offline Batch Processing to Online Services

    Document OCR is often not a single model call but a complete pipeline. A typical flow includes image intake, preprocessing, layout analysis, text recognition, structure reconstruction, quality checking, result storage, task callbacks, and monitoring or alerting.

    6.1 Preprocessing

    Preprocessing can significantly affect final results. Common operations include denoising, enhancement, skew correction, perspective transformation, binarization, resolution normalization, and illumination compensation. For low-quality scans, reasonable preprocessing is often more effective than simply switching to a larger model.

    6.2 Routing Strategy

    Different documents vary greatly in difficulty, so tiered routing can be used:

    • Simple and clear documents can go through a lightweight parsing pipeline.
    • Complex layouts can go through a structure-enhanced pipeline.
    • Documents with high quality requirements can enter a high-precision model or human review.
    • Batch tasks can use asynchronous queues to control resource peaks.
    • Long documents can be split by page to reduce the complexity of individual tasks.

    This strategy is usually more stable and more resource-efficient than sending all requests through the same expensive pipeline.

    6.3 Asynchronization and Batch Processing

    For batch document processing, tasks can be placed in a message queue and consumed asynchronously by multiple processing instances. Original files and parsing results can be stored in object storage, while processing status and logs can be written to a logging service. Failed tasks can enter a retry queue. This improves throughput and makes failure recovery easier.

    6.4 Monitoring and Alerting

    Production systems need continuous monitoring of metrics such as:

    • Total number of requests.
    • Number of successful and failed requests.
    • Average latency and tail latency.
    • Processing time per page.
    • Failure rate for complex documents.
    • Proportion of documents sent to human review.
    • Storage growth trend.
    • Quality fluctuations after model version changes.

    If the deployment uses Baidu AI Cloud, object storage can be used to store original files and parsing results, container platforms can host parsing services, message queues can decouple task scheduling, and logging and monitoring can be used to track pipeline status. Such general cloud capabilities can help turn an OCR parsing system from a standalone script into an operable, observable, and scalable service.

    6.5 Version Management and Quality Regression

    Model upgrades or changes to prompts and post-processing rules may affect final results. Therefore, before each change, a fixed evaluation set should be run to check whether key metrics have degraded. For core business documents, historical samples should also be retained for regression testing to avoid a situation where offline metrics improve while online experience declines.


    7. How to Decide Whether Diffusion Decoding Fits Your Business

    Diffusion decoding provides a new technical path for document OCR, but it does not mean that every scenario should adopt it first. The decision can be based on the following considerations.

    First, consider document complexity. If business documents are mainly simple printed text, a traditional pipeline may already be stable and cheaper to deploy. If the business heavily involves complex tables, formulas, multi-column papers, or mixed image-and-text layouts, diffusion decoding or a more structure-oriented parsing approach may provide greater value.

    Second, consider latency requirements. Real-time interfaces are sensitive to latency, so denoising steps, model size, and hardware conditions need to be evaluated. Offline batch processing has looser latency requirements and may be more suitable for high-precision structured parsing.

    Third, consider structured output requirements. If results are only used for simple retrieval, text quality may be sufficient. If results are used for knowledge base question answering, field extraction, report generation, or formula reasoning, layout, table, and formula structure must be carefully evaluated.

    Fourth, consider operational complexity. Diffusion models usually require more complex inference scheduling, resource management, and quality monitoring. If the team lacks the corresponding engineering capability, it is better to first complete infrastructure such as storage, queues, containers, logging, and monitoring through general cloud services.

    Fifth, consider business tolerance for errors. High-tolerance scenarios may accept automatic parsing followed by human review. Low-tolerance scenarios require stronger confidence mechanisms and traceable logs.


    8. Conclusion

    Document OCR evaluation is shifting from comparing the character recognition accuracy of a single model to comparing the overall capability of a complete parsing pipeline. For developers, the most important thing is not to chase a leaderboard ranking, but to build an evaluation system that fits the business scenario. Such a system should consider text recognition quality, layout structure, tables and formulas, long-document stability, latency and throughput, and operational cost.

    Autoregressive decoding is mature and stable, making it suitable for many traditional recognition tasks. Diffusion decoding demonstrates the potential of parallel denoising, global structure modeling, and inverse-rendering-style understanding. In the future, document parsing systems are unlikely to rely on a single model alone. Instead, they will likely be composed of preprocessing, layout analysis, recognition decoding, structure reconstruction, quality monitoring, and human review. Only by combining model capability with engineering capability can we build stable, reliable, and scalable document intelligence infrastructure.