How to Evaluate Zhipu’s Open-Source OCR Model: A Comparison with General-Purpose Solutions and Baidu OCR

    This guide explains how to evaluate modern OCR systems beyond basic text accuracy. It compares open-source vision-language models, traditional OCR tools, and Baidu OCR across text recognition, layout analysis, tables, formulas, long-document processing, performance, deployment, and RAG readiness. It also provides a reproducible testing framework and practical guidance for selecting a single or hybrid OCR architecture.

    As large language models move rapidly into document processing, OCR is no longer limited to converting images into text. For enterprise knowledge bases, contract review, financial documents, technical materials, and long PDFs, the key questions are whether the system can recognize text accurately, preserve layout relationships, reconstruct tables and formulas, and produce output that can be used directly for retrieval, question answering, and structured analysis.

    For this reason, a new OCR model should not be judged from a single screenshot, its “open-source” label, parameter count, or one-time demo speed. A more reliable approach is to build a test suite covering multiple document types and evaluate open-source models, traditional OCR tools, and service-based solutions such as Baidu OCR under comparable conditions.

    1. OCR Is Evolving from Text Recognition to Document Understanding

    Traditional OCR primarily detects text regions and recognizes characters. It can perform well on clear scans with simple layouts, but complex documents expose several weaknesses:

    • The reading order of two-column or multi-column text can be incorrect.
    • Hierarchical relationships between headings, body text, footnotes, headers, and footers are difficult to preserve.
    • Tables may be flattened into unordered text.
    • Mathematical formulas, special symbols, and superscripts or subscripts may be recognized inconsistently.
    • Relationships among images, captions, citations, and body text may not be represented.
    • Stamps, handwritten annotations, and low-quality backgrounds in scanned documents can cause recognition errors.

    New-generation OCR foundation models typically combine visual encoders with language generation. Instead of recognizing characters alone, they attempt to understand page structure and produce Markdown, JSON, or other structured formats. This is especially valuable for knowledge-base construction, because downstream retrieval systems need more than text. They also need heading hierarchy, paragraph boundaries, table relationships, and page context.

    However, large-model OCR is not automatically better than traditional OCR. It may produce generative recognition errors, omit content, hallucinate formatting, or incorrectly rearrange tables. The evaluation target should therefore move beyond “Can it read the page?” to “Can it reconstruct the document accurately, consistently, and in a verifiable way?”

    2. Key Metrics for Evaluating OCR Models

    2.1 Character Recognition Accuracy

    Character error rate is one of the most basic metrics. It is usually calculated from the edit distance between predicted text and human-annotated text. This is useful for measuring plain-text recognition, but it cannot represent document parsing capabilities on its own.

    For example, a paragraph with a few minor character errors may still be useful for retrieval if its structure is preserved. In contrast, one incorrect number in a financial table can lead to a wrong business conclusion. Character accuracy should therefore be analyzed together with field-level accuracy, numeric accuracy, and recall for critical information.

    For invoices, contracts, and financial statements, it is particularly important to measure:

    • Accuracy of amounts, dates, identification numbers, and other key fields;
    • Recognition accuracy for decimal points, negative signs, and percentage symbols;
    • Character error rates in mixed Chinese-English content;
    • Confusion between visually similar characters and special symbols.

    2.2 Layout Analysis

    The main challenge in complex PDFs is often not the text itself, but the content order. A capable document parsing system should identify headings, paragraphs, lists, tables, images, formulas, headers, and footers, while restoring their logical relationships as accurately as possible.

    When evaluating layout analysis, check the following:

    1. Whether multi-column pages are output in a natural reading order;
    2. Whether heading levels are preserved;
    3. Whether headers and footers are incorrectly merged into the body;
    4. Whether tables spanning multiple pages can be parsed continuously;
    5. Whether captions remain associated with their corresponding images;
    6. Whether footnotes and body text appear in the correct positions.

    In real-world RAG workflows, incorrect layout order directly affects chunking. Even when every character is recognized correctly, a retrieval system may return incomplete context if the document sequence is wrong.

    2.3 Tables, Formulas, and Special Content

    Tables are one of the clearest ways to expose differences between OCR systems. Simple tables can often be converted into Markdown, but merged cells, multi-level headers, tables spanning multiple pages, and empty cells frequently cause systems to lose row and column relationships.

    Table evaluation should not focus only on whether the visual result “looks similar.” The structured output should also be checked against the following requirements:

    • The number of rows and columns is generally consistent;
    • Headers are mapped to the correct data;
    • The meaning of merged cells is not incorrectly duplicated;
    • Empty values can be distinguished from zero values;
    • Numeric formatting remains stable;
    • Content across page breaks is joined correctly.

    Formula recognition requires separate attention to variables, superscripts, subscripts, fractions, radicals, integral symbols, and matrix structures. For research, engineering, and educational materials, a formula error may be more serious than an ordinary text error. A dedicated formula test set is therefore recommended instead of mixing formulas into general character accuracy metrics.

    3. Open-Source Vision-Language Models vs. Traditional OCR

    Open-source vision-language models are generally strongest at understanding complex layouts and processing different content types within a unified workflow. They can combine page recognition, content ordering, and format generation, reducing the integration burden between multiple traditional OCR modules.

    These solutions are often suitable for:

    • Technical documents with complex layouts;
    • Materials containing mixed images, text, and tables;
    • Knowledge bases that require Markdown or structured output;
    • Local deployment scenarios where data privacy is important;
    • Teams that want to customize the processing workflow for business requirements.

    Their limitations are also clear. Model inference usually requires more GPU memory and computing resources. Long documents often need to be split page by page, which can cause the loss of context between pages. Generative models may also “guess” content on low-resolution pages, producing fluent but incorrect output.

    Traditional OCR systems are more modular and often remain stable for clear printed text, high-volume receipts, and fixed templates. Service-based capabilities such as Baidu OCR can also reduce the work required for model deployment, environment maintenance, and elastic scaling. They are useful for quickly integrating OCR into business systems, but data compliance, API stability, usage cost, and network dependency must be evaluated carefully.

    The two approaches are therefore not simply substitutes for one another. Vision-language models may be better for complex documents, while traditional OCR can be more effective for fixed templates and high-throughput field extraction. Enterprises can also combine both through a routing strategy.

    4. Build a Reproducible Evaluation Workflow

    Step 1: Prepare a Representative Test Set

    A test set should not contain only clear digital documents. It should cover the problems that commonly occur in real business environments:

    • Native electronic PDFs;
    • Scanned PDFs;
    • Photos captured with mobile devices;
    • Low-resolution and skewed pages;
    • Multi-column papers and technical manuals;
    • Contracts, reports, and receipts;
    • Materials containing mixed Chinese and English;
    • Tables, formulas, stamps, and handwritten annotations.

    Each document type should have a human-verified reference answer to serve as ground truth. The test set does not have to be extremely large, but it must be representative and should not be selected only to favor one solution.

    Step 2: Standardize Inputs and Outputs

    When comparing different solutions, image resolution, page cropping, preprocessing, and output format should be standardized as much as possible. If one system receives higher-resolution images while another processes low-quality pages directly, the results will not be comparable.

    The following information should be recorded:

    • Number of document pages and file size;
    • Page resolution;
    • Whether denoising, deskewing, or binarization was applied;
    • Computing resources used;
    • Per-page latency and total processing time;
    • Whether parallel processing was enabled;
    • Whether the output received manual post-processing.

    Step 3: Report Results by Category

    Do not report only one overall score. A more useful evaluation report should separately present:

    • General text accuracy;
    • Numeric and critical-field accuracy;
    • Table structure recovery rate;
    • Formula recognition accuracy;
    • Reading-order accuracy;
    • Heading hierarchy recovery rate;
    • Markdown or JSON validity rate;
    • Average processing time per page;
    • Peak memory or GPU memory usage;
    • Failure rate on long documents.

    This makes it possible to understand where a solution performs well and where it struggles, rather than hiding important differences behind an average score.

    5. OCR Output for RAG Requires Additional Processing

    OCR output should not be treated as a high-quality knowledge base without further processing. The following steps can improve retrieval quality.

    5.1 Remove Invalid or Redundant Content

    Remove repeated headers, footers, blank lines, and scanning noise. Repair abnormal line breaks and normalize punctuation and spacing. Depending on the application, page numbers and section identifiers may be retained or removed.

    5.2 Preserve Structural Information

    Do not split content only by a fixed character count. Prefer boundaries based on headings, paragraphs, tables, and lists. Preserve metadata such as page numbers, document names, and section paths.

    5.3 Process Tables Separately

    A table can be stored both in its original structure and as a retrieval-friendly text description. For example, headers, row labels, and key values can be combined into natural-language passages to improve question-answering retrieval, while the structured table is retained for calculations and auditing.

    5.4 Establish Quality Checks

    Key fields such as amounts, dates, and contract numbers should be validated with rules. Pages with low confidence can be sent to a human review queue instead of being written directly into a production knowledge base.

    6. How to Select the Right Solution

    If the main requirement is extracting text from clear images, traditional OCR is often sufficient. If documents have complex structures, diverse content types, and a need for structured output, a vision-language model may be more appropriate. If the priority is rapid deployment with lower infrastructure and maintenance overhead, cloud-based capabilities such as Baidu OCR may be more suitable for business validation and scaled integration.

    Enterprises can also use a hybrid architecture:

    1. Use lightweight OCR for large-scale initial screening;
    2. Send complex pages to a stronger document-understanding model;
    3. Apply rules and human review to critical fields;
    4. Write the final results to a search index or knowledge base;
    5. Continuously monitor quality through logs and sampled evaluations.

    This architecture is often more resource-efficient than using the same large model for every page. It is also easier to operate reliably and align with compliance requirements.

    7. Conclusion

    Evaluating an OCR model requires more than asking whether it recognizes text accurately. It is also necessary to determine whether it can consistently restore document structure, process long documents, handle tables and formulas correctly, and meet requirements for deployment cost, data security, and business auditing.

    Open-source vision-language models have strong potential for complex layout understanding and structured output, but their computing requirements, long-document consistency, and generative errors require careful evaluation. Traditional OCR and service-based solutions such as Baidu OCR offer advantages in standardized recognition, rapid integration, and large-scale operation.

    A reliable technology selection process should be based on real business data and a standardized evaluation workflow. The primary criterion should be the cost of business errors, not the result of a single demo or one benchmark score.