High-Resolution Document Parsing: Two-Stage Inference and Engineering Practices
This article examines why high-resolution document parsing is harder than general image recognition, and how a two-stage, coarse-to-fine design that decouples global layout analysis from local content recognition addresses those challenges. It compares traditional pipeline methods with end-to-end vision-language models, then details the engineering of each stage: how low-resolution layout detection produces a structured page skeleton with regions, coordinates, and reading order, and how local recognition crops the original high-resolution image to preserve fine detail. It also covers using a single unified multimodal model for both stages, the data engineering, training strategies, and hallucination control that determine real-world results, along with inference optimization, degradation strategies, and an evaluation framework that goes beyond character accuracy. Readers will come away with a practical blueprint for building a stable, scalable, and operable high-resolution document parsing system.
High-Resolution Document Parsing: Two-Stage Inference and Engineering Practices
1. Why High-Resolution Document Parsing Is Harder
In general image recognition, a picture usually answers a fairly focused question such as “what is in this image” or “what does the text say.” Document parsing deals with a more demanding class of data: high-resolution scans, receipts, contracts, papers, tables, invoices, report pages, and web screenshots. These images tend to have larger pixel dimensions, higher text density, more complex layout structures, and stricter field-alignment requirements.
Compared with simpler scenarios, high-resolution document parsing faces at least three core challenges.
First, there is a tension between resolution and computational cost. Text in documents is often very small. If the whole image is downsampled, fine detail is lost quickly and recognition errors follow. But encoding the full image at high resolution sharply increases visual compute, hurting inference throughput and deployment cost.
Second, there is a tension between dense text and long sequences. A document page typically contains many text lines, table cells, headers, footers, and multi-column content. If a model processes an entire page at once, it is prone to missing text, duplicate recognition, or semantic drift over long contexts. These errors are amplified further when the scan is low quality, blurry, shadowed, or has a complex background.
Third, recognition accuracy depends on layout understanding. Document parsing is not merely character recognition. It also requires determining reading order, identifying region types, separating body text from noise, reconstructing table structure, and handling complex cases such as cross-column, cross-page, and formula content. If layout analysis is unreliable, even a strong local recognition model will produce incorrect structured output.
So the core of high-resolution document parsing is not only recognizing text more accurately, but solving four layers of problems at once: seeing clearly, locating accurately, reading correctly, and being deployable.
2. Traditional Pipelines vs. End-to-End Approaches
2.1 Traditional pipeline methods: modular and clear, but prone to error accumulation
Early document parsing usually relied on modular pipelines: first text detection, then text recognition, followed by layout reconstruction, table recognition, and reading-order recovery. The advantage is a clear structure with well-defined module responsibilities, which makes individual components easy to swap and debug.
But the drawbacks are equally clear. First, errors propagate stage by stage. If text detection misses a small region, downstream recognition cannot recover it. If reading order is judged incorrectly, the final structured result loses its business value even when the text itself is correct. Second, complex layouts are hard to cover with fixed rules. Cases such as multi-column newspapers, nested tables, cross-page tables, stamp occlusion, and handwritten annotations often expose weak generalization in traditional methods.
2.2 End-to-end vision-language model methods: strong semantics, but pressure on compute and stability
End-to-end methods typically feed the document image directly into a vision-language model that generates structured text, coordinates, or semantic descriptions. Their strength is stronger semantic understanding: they can reason with context and reduce the fragmentation between multiple independent modules.
But in high-resolution document scenarios, end-to-end methods also face clear challenges. On one hand, feeding a full page at high resolution significantly increases the visual encoding burden. On the other hand, models may hallucinate on long documents or dense text, such as inventing non-existent fields, incorrectly merging text, or misjudging heading levels. Without strong constraints, end-to-end methods also tend to over-generalize local details, which hurts precise recognition.
This points to a more engineering-oriented idea: keep the multimodal understanding of end-to-end models while avoiding the efficiency pressure of high-resolution input; leverage the value of layout analysis while preventing layout errors from propagating downstream. This naturally leads to a two-stage inference design built on decoupling and coarse-to-fine processing.
3. Core Design: Decoupling Global Layout from Local Recognition
The central idea of the two-stage approach is to split document parsing into two relatively independent tasks:
- Global layout analysis: at lower resolution, quickly determine which regions a page contains, along with their types, coordinates, and approximate reading order.
- Local content recognition: based on the global layout, crop the corresponding regions from the original high-resolution image and recognize them more precisely within local windows.
The essence of this design is “find the structure first, then read the content.” It is not simply splitting a model into two modules, but decomposing a complex task reasonably so that different stages handle information at different granularities.
3.1 Why global layout can run at lower resolution
Global layout analysis focuses on region position and type rather than character-by-character recognition. Structural information such as titles, body text, tables, images, headers, footers, and page numbers can usually be determined effectively even at lower resolution. So the first stage does not need to process the full original high-resolution image, which substantially reduces visual encoding cost.
3.2 Why local recognition needs high resolution
The local recognition stage deals with concrete text, table cells, formulas, or field content, which often contain fine strokes, dense symbols, and complex textures. If a downsampled image is still used, key information may already be destroyed. So the second stage must crop regions from the original high-resolution input and recognize them within local windows to preserve detail.
3.3 How the two stages improve robustness
When local recognition struggles, the system can use the global layout as context, for example knowing whether a region is a table header, a footnote, or body text. When the global layout is uncertain, local recognition results can in turn verify the region type. In other words, the two stages are not strictly one-directional; in an engineering implementation they can form a local feedback loop, such as re-recognizing low-confidence regions, validating table structure, and reordering reading order.
4. Stage One: Engineering Global Layout Analysis
4.1 Task objective
The first stage’s main goal is not to recognize all the text, but to generate a structured skeleton of the document page. Common outputs include:
- Text regions: titles, paragraphs, list items, headers, footers, and so on;
- Table regions: overall table boundaries, headers, cells, or row/column structure;
- Graphic regions: images, stamps, logos, charts, and the like;
- Formula regions: inline formulas and standalone formulas;
- Reading order: the logical reading path between regions.
4.2 Input design
The global layout stage usually takes a reasonably scaled full-page image as input. The scaling strategy must balance two goals: retaining enough spatial structure information while controlling computation. In practice, the scaling ratio can be configured based on page size, the minimum target region size, and the inference latency budget.
4.3 Output design
Layout analysis is best output as a standardized structure rather than free text. This reduces downstream parsing and validation cost. A sample output looks like this:
{"page_size": {"width": 2400,"height": 3200},"regions": [{"id": 1,"type": "header","bbox": [120, 80, 2280, 220],"confidence": 0.96},{"id": 2,"type": "title","bbox": [200, 260, 2200, 420],"confidence": 0.97},{"id": 3,"type": "table","bbox": [160, 900, 2240, 1800],"confidence": 0.93}]}
This kind of structured output makes it easy to perform deduplication, sorting, merging, filtering, and re-recognition downstream.
4.4 Handling coordinates and reading order
One of the most error-prone parts of the layout stage is reading order. Common practices include:
- Sorting by a basic top-to-bottom, left-to-right rule;
- Adding column-level segmentation for multi-column documents, then sorting within each column;
- Preserving the overall reading logic of table regions to avoid cell-by-cell disorder;
- Detecting and isolating headers and footers to keep them out of the body text;
- Clipping and filtering abnormal coordinates to prevent out-of-bounds regions from affecting later recognition.
Reading order is not simple coordinate sorting; it is part of reconstructing the document’s logic. For rigorously structured documents such as contracts, papers, and financial reports, an ordering error directly leads to semantic errors.
5. Stage Two: Engineering Local Content Recognition
5.1 Region cropping and window design
The second stage crops the corresponding regions from the original high-resolution image based on the region coordinates from the first stage. Cropping must handle boundary expansion, occlusion preservation, and local context supplementation. For example, if a text line is cropped too tightly, the first and last characters may be lost; if cropped too loosely, neighboring regions introduce interference.
Local windows can use the following strategies:
- Crop text regions by line or paragraph;
- Crop table regions by cell, by row/column, or by overall structure;
- Crop formula regions separately to avoid mixing them with body text;
- Set overlapping windows for cross-region content, then merge the results;
- Recognize long text in blocks, then stitch the results together.
5.2 Different recognition strategies for different region types
Even if a single unified multimodal model is used, different region types are best handled with different task prompts or structured-output requirements. For example:
- Text regions: require output of the character sequence with coordinates;
- Table regions: require output of row/column structure, header relationships, and merged cells;
- Formula regions: require output of formula structure rather than a natural-language description;
- Stamp or signature regions: can output a type judgment, position, and quality confidence.
This approach of “one unified model, differentiated tasks” helps improve local recognition accuracy while keeping the architecture simple.
5.3 Handling low-confidence regions
For low-confidence regions, it is not advisable to output the final result directly. Engineering options include:
- Enlarging the local window and re-recognizing;
- Re-cropping at higher resolution;
- Performing secondary validation with context;
- Filtering out suspected noise regions;
- Enabling manual review or rule-based validation for critical fields.
For example, once confidence is insufficient for key fields such as amounts, dates, or IDs, a fallback should kick in through business rules or a secondary recognition mechanism.
6. How a Unified Multimodal Model Handles Both Stages
A two-stage architecture does not necessarily require two completely independent models. A common approach is to use a unified multimodal vision-language model as the base capability, and through task prompts, output formats, and training objectives, have it take on both layout analysis and local recognition.
6.1 Advantages of a unified model
- Shared visual understanding, reducing duplicated development;
- Easier unified deployment and maintenance;
- Task switching via prompts, improving engineering flexibility;
- Easier to introduce a unified evaluation and monitoring system.
6.2 Challenges of a unified model
- The layout stage and the recognition stage have different output-format requirements, so instructions must be designed carefully;
- Training data must cover multiple tasks: coordinate prediction, region classification, text recognition, and table reconstruction;
- Poorly designed task prompts can cause the model to confuse layout with recognition;
- For long documents, context length and hallucination control still need attention.
So a unified model is not simply gluing two tasks together. It requires high-quality instruction design, structured-output constraints, and multi-task training so the model can reliably exercise different capabilities at different stages.
7. Data Engineering: The Foundation That Determines Two-Stage Results
The two-stage approach places demands on data that are no lower than on the model architecture itself. Without high-quality data, even a good architecture is hard to deploy reliably.
7.1 Data coverage
Document parsing data should cover as many of the following types as possible:
- Ordinary body-text documents;
- High-density newspapers or periodicals;
- Tables, financial reports, invoices, and receipts;
- Multi-column papers and technical documents;
- Low-quality scans, skewed images, noisy images, and compression-distorted images;
- Documents with stamps, signatures, and handwritten annotations;
- Mixed pages containing formulas, flowcharts, and charts.
7.2 Annotation content
For two-stage models, annotation usually covers more than just “text content,” and should also include:
- Region bounding boxes;
- Region types;
- Text line or paragraph membership;
- Reading order;
- Table structure;
- Formula structure;
- Noise-region or irrelevant-region markings.
7.3 Synthetic data and hard-case augmentation
High-quality real data is often expensive and hard to cover across all scenarios, so synthetic data is very important in engineering. Common approaches include:
- Adding blur, noise, and compression distortion to clean documents;
- Applying random font, font-size, and line-spacing variations to text;
- Simulating merged cells, cross-page layout, and line wrapping in tables;
- Applying rotation, perspective transforms, and occlusion to pages;
- Manually annotating or rule-validating low-quality scans.
7.4 Hard-case mining
For a two-stage model to genuinely improve, it needs to continuously accumulate hard cases, such as:
- Small-font dense text;
- Long text auto-wrapped inside tables;
- Cross-page tables;
- Headers and footers merged with body text;
- Stamps covering key fields;
- Multi-column reading-order errors;
- Formulas mixed inline with body text.
These hard cases should go into a dedicated evaluation set used for regression testing and version iteration.
8. Training Strategy: Making the Model Reliably Output Structured Results
8.1 Multi-task training
The two-stage approach usually needs to optimize several task types at once, such as layout detection, region classification, coordinate regression, text recognition, and table-structure generation. During training, a joint multi-task loss can be used, or different objective weights can be applied at different stages.
Common design ideas include:
- Focus the layout stage on coordinate accuracy and region type;
- Focus the recognition stage on character accuracy and structural completeness;
- Add structural constraints for tables and formulas;
- Add negative-sample constraints for text-free regions to reduce hallucination.
8.2 Instruction and prompt design
Prompt design directly affects model behavior. Different tasks should use clear, stable, and parseable instructions. For example, the layout stage can require the model to output a structured list of regions, while the recognition stage can require output of text or coordinates in a specified format.
Design principles include:
- Keep instructions concise and clear, avoiding ambiguity;
- Fix the output format for easy parsing;
- Provide fallback explanations for abnormal input;
- Allow an uncertainty marker for low-confidence cases;
- Strengthen constraints on key fields.
8.3 Anti-hallucination mechanisms
In document parsing, hallucination is often more dangerous than missed recognition, because it produces plausible-looking but incorrect structured results. Common controls include:
- Adding negative samples for empty and low-information regions;
- Setting thresholds on low-confidence output;
- Introducing post-processing validation;
- Enabling rule-based constraints on key business fields;
- Explicitly labeling content that “should not be recognized” in the training data.
9. Inference Optimization: From Model Quality to Service Efficiency
The advantage of the two-stage approach lies not only in clearer recognition logic, but also in the fact that it is naturally suited to inference optimization.
9.1 Stage-level optimization
The global layout stage takes lower-resolution input and is suited to fast batch processing; the local recognition stage handles many regions and is suited to parallel inference. In engineering, each stage can be given its own timeout, retry, caching, and degradation policies.
9.2 Region-level optimization
For a document containing many regions, you can:
- Batch small text regions;
- Route large table regions separately;
- Reuse templates for repeated template-like regions;
- Asynchronously re-recognize low-confidence regions;
- Send text-free or pure-graphic regions through a lightweight path.
9.3 Service-level optimization
When deploying in the cloud, you can combine general capabilities such as object storage, container platforms, logging services, and monitoring alerts to build an elastic document parsing service. A typical flow is:
- The original document is uploaded to object storage;
- The scheduler triggers a parsing task;
- The inference service reads the image and runs two-stage parsing;
- Results are written to structured storage or a message queue;
- The monitoring system collects latency, failure rate, and confidence distribution;
- Abnormal tasks trigger alerts and manual review.
def parse_document(image):# Stage 1: Global layout analysisdownsampled = downsample(image)layout = infer_layout(downsampled)# Region sorting and filteringregions = sort_by_reading_order(layout.regions)regions = filter_low_confidence(regions)# Stage 2: Local content recognitionresults = []for region in regions:crop = crop_original(image, region.bbox)content = infer_content(crop,task_type=region.type,prompt=build_prompt(region.type))results.append(merge_region(region, content))# Post-processing and structured outputreturn postprocess(regions, results)
9.4 Degradation strategies
When the inference service is under heavy load or the risk of timeouts increases, degradation strategies can be applied:
- Reduce recognition precision for non-critical regions;
- Prioritize key fields;
- Move complex tables into an asynchronous queue;
- Return partial results and mark regions to be completed later;
- Trigger a secondary recognition task rather than failing immediately.
10. Evaluation: Character Accuracy Alone Is Not Enough
A document parsing system cannot be evaluated with a single metric, because the final result usually serves business-level structuring. For example, invoice recognition focuses on field accuracy, contract parsing on clause completeness, paper parsing on section order, and financial reports on table-structure reconstruction.
10.1 Layout evaluation
Common metrics include:
- Region detection rate;
- Region false-positive rate;
- Region-type classification accuracy;
- Coordinate error;
- Reading-order accuracy.
10.2 Recognition evaluation
Text recognition can focus on:
- Character accuracy;
- Field-level accuracy;
- Line-level accuracy;
- Table cell accuracy;
- Formula-structure accuracy.
10.3 End-to-end evaluation
For business scenarios, the end-to-end result matters more, such as:
- Whether key fields are fully extracted;
- Whether tables can be reconstructed into structured data;
- Whether reading order matches the business reading logic;
- Whether errors can be localized;
- Whether manual review cost has decreased.
10.4 Continuous regression
Every model or prompt change should be backed by a fixed regression set covering typical documents, long-tail documents, and the hard-case set. Only through version comparison, failure analysis, and monitoring feedback can system stability be improved continuously.
11. Common Engineering Problems and Optimization Suggestions
11.1 Improper global layout resolution
If the layout stage resolution is too low, small regions may be missed; if too high, compute cost rises. It is recommended to configure the scaling ratio dynamically based on page size and the minimum target region size.
11.2 Reading-order reconstruction errors
Multi-column, table, and caption-heavy pages are prone to ordering errors. It is recommended to introduce column-level detection, region priorities, and rule-based correction, rather than relying entirely on free-form model output.
11.3 Over-reliance on text recognition for tables
A table is not just a collection of text but a two-dimensional structure. For complex tables, a dedicated structure-reconstruction strategy should be designed, covering merged cells, cross-page tables, and header mapping.
11.4 Lack of robustness to low-quality scans
Low-quality images should be stabilized through data augmentation, secondary recognition, and confidence mechanisms. For key business documents, it is not recommended to write a single low-confidence output directly into storage.
11.5 Chasing end-to-end while ignoring explainability
Document parsing often needs to be debuggable and traceable. The two-stage approach naturally offers stage-level observability: layout errors can be checked against the region output, and recognition errors can be checked against the local crops and recognition results.
12. Summary
The core difficulty of high-resolution document parsing is not the accuracy of a single recognition model, but how to balance complex layouts, dense text, and high-resolution images. A two-stage, coarse-to-fine design that decouples layout from recognition offers a more engineering-oriented path: the first stage completes global structure understanding at low cost, while the second stage uses the original resolution to guarantee local detail recognition, improving overall efficiency and robustness.
In real-world deployment, technical teams also need to focus on data engineering, prompt design, inference optimization, an evaluation feedback loop, and a monitoring system. Only by combining model capability, data structures, service architecture, and business rules can you truly build a stable, scalable, and operable high-resolution document parsing system.
