Decoding as Recognition: A Technical Breakdown of Multimodal OCR Models

    This article examines how multimodal OCR models turn document images into structured text through a decode-and-generate paradigm. It walks through the encoder-decoder architecture, the role of the cross-modal connector in resolving the resolution-versus-cost tradeoff, and two core training strategies—multi-token prediction and stable all-task reinforcement learning with verifiable rewards. It then covers a practical two-stage inference pipeline that combines layout analysis with parallel recognition, along with guidance on data construction, evaluation, deployment, and quality assurance. Developers building document-understanding systems will find a reusable design framework for balancing accuracy, throughput, and cost across their own document distributions.

    1. From “segment-and-recognize” to “decode-and-generate”: the paradigm shift in document OCR

    Electronic invoices, scanned contracts, financial reports, academic papers, receipt archives, and handwritten forms make up one of the largest categories of unstructured data in enterprise digitalization. The text, tables, and layouts inside them are often the first entry point for risk review, financial automation, and knowledge-base construction.

    Reading these documents accurately, correctly, and completely is not easy. Take a typical traditional OCR pipeline: it is usually a chain of four stages:

    1. Text detection: locate text lines or blocks.
    2. Orientation correction: handle skew, rotation, and perspective distortion.
    3. Text recognition: convert each image region into a character sequence, line by line.
    4. Post-processing and structuring: correct errors with rules, dictionaries, or language models, then reconstruct the layout from templates or geometric relationships.

    This pipeline is mature and modular, but it has clear structural limitations on complex documents:

    • Fragile rules: multi-column papers, two-up receipts, and reports with merged cells all break template-based rules as soon as the layout shifts slightly.
    • Disconnected stages: if detection misses or truncates a box, the recognition stage cannot recover the missing text no matter how hard it tries—errors only propagate forward.
    • No global semantics: line-by-line recognition sees no context, so it cannot decide whether the second character in “1O1” is the letter O or the digit 0, nor exploit structural cues like heading hierarchy or table headers.
    • Weak structuring: reading order, table reconstruction, and formula typesetting usually require bolting on a separate set of algorithms.

    To break through these bottlenecks, the industry has developed three main modeling paradigms:

    Paradigm Typical structure Strengths Limitations
    Multi-stage pipeline Detection model + recognition model + structuring rules Independent modules, easy to swap, controllable inference cost Poor layout adaptation, errors propagate stage by stage
    Document-specific model Vision encoder + lightweight connector + small decoder Optimized for documents, outputs structured results like Markdown and LaTeX end to end Limited general knowledge and open-ended understanding
    General multimodal LLM General vision encoder + large language model Strong semantic understanding and instruction following, also handles document QA Vision resolution and inference cost constrain each other

    Overall, multimodal OCR is evolving along two main lines:

    • Make the vision encoder “see more clearly,” preserving small text and thin lines rather than losing details.
    • Make the decoder “write more accurately,” directly generating structured markup text instead of character fragments.

    Document-understanding multimodal OCR models recently released by the open-source community are representative attempts along this path: they use a lightweight language decoder to control cost while producing structured results directly under complex layouts. The sections below break down their architecture, training strategy, and inference flow.


    2. Overall architecture: a three-part encoder-decoder design

    These models follow the classic encoder-decoder paradigm, built from three core components:

    1. Document image block Vision encoder Cross-modal connector Language decoder
    2. ┌──────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
    3. │ Image │──patch──▶ │ ViT-style │──hidden▶│ Token down- │──▶ │ Lightweight │──▶ Text / Markdown
    4. │ block │ │ vision │ states │ sampling + │ │ language │ / LaTeX / JSON
    5. │(table/para)│ │ backbone │ │ projection │ │ decoder │
    6. └──────────┘ └──────────────┘ └──────────────┘ └──────────────┘

    The overall forward pass can be summarized as:

    1. The image is split into fixed-size blocks, which the vision encoder converts into a feature sequence.
    2. The cross-modal connector compresses and projects the visual tokens into a sequence aligned with the language model’s embedding dimension.
    3. The visual tokens are concatenated with the text instruction and fed into the language decoder.
    4. The decoder generates the result autoregressively, token by token.

    2.1 Vision encoder: learning to “read” layout from large-scale image-text data

    The vision encoder typically uses a patch-based Transformer backbone, pretrained on large-scale paired image-text data to learn the alignment between image regions and language descriptions. When transferred to the document domain, it offers two standout advantages over general-purpose vision backbones:

    • Sensitivity to lines and structure: layout cues like table borders, heading font sizes, and indentation often determine document structure more than the text strokes themselves.
    • Joint modeling of glyph and semantics: character shapes and contextual semantics are captured in the same representation, giving better robustness on handwriting and stylized type.

    Notably, the document domain demands far higher input resolution than natural images. The stroke width of body text at a standard font size may span only a few pixels, and small footnotes and thin table lines are even harder to resolve. If the input resolution is insufficient, no decoder can conjure back these details. “Seeing clearly” is the prerequisite for “recognizing accurately.”

    2.2 Cross-modal connector: solving the “resolution-cost” dilemma with token downsampling

    Where the problem comes from: the number of tokens the vision encoder outputs is proportional to the image area. Assuming a block size of 16×16 pixels:

    • A 512×512 image produces 1024 visual tokens.
    • A 1024×1024 image produces 4096.
    • A 2048×2048 image produces 16384.

    Meanwhile, attention computation in the language decoder scales quadratically with sequence length. Doubling the resolution grows the computation to roughly sixteen times, and memory usage climbs just as sharply. This is the “resolution-cost” tension that multimodal models commonly face, and it is the most prominent engineering constraint in long-document scenarios.

    Compression strategies: the core task of the cross-modal connector is to compress the token count while preserving as much key information as possible. Mainstream approaches fall into three categories:

    1. Spatial-shuffle compression: concatenate adjacent 2×2 visual tokens into one large-channel vector, then project it into the language model space through an MLP, reducing the token count to a quarter.
    2. Query-based resampling: use a set of learnable query vectors to aggregate information from many visual tokens via cross-attention, compressing a visual sequence of arbitrary length into a fixed number of tokens.
    3. Lightweight projector: use only a two-layer MLP for dimension alignment, without compressing tokens—minimal information loss but the highest cost.

    The key tradeoff: a higher compression ratio is not always better.

    • Too few tokens smooth away high-frequency details like small text, thin lines, and formula subscripts and superscripts, degrading table and layout reconstruction.
    • These models therefore typically use a milder downsampling ratio, or retain a certain number of fine-grained tokens.
    • A common engineering compromise is “global thumbnail + local high-resolution blocks”: first use a low-resolution image to grasp the overall structure, then encode accuracy-critical regions separately at higher resolution.

    Relationship to general multimodal frameworks: regardless of the specific implementation, the connector’s role is identical to that of a “projection layer”—mapping the visual representation into the language model’s embedding space. At this level, document-specific models and general multimodal models share the same design philosophy; the differences lie mainly in compression strategy, input resolution, and decoder size.

    2.3 Language decoder: trading small scale for controllable cost

    The decoder uses a lightweight language model under a billion parameters, for three reasons:

    • Focused task: the core of OCR is “transcribing what is seen,” which emphasizes faithful reconstruction of visual features and depends less on large-scale world knowledge.
    • Controllable cost: the decoder drives autoregressive generation and dominates inference cost in high-concurrency batch scenarios; the smaller it is, the lower the per-page cost.
    • Ease of distribution: small models can be deployed on a single GPU or even consumer-grade hardware, and are also easier to distill, quantize, and ship in private deployments.

    It is worth stressing that “lightweight” does not mean “weak.” Decoding quality depends more on three things: the coverage of the training data, whether the tokenizer fully covers formula and table symbols, and whether output-format constraints are in place.


    3. Training strategy one: multi-token prediction loss

    3.1 The basic idea

    The standard language-modeling objective is “next-token prediction”: at each position, the model is asked only to predict the single token that immediately follows. Multi-Token Prediction (MTP) instead asks the model to predict several future tokens at each position.

    Concretely, this sets up multiple independent prediction heads on top of the backbone:

    • The 1st head predicts the next token, equivalent to the standard objective.
    • The 2nd head predicts the token after that, and so on.
    • During training, the cross-entropy losses of the heads are combined as a weighted sum.

    The diagram below illustrates this:

    1. Input positions: ... t-1 t t+1 t+2 ...
    2. Backbone hidden: ... h(t-1) h(t) h(t+1) h(t+2) ...
    3. │ │ │ │
    4. ┌─────────┼──────────┼──────────┼─────────┼─────────┐
    5. ▼ ▼ ▼ ▼ ▼ ▼
    6. Head 1 (next) ŷ(t) ŷ(t+1) ŷ(t+2) ŷ(t+3) ŷ(t+4) ──▶ standard cross-entropy
    7. Head 2 (next 2) ŷ(t+1) ŷ(t+2) ŷ(t+3) ŷ(t+4) ŷ(t+5) ──▶ cross-entropy (weight λ₂)
    8. Head 3 (next 3) ŷ(t+2) ŷ(t+3) ŷ(t+4) ŷ(t+5) ŷ(t+6) ──▶ cross-entropy (weight λ₃)
    1. import torch
    2. import torch.nn as nn
    3. import torch.nn.functional as F
    4. def mtp_loss(hidden, targets, heads, weights, ignore_index=-100):
    5. """
    6. Illustrative implementation of the multi-token prediction loss.
    7. hidden : [B, T, D] hidden states output by the backbone
    8. targets: [B, T] target token sequence (already shifted right by one)
    9. heads : nn.ModuleList; head i predicts the (i+1)-th future token
    10. """
    11. total = hidden.new_zeros(())
    12. for i, head in enumerate(heads):
    13. logits = head(hidden) # [B, T, V]
    14. if i > 0:
    15. # The last i positions lack future labels, so trim them out of the loss
    16. logits = logits[:, :-i, :]
    17. tgt = targets[:, i:]
    18. else:
    19. tgt = targets
    20. total = total + weights[i] * F.cross_entropy(
    21. logits.reshape(-1, logits.size(-1)),
    22. tgt.reshape(-1),
    23. ignore_index=ignore_index,
    24. )
    25. return total / sum(weights)

    3.2 Three benefits for OCR tasks

    • Denser supervision signal: document text is highly regular in structure. Patterns like table headers, formula delimiters, and Markdown markup sequences recur at nearby positions. Multi-token prediction forces the model to “look a few steps ahead” at each position, learning these local patterns earlier.
    • Stronger intermediate representations: to predict farther targets, the backbone must encode longer-range contextual dependencies, making the representation more sensitive to layout details and semantic associations and improving the stability of table-structure reconstruction and formula recognition.
    • Higher sample efficiency: the same batch of training data provides multiple supervision targets at each position, effectively expanding the amount of supervision without increasing annotation cost—especially valuable given how expensive document annotation is.

    3.3 From training objective to inference acceleration

    MTP was originally proposed by researchers at Meta and other institutions to improve language-model training. In models such as DeepSeek-V3, it was further applied to speculative acceleration at inference time. The idea works as follows:

    • Based on the main prediction head’s output, the extra heads “guess” several subsequent tokens in a single pass.
    • The model then verifies these candidates in parallel against itself, accepting the correct prefix and thereby generating multiple tokens per forward pass.

    This idea applies equally to OCR inference. Document text has strong local regularities, so the model’s “hit rate” on guessing table delimiters and repeated field names is often high, and speculative decoding has the potential to deliver meaningful throughput gains. Note that the speedup varies by task, document type, and implementation, so it should be validated empirically. If acceleration is not the goal, you can keep only the main prediction head at inference time; the extra heads add no overhead.

    3.4 Points to watch when using it

    • Error accumulation: the k-th head must predict the k-th future token, and farther targets are harder to predict. Weights should decay with depth to prevent noise from distant heads from backwashing into the backbone.
    • Sensitivity to label noise: typos and punctuation errors mixed into OCR annotations get amplified across multiple heads, so thorough data cleaning is especially important for MTP training.
    • Not a panacea: gains are limited in short-text, low-information-density scenarios. Use ablation studies to decide whether it is worth introducing.

    4. Training strategy two: stable all-task reinforcement learning

    4.1 Why reinforcement learning is needed beyond supervised fine-tuning

    Supervised fine-tuning (SFT) teaches the model to imitate annotations, but it has two inherent limitations:

    1. Metric mismatch: the training objective is token-level cross-entropy, while the business cares about sequence-level discrete metrics such as whole-sentence edit distance, field accuracy, and table-structure correctness.
    2. No self-correction: once the model drifts during inference, SFT cannot teach it to recover from an erroneous state.

    Reinforcement learning fills exactly these gaps: it uses computable task metrics directly as rewards, gives feedback on complete outputs, and encourages the model to explore better transcription and layout strategies on its own. “All-task” means the reward signal covers multiple document-understanding tasks: text recognition, table structuring, formula recognition, field extraction from receipts and licenses, and document QA. Multiple tasks share the same policy model, with unified reward design and unified optimization.

    4.2 Designing stable, verifiable rewards

    Whether reinforcement learning converges stably hinges on reward design. The industry widely adopts a “verifiable reward” approach: rewards are decided automatically by programs rather than by another model’s scoring, avoiding the bias and noise of the reward model itself. A practical combination looks like this:

    Reward term Computation Purpose
    Text consistency 1 − normalized edit distance Measures transcription accuracy
    Structural correctness Cell-level / field-level accuracy Measures row-column and key-value reconstruction quality
    Format legality Whether output parses as valid JSON, a Markdown table, or renderable formula syntax Ensures output is directly consumable downstream
    Redundancy penalty Ratio of repeated segments, degree to which length exceeds the reference Suppresses hallucination and repetition
    Length normalization Reward normalized by actual output length Prevents longer answers from winning simply by “scoring more”
    1. import json
    2. def length_normalized_edit_reward(pred: str, ref: str) -> float:
    3. """Illustrative normalized text reward based on edit distance."""
    4. dist = edit_distance(pred, ref) # call an edit-distance implementation
    5. denom = max(len(ref), 1)
    6. return max(0.0, 1.0 - dist / denom)
    7. def field_accuracy(pred: dict, ref: dict) -> float:
    8. """Field-level accuracy for scenarios like receipts and licenses."""
    9. keys = set(ref.keys())
    10. if not keys:
    11. return 0.0
    12. hit = sum(1 for k in keys if pred.get(k, "").strip() == ref[k].strip())
    13. return hit / len(keys)
    14. def verifiable_reward(pred_text: str, task: dict) -> float:
    15. """Compute the verifiable reward by task type (illustrative)."""
    16. try:
    17. pred = json.loads(pred_text)
    18. except json.JSONDecodeError:
    19. return -1.0 # invalid format, apply a penalty
    20. if task["type"] == "table":
    21. return cell_level_accuracy(pred, task["gt"])
    22. if task["type"] == "kv":
    23. return field_accuracy(pred, task["gt"])
    24. if task["type"] == "text":
    25. return length_normalized_edit_reward(pred_text, task["gt_text"])
    26. return 0.0

    4.3 What “stable” involves on the engineering side

    Multi-task reinforcement learning is prone to “robbing Peter to pay Paul” and to training divergence. Stability usually requires coordination across several fronts:

    1. Simplified advantage estimation: the current mainstream approach borrows from Group Relative Policy Optimization (GRPO): sample multiple responses per prompt and normalize the group’s rewards by their mean and standard deviation to obtain advantages, eliminating the separate value network and significantly cutting memory and tuning costs.
    2. Reference-policy constraint: add a KL penalty against the reference model to the objective, limiting how far the policy drifts from the SFT model and preventing “format collapse” and overconfidence.
    3. Balanced task sampling: task sample sizes and difficulties vary widely. Sampling probabilities are set by preset ratios or adjusted dynamically by current performance, avoiding high-frequency easy tasks dominating the gradient.
    4. Difficulty curriculum: raise difficulty progressively along “short text → single table → multi-column long document → multi-page document,” while continuously monitoring each task’s reward curve.
    5. Reward clipping and normalization: truncate or standardize anomalous rewards to prevent individual samples from producing overly large gradients.
    1. import torch
    2. def grpo_style_loss(group_rewards, group_logps, ref_logps, beta=0.05, eps=1e-4):
    3. """
    4. Illustrative GRPO-style objective.
    5. group_rewards: [G] rewards of G samples for the same prompt
    6. group_logps : [G, T] log-probabilities of the sampled tokens under the current policy
    7. ref_logps : [G, T] corresponding log-probabilities under the reference model
    8. """
    9. mean = group_rewards.mean()
    10. std = group_rewards.std(unbiased=False).clamp_min(eps)
    11. advantages = (group_rewards - mean) / std # group-relative advantage
    12. ratio = torch.exp(group_logps - group_logps.detach()) # always 1 in a single update step; for illustration
    13. policy_loss = -(ratio * advantages.unsqueeze(-1)).mean()
    14. # token-level KL approximation, constraining the policy not to drift too far from the reference model
    15. kl = (group_logps - ref_logps).mean()
    16. return policy_loss + beta * kl, advantages

    4.4 Realistic boundaries

    Reinforcement learning is not a panacea:

    • It cannot fix annotation errors: when the reward itself is wrong, the policy just learns the wrong thing more efficiently.
    • Rewards can be gamed: the model may learn to produce format-valid but content-empty answers, so format rewards must be used together with content rewards.
    • It is costly: sampling and scoring both require extra compute, and training takes longer than pure SFT.

    A sensible order is: first establish baseline capability with high-quality supervised fine-tuning, then use reinforcement learning for task-level fine-tuning.


    5. Inference flow: a two-stage scheme of layout analysis and parallel recognition

    5.1 The bottleneck of single-stage direct decoding

    The most direct approach is to feed the entire page image into the model and generate the complete result in one pass. It is simple to implement but faces three bottlenecks:

    1. Overly long sequences: a single A4 page often contains thousands of text tokens, and decoding time grows linearly with output length, making end-to-end latency hard to meet for online services.
    2. Scattered attention: when both input and output are long, the model tends to skip regions, repeat output, or “drift off-topic” in the later part of the page.
    3. High memory pressure: full-page high-resolution input yields a huge number of visual tokens.

    Borrowing the “layout first, recognition second” divide-and-conquer idea from traditional pipelines and combining it with multimodal capabilities, the industry has formed a two-stage flow of “layout analysis + parallel recognition.”

    5.2 Stage one: layout analysis

    Using a layout analysis model (such as a lightweight layout detection approach optimized for the document domain), the page is analyzed to output each region’s category and coordinates. Common categories include body text, headings, tables, figures, formulas, headers and footers, stamps, and handwriting regions. On this basis, two key tasks are completed:

    • Reading-order sorting: primarily “top to bottom, left to right,” combined with cues like column gaps and indentation, to determine the correct reading order for single-, double-, and even triple-column documents.
    • Region filtering: removing noisy regions like headers, page numbers, and watermarks to reduce invalid recognition and interference.

    This stage is essentially about “reducing the recognition burden”: it breaks a whole page of complex layout into several structurally simpler blocks, making subsequent recognition easier and more parallel.

    5.3 Stage two: parallel recognition

    Each region is cropped into a block and sent to the multimodal model for batch recognition; regions are independent and do not block one another. Several practices matter on the engineering side:

    • Dynamic batching: bucket and batch blocks from different sources and sizes by dimensions to improve GPU utilization.
    • Resolution adaptation: enlarge text regions appropriately, while focusing on semantic content for figures and stamps.
    • Fine-grained task routing: select the output format by region type—for example, plain text or Markdown for body text, structured cells for tables, and LaTeX markup for formulas.
    • Type-aware prompts: state the current region’s type explicitly in the instruction (e.g., “recognize the following content as a Markdown table”) to guide the output format.
    1. from concurrent.futures import ThreadPoolExecutor
    2. from dataclasses import dataclass, field
    3. @dataclass
    4. class Block:
    5. id: str
    6. label: str # text / table / formula / figure ...
    7. bbox: tuple # (x1, y1, x2, y2)
    8. page: int = 1
    9. result: str = ""
    10. confidence: float = 0.0
    11. def recognize_block(block: Block, image, model) -> Block:
    12. """Recognize a single region and return structured results."""
    13. prompt = PROMPTS.get(block.label, PROMPTS["text"])
    14. block.result = model.infer(image, block.bbox, prompt)
    15. block.confidence = model.last_confidence
    16. return block
    17. def parse_document(image, layout_model, ocr_model, max_workers=8):
    18. # Stage one: layout analysis + reading-order sorting
    19. blocks = [Block(**b) for b in layout_model.analyze(image)]
    20. # Stage two: parallel recognition
    21. with ThreadPoolExecutor(max_workers=max_workers) as pool:
    22. list(pool.map(lambda b: recognize_block(b, image, ocr_model), blocks))
    23. # Assemble into Markdown output by reading order
    24. return render_markdown(sorted(blocks, key=lambda b: (b.page, b.bbox[1], b.bbox[0])))

    5.4 Output example

    1. # 2024 Annual Business Analysis ← title
    2. ## 1. Overall Business Performance ← title
    3. During the reporting period, the company achieved revenue ... ← text
    4. | Item | 2024 | 2023 | ← table (structured cells)
    5. | Revenue | ... | ... |
    6. $EBITDA = operating profit + depreciation + amortization$ ← formula (formula markup)

    5.5 Hybrid strategy: balancing global and local

    The pure two-stage flow still has weaknesses:

    • Loss of cross-region information: cross-page tables, paragraphs governed by a heading spanning multiple columns, and references like “see table below” in body text can only be understood correctly within a larger context.
    • New errors from the layout model: misses and misdetections in layout analysis directly affect recognition results.

    In practice, a “hybrid strategy” can be adopted:

    1. First feed the model a full-page thumbnail for a global view, to determine the document type and overall structure.
    2. Then perform fine-grained parallel recognition on each region.
    3. Finally, an “integration stage” unifies numbering, stitches together cross-page content, and verifies the correspondence between figure/table captions and body text.

    Meanwhile, each region’s recognition result should carry coordinates and a confidence score to support traceability and spot checks.


    6. Engineering: data, evaluation, and deployment

    6.1 Data construction

    • Diverse sources: real scanned documents, receipts, and archives, combined with synthetic multi-layout documents and layout perturbations (skew, noise, stamp occlusion).
    • Structured annotation: annotate body text, tables, and formulas as plain text, structured cells, and formula markup respectively, so formats are directly consumable.
    • Difficulty grading: stratify by layout complexity and scan quality to support curriculum learning and per-scenario evaluation.

    6.2 Evaluation system

    Coverage should include at least the following dimensions:

    • Accuracy: use normalized edit distance and character error rate at the text level, cell and row-column structure correctness at the table level, and syntax legality and symbol accuracy at the formula level.
    • Completeness: region recall and cross-page stitching correctness, to check for missed reads and misordering.
    • Stability: consistency of results across different resolutions and scan qualities for the same document.
    • Efficiency: average per-page time, throughput, and inference cost per thousand pages.

    Build the evaluation set to reflect your real business layout distribution, and avoid “gaming the score” only on public benchmarks.

    6.3 Deployment and cloud architecture

    A multimodal OCR service typically includes the following stages:

    1. Aggregate raw scans and recognition results via object storage for archiving and traceability.
    2. Use a message queue to level out batch-task spikes and valleys, achieving asynchronous decoupling.
    3. Deploy the layout analysis service and parallel recognition service elastically on a container platform, auto-scaling with load.
    4. Track task success rate, confidence distribution, and average latency via logging and monitoring/alerting systems.

    Take Baidu AI Cloud as an example: general capabilities like text recognition can cover standardized scenarios such as invoices, licenses, and general printed text; for in-house document models with special layouts, training and elastic inference can rely on GPU compute and containerization. Combining general capabilities with specialized models forms a tiered strategy: “general services for standard scenarios, self-built models for long-tail scenarios.”

    In addition, quantization methods like INT8 and FP8 can reduce memory usage; combined with continuous batching to improve throughput, and caching of layout analysis results to avoid redundant computation.

    6.4 Quality assurance: recognition confidence and human review

    Generative OCR carries a “hallucination” risk: the model may add content that does not exist in the original. Production systems should therefore establish a layered “automatic + human” mechanism:

    • Regions with confidence below a threshold automatically enter a review queue.
    • Key fields like amounts, account numbers, and dates undergo secondary validation by business rules.
    • Review results flow back as training data, forming a data flywheel.

    7. A candid look at limitations and future directions

    Objectively, these models still have clear boundaries of applicability:

    • Extremely long documents: with many pages, the single context window and inference cost remain a constraint, requiring retrieval and chunking strategies.
    • Extreme conditions: severely blurred, low-resolution, heavily occluded, or heavily stylized handwriting still shows a clear drop in accuracy.
    • Layout dependence: the two-stage flow’s results partly depend on layout analysis quality.
    • Semantic ambiguity: generative models may “polish” the original wording, which calls for special caution in high-fidelity scenarios like contracts and receipts.

    Future directions are roughly three:

    • Higher resolution and a better visual-token budget: see more details while keeping compute controllable.
    • Multi-task reinforcement learning centered on verifiable rewards: extend rewards to cover more structured targets like tables, formulas, and fields.
    • From “recognition” to “understanding”: unify layout analysis, text recognition, information extraction, and document QA into a single model, so document data directly serves business workflows.

    8. Conclusion

    Document-understanding solutions represented by open-source multimodal OCR models offer a reusable design paradigm across three levels—architecture, training, and inference:

    • Architecture: the vision encoder handles “seeing clearly,” the lightweight connector handles “compression,” and the small language decoder handles “writing accurately.”
    • Training: multi-token prediction strengthens local-structure modeling and intermediate representations, while stable all-task reinforcement learning aligns with business metrics.
    • Inference: the two-stage “layout analysis + parallel recognition” flow balances accuracy and throughput.

    For developers, the key to deployment is not chasing a single model, but building an evaluation system around your own document distribution, putting confidence scoring and human review mechanisms in place, and controlling overall cost with a “general capability + specialized model” combination.