Document OCR Decoding Compared: Diffusion Decoding vs. Autoregressive
This article examines why the decoding strategy has become a decisive variable in document OCR and parsing for RAG pipelines. It breaks down the structural bottlenecks of autoregressive decoding—serial speed limits, error accumulation, and unidirectional context—and explains how diffusion decoding reframes OCR as an inverse-rendering problem solved through iterative denoising. It also covers engineering considerations for training and inference, compares pipeline, autoregressive, and diffusion-based approaches, and outlines a practical evaluation framework and scenario-based selection guidance for choosing the right parsing solution.
Introduction: Document parsing is the most underestimated step in the RAG pipeline
When building a retrieval-augmented generation (RAG) system, most teams pour their effort into vector retrieval, reranking, and prompt design—while overlooking the very first stage of the pipeline: document parsing. Whether the source material is a PDF, a scan, or an image directly caps how good a knowledge base can ever be. Whatever content is lost during parsing can never be recovered by the retrieval and generation stages that come afterward.
Over the past two or three years, document parsing has evolved quickly: from the traditional “text detection + text recognition” pipeline, to end-to-end vision-language models that output Markdown directly, and now to diffusion decoding, which models recognition as an “inverse rendering” problem. Each shift in decoding has redrawn the boundaries of speed, stability, and structural fidelity. Rather than focusing on any single product, this article explains the differences between diffusion decoding and autoregressive decoding—and where each one fits—across three levels: technical principles, engineering practice, and evaluation methodology.
1. What a typical document parsing pipeline does
Most mainstream open-source document parsing tools are built on the same pipeline and expose similar capabilities:
- Multi-format input: parse electronic documents natively, and render scans and images to images before recognizing them;
- Structured output: emit hierarchical formats such as Markdown and JSON so results can be ingested directly;
- Layout analysis: detect regions such as titles, body text, tables, images, formulas, and headers/footers;
- Reading-order restoration: handle complex layouts like multi-column text, cross-page paragraphs, and mixed text/images;
- Multilingual and multi-platform: support many languages and run across a range of hardware.
Within the open-source ecosystem, two families of projects have steadily become de facto standards: “pipeline” approaches built from modular components, and end-to-end approaches built around a vision-language model. The former prioritizes controllability and generality; the latter has long sat in the top tier of document evaluation leaderboards and is often used as the baseline that new models are measured against.
The most notable recent shift is that research teams have brought diffusion decoding into document recognition, proposing to redefine document OCR as “solving an inverse-rendering problem through diffusion decoding.” In other words, the decoding strategy itself is becoming a key variable that shapes both quality and efficiency.
2. Why decoding is the core variable in OCR
The “encoder–decoder” structure is nearly universal across today’s vision recognition models. Take an end-to-end vision-language model as an example: the visual encoder turns the page image into a sequence of features, and the language model then generates the target text layer by layer. The encoder determines what the model “sees,” while the decoder determines “how the model writes down what it saw.”
The choice of decoding shows up along three dimensions at once:
- Speed: how latency grows as the output gets longer;
- Stability: whether errors keep amplifying along the sequence;
- Structural fidelity: whether structure such as title hierarchy, reading order, and table rows and columns can be reconstructed correctly.
3. The structural bottlenecks of autoregressive decoding
Today’s mainstream approaches all essentially follow the same path: a visual encoder extracts features, and a language model generates tokens one at a time from left to right. This is intuitive and compatible with general-purpose large-model architectures, but for document recognition it has three structural shortcomings.
3.1 The speed bottleneck of serial decoding
Autoregressive decoding is inherently serial: to generate the 100th token, the model must wait for the first 99 to finish. Total time is roughly proportional to output length. For long documents that easily run to tens of thousands of characters, latency accumulates linearly. During batching, samples of differing lengths leave the ones that finish early waiting for the rest, which pushes overall throughput down further.
In addition, token-by-token decoding typically maintains a key-value cache (KV Cache) that grows with the sequence. In memory-constrained environments, this directly shrinks the number of requests that can run concurrently.
3.2 Error accumulation and “exposure bias”
During training the model sees the true prefix; during inference it sees text it generated itself. Once a missing character, a wrong character, or a repetition appears, everything generated afterward is built on top of that error. In long documents this bias tends to amplify in the later portion, showing up as:
- paragraphs that loop and repeat;
- table rows that shift out of alignment or bleed columns across rows;
- entire sections that go missing with no explicit signal.
3.3 The limits of unidirectional context
When generating left to right, the model cannot see content to the right while predicting the current token. Yet many document recognition decisions require exactly that kind of “looking ahead”—for example:
- disambiguating an unclear character based on the semantic context that follows it;
- deciding, from the surrounding text, whether a line break is an intra-paragraph wrap or a paragraph boundary;
- determining whether the current cell is a header or data.
This kind of disambiguation that depends on global information can only “guess and revise” under unidirectional decoding—and autoregressive decoding, once it has emitted a token, cannot go back to change it.
4. Diffusion decoding: treating OCR as an inverse-rendering problem
4.1 The core intuition: from “canvas” to “content”
Rendering is “given text and layout, produce an image”; recognition is its inverse: “given an image, recover the text and structure that produced it.” A document image can be viewed as something “rendered” from a page’s source, so recognition is the act of solving that inverse process.
Diffusion decoding is designed along exactly these lines:
- Initial state: a token sequence the same length as the input, with every position masked—effectively a “blank canvas”;
- Iteration: at each step the model uses bidirectional attention to predict all positions on the canvas simultaneously; the highest-confidence positions are committed, and the rest are re-masked for the next round;
- Convergence: when the steps run out, the whole canvas is filled in, and the denoising is complete.
This is fundamentally different from autoregression’s “append one token at a time.” The following illustrative pseudocode makes the contrast concrete:
# Autoregressive decoding: only one position can be determined at a timedef autoregressive_decode(image, max_len):tokens = [BOS]while tokens[-1] != EOS and len(tokens) < max_len:# Needs one full forward pass, depending on the entire generated prefixnext_token = language_model(image, tokens)[-1]tokens.append(next_token)return tokens# Diffusion decoding: update all undetermined positions in parallel each stepdef diffusion_decode(image, num_steps, length):tokens = [MASK] * length # Initial "canvas"ratios = mask_schedule(num_steps) # Keep ratio for each stepfor t in range(num_steps):logits = bidirectional_model(image, tokens)[0] # One forward pass predicts all positionsprobs = softmax(logits)# 1) Only focus on positions still in the masked stateconfidence = masked_select(probs, is_masked(tokens))# 2) Keep the ratio[t] proportion of positions, ranked by confidence high to lowkeep_count = int(num_masked(tokens) * ratios[t])tokens = commit_best(tokens, confidence, keep_count)# 3) Re-mask the remaining positions for the next round (actual implementations are also affected by a scheduling decay factor)tokens = remask_remaining(tokens)return detokenize(tokens)
The pseudocode highlights three key differences:
- Every step predicts all undetermined positions in parallel;
- Tokens committed early can still be revised in later steps, giving the process a built-in ability to “go back and fix”;
- Total time depends mainly on the number of denoising steps, and its correlation with output length is significantly weakened.
4.2 A direct response to the three bottlenecks
- Speed: when output length doubles, autoregressive time roughly doubles as well, whereas diffusion time is governed mainly by the number of steps—a more pronounced advantage for long documents. Each step is also a regular, batched computation that suits GPU parallelism, which greatly relieves the “waiting within a batch” problem.
- Error accumulation: an error at one step is not permanently “locked in”; later steps can re-predict and correct it using a more complete bidirectional context.
- Unidirectional context: bidirectional attention lets every position see the whole line, and even the whole page, at once—so disambiguation of unclear characters and reading-order decisions come closer to being solved globally.
5. Engineering essentials for using diffusion decoding
5.1 Training objective: “reconstruction” instead of “continuation”
Training a diffusion-based text model centers on masked reconstruction: randomly mask a proportion of tokens and have the model recover the masked content from the remaining context, so it learns to predict at “any position, in any order.” Document corpora also need to cover two kinds of hard cases during training:
- Structure-sensitive samples: tables, formulas, and multi-column layouts;
- Degraded samples: scan noise, skew, seal occlusion, and low resolution.
5.2 Scheduling strategy: lay out the “skeleton” first, then fill in the “flesh”
How many tokens to keep at each step is decided by the mask schedule. In practice, decay curves such as linear and cosine are common, and the core idea resembles painting “outline first, details later”:
- early steps settle the skeleton—title hierarchy, paragraph boundaries, table structure;
- later steps concentrate on confusable characters, punctuation, and superscripts/subscripts.
It’s worth noting that different implementations vary in the details of their mask schedules. Some stop re-masking between certain steps, letting the first and second halves of the sequence advance in “block-wise parallelism.” From an engineering standpoint, two things matter: whether the scheduling curve matches the actual number of steps, and whether the step count can adapt dynamically to page complexity.
5.3 Structure awareness: let the model “see” the layout
Beyond image features, layout information should be introduced explicitly:
- text block coordinates, width, height, and category (title/body/table/formula);
- page number, page order, and cross-page continuation relationships;
- table row and column indices, which can be supplied as additional tokens or conditioning information.
5.4 Inference engineering: batching and caching
The computation of each diffusion step is highly regular, which suits static shapes and large-granularity batching. When deploying, consider:
- grouping pages by size and normalizing them to the same resolution to reduce padding waste;
- using fewer steps for short documents and more for long ones, avoiding the waste of a “one-size-fits-all” setting;
- caching parse results by file fingerprint, and recomputing only the pages that change when a document is updated.
6. Comparing the three approaches side by side
It’s worth stressing that pipeline, autoregressive end-to-end, and diffusion decoding models are not a simple case of one “replacing” another—they are three tradeoffs aimed at different constraints.
| Dimension | Traditional pipeline | Autoregressive end-to-end model | Diffusion decoding end-to-end model |
|---|---|---|---|
| Time vs. output length | Each stage’s time is relatively fixed | Roughly linear growth | Mainly depends on the number of denoising steps |
| Batch efficiency | High | Affected by differences in sequence length | High, with regular computation |
| Error propagation | Upstream misses cannot be recovered | Errors accumulate easily and are hard to revise | Later steps can correct earlier ones |
| Context usage | Modules are relatively isolated | Mostly unidirectional | Bidirectional and global |
| Controllability and interpretability | Strong; segments can be swapped out individually | Weaker | Weaker |
| Engineering barrier | Many components, tedious tuning | Mature ecosystem | Requires a custom inference flow |
| Suitable scenarios | Regular layouts, throughput-first | General use, complex semantics | Long documents, high concurrency |
That some open-source projects offer a switchable “pipeline + model” dual backend is itself evidence of this trend: different scenarios call for different combinations of decoding and architecture, and there is no single approach that wins everywhere.
7. How to evaluate a document parsing solution objectively
To answer “how good is it,” you can’t rely on a single leaderboard score. It’s better to build an evaluation system across three levels.
7.1 Per-task metrics
- Text recognition: character error rate, word error rate, and normalized edit distance;
- Table recognition: cell-content matching, together with tree-edit-distance-style metrics that reflect row and column structure;
- Formula recognition: expression-level edit distance and a comparison against the rendered restoration;
- Layout analysis: precision/recall of region detection, and whether the reading order is correct;
- Document-level comprehensive evaluation: the industry already has dedicated document parsing benchmarks that models frequently use as points of comparison. When using such leaderboards, also check whether the evaluation version is consistent, how the weights of text versus formulas are set, and whether scanned and digital PDFs are distinguished.
7.2 Engineering metrics
- Throughput and latency: average time per page, single-page P95 latency, and total processing time for long documents;
- Hardware cost: CPU/GPU resource usage, peak VRAM, and achievable concurrency;
- Stability: whether repeated runs on the same document produce consistent results, and whether the system degrades gracefully under extreme layouts;
- Output usability: whether the emitted Markdown has properly closed tags and aligned tables, and whether it can be consumed correctly by downstream parsers.
7.3 Task-level metrics
In the end it comes back to business goals. In a RAG setting, you can build a question-answering evaluation set and compare answer correctness and citation accuracy across parsing approaches; in a data-ingestion setting, you can compare the number of valid chunks produced after cleaning.
7.4 Build your own test set
Public leaderboards can’t substitute for your own data. Sample along these dimensions:
- Source type: native electronic documents, scans, phone photos, and screenshots;
- Layout complexity: single column, multi-column, cross-page tables, and mixed text/images;
- Content type: body text, tables, formulas, and footnotes;
- Language: mixed Chinese–English and low-resource languages;
- Degree of degradation: skew, blur, seal occlusion, and low resolution.
Above all, keep the “hard cases” from real business traffic—they often reveal an approach’s weak spots far better than any leaderboard.
8. Putting it into practice: choosing by scenario
- Regular layouts, high throughput: large batches of standardized documents (e-invoices, forms, IDs) can favor pipeline or diffusion approaches for predictable latency and throughput.
- Complex layouts and low-quality scans: multi-column academic papers, borderless tables, and complex formulas are areas where end-to-end models usually have an edge in structural and semantic understanding. If speed and controllability matter more, diffusion decoding is worth validating as a priority.
- Hybrid routing: use a lightweight classifier to judge the page type first, sending regular pages to the pipeline and complex pages to an end-to-end model, backed by low-confidence recomputation and manual spot-checks—balancing cost and quality.
- Ecosystem coordination: parsing is only the starting point, and its output must connect with chunking, vectorization, and retrieval. For example, keep tables and formulas as intact blocks that aren’t split apart, and map title hierarchy to chunk boundaries to improve retrieval hits.
Conclusion
Document parsing is moving from “module assembly” toward “end-to-end generation,” and the generation itself is moving from token-by-token autoregression toward parallel denoising. Through bidirectional context, a revisable iterative process, and latency that is only weakly correlated with length, diffusion decoding strikes directly at autoregressive decoding’s weak points in speed, error accumulation, and global modeling—and it turns the intuition of “inverse rendering” into a concrete mathematical formulation.
But technology selection can’t hinge on leaderboard scores alone. For RAG and document-processing teams, the more pragmatic path is to understand the underlying differences between decoding strategies, build an evaluation set that fits your own data, and let pipeline, autoregressive, and diffusion approaches each do what they do best. Only when parsing is done on solid ground does the ceiling for retrieval and generation truly rise.
