Recursive Self-Improvement: Solving the Annotation Data Bottleneck in Document Parsing

    Document parsing remains a hard, unsolved problem in production: unlike plain OCR, it must recover a document's structure, reading order, and semantics from messy real-world inputs like two-column papers, split tables, formulas, and handwritten notes. The core obstacle is the astronomical cost of labeled training data. This article explores recursive self-improvement (RSI) as a way to break that dependency, using self-training and pseudo-labeling loops with safeguards against error amplification. It also covers what a genuinely useful parser must do, why structured output (Markdown, LaTeX, JSON) matters, how parsing quality sets the ceiling for RAG systems, and the engineering value of lightweight open-source models that can be deployed on-premise for compliance-sensitive scenarios.

    1. Document Parsing: An Underrated Technical Mountain

    Intuitively, “recognizing text in an image” seems like a problem that was solved long ago. But once you move into production, document parsing turns out to be a problem that has never been fully cracked. It is fundamentally different from text recognition: the latter cares about mapping pixels to characters, while the former requires recovering a document’s structure, semantics, and reading order at the same time as recognizing its text.

    A seemingly ordinary PDF may contain a two-column academic paper, a large table broken across pages, handwritten annotations, mathematical formulas, and even chemical structures. Traditional OCR facing such input often degrades into a stream of text that “looks like words but is completely unusable.” And once this poor-quality extraction enters a downstream pipeline, it almost inevitably causes overall quality to collapse.

    2. Why Traditional OCR Keeps Failing

    The typical traditional pipeline looks like: layout detection → text detection → text recognition → simple post-processing rules. This serial, module-by-module approach works acceptably on well-structured documents, but on real-world complex inputs its weaknesses are amplified quickly:

    • Scrambled reading order in multi-column layouts: Two-column papers are often processed with a “left-to-right, line-by-line” scan, causing content from the left and right columns to be concatenated serially and semantics to break entirely.
    • Handwriting and formulas fall apart: Handwritten glyphs vary enormously, and mathematical formulas are two-dimensional layout structures. A generic recognizer outputs a one-dimensional text stream that cannot reconstruct superscripts, fractions, radicals, and similar relationships.
    • Cross-page table structures disintegrate: A table gets split into isolated cell text, the row-column correspondence is lost, and it cannot be rebuilt into usable structured data.
    • Non-text elements are discarded: Chemical structures and diagrams are often filtered out as “cluttered lines,” even though they are exactly the key information for understanding the content.

    The common root cause of these problems: each submodule only optimizes a local objective, lacking a unified modeling and joint reasoning capability over the document as a whole.

    3. Annotation Data: The Bottomless Pit Hanging Over OCR

    To train a model that understands complex layouts, the most direct approach is to prepare massive amounts of high-quality annotations. Yet the cost of annotating documents is astonishingly high:

    1. Many annotation dimensions: You must not only draw boxes around text but also label reading order, layout categories (title/body/table/formula), and hierarchical relationships between elements. Single-page annotation effort far exceeds ordinary image classification.
    2. High expertise barrier: Chemical formulas, specialized equations, and complex tables require domain experts, and labor costs rise geometrically.
    3. Hard to cover the long tail: Real-world document element combinations are nearly infinite, and manual annotation can never keep up with the long tail of the distribution.

    This is the so-called “annotation data bottomless pit” — enormous investment with no way to be exhaustive. The way to break it is to reduce the model’s rigid dependence on manual annotation. Recursive Self-Improvement (RSI) is one approach to this dilemma: letting the model leverage its own reasoning ability to continuously “self-produce” high-quality training signals from a small amount of labeled data, forming a self-reinforcing positive loop.

    4. Recursive Self-Improvement: Let the Model Teach Itself

    In the context of document parsing, recursive self-improvement is usually implemented as an iterative process combining self-training and pseudo-labeling. The core idea: use the current model to infer on unlabeled data, filter out high-confidence results as “pseudo-labels,” feed them back into the training set, train a stronger model, and repeat. Below is a generic flow diagram of this idea:

    1. # Generic self-training loop for recursive self-improvement
    2. labeled_set = load_annotated_samples() # a small amount of finely annotated data
    3. unlabeled_pool = load_raw_documents() # a large pool of unlabeled documents
    4. model = init_model()
    5. for round_idx in range(num_rounds):
    6. # 1. Train on current (finely labeled + accumulated pseudo-labeled) data
    7. model = finetune(model, labeled_set)
    8. # 2. Infer on unlabeled data and output confidence
    9. new_pseudo = []
    10. for doc in unlabeled_pool:
    11. result, confidence = model.infer_with_confidence(doc)
    12. # 3. Only feed back high-confidence, structurally valid samples
    13. if confidence > tau and is_structurally_valid(result):
    14. new_pseudo.append((doc, result))
    15. labeled_set.extend(new_pseudo)
    16. unlabeled_pool = remove(unlabeled_pool, new_pseudo)

    For this loop to genuinely get “better with each round” rather than continuously amplifying errors, several key engineering constraints are needed:

    • Confidence and consistency dual filtering: Look not only at single-point confidence but also at whether multiple perturbed inferences (e.g., different crops, different resolutions) agree, filtering out “confidently wrong” results.
    • Structural validity checks: If a parsing result violates basic constraints (e.g., table row/column counts don’t match, LaTeX fails to compile), it should be discarded outright rather than included in training.
    • Anti-drift mechanisms: Set an upper limit on the pseudo-label ratio and keep a proportion of finely labeled data as an anchor, preventing the model from collapsing into a self-reinforcing error distribution.
    • Difficulty curriculum: Prioritize feeding back “medium-difficulty” samples; too-easy samples carry little information, while too-hard samples carry high noise.

    It is worth stressing that RSI is not about eliminating manual annotation entirely, but about freeing humans from the high-intensity repetitive labor of “annotating page by page” and shifting them toward “reviewing high-value samples” and “setting quality constraints.” This is the fundamental improvement in cost structure it offers over a purely manual approach.

    5. From Text to Structure: Key Capabilities for Complex Document Parsing

    A truly useful parser should be able to uniformly ingest multiple categories of document elements, not just extract plain text. Ideally, a system should recognize and reconstruct more than a dozen element types: titles, body paragraphs, tables, display/inline formulas, code blocks, images, chemical equations, footnotes, headers and footers, stamps, page numbers, figure and table captions, and so on, and rebuild the hierarchy and reading order among them.

    In typical scenarios, the value of this capability shows up in:

    • Magazines and multi-column literature: Outputting structured results that preserve complete semantic hierarchy and correct reading order, rather than scrambled text.
    • Mathematical formulas: Directly converting them into standard math markup (such as LaTeX) so formulas can be computed and rendered, rather than becoming garbled text.
    • Chemical structures: Restoring a two-dimensional structure into a structured representation, instead of discarding it as noise.
    • Exam papers and handwritten notes: Effectively parsing handwritten content as well, greatly widening the boundary of processable documents.

    Notably, with the self-improvement approach described above, lightweight models on the order of billions of parameters can already demonstrate competitiveness comparable to large models on several public document-parsing benchmarks. This means complex document parsing is no longer an arms race where “bigger parameters always win,” but can be achieved under limited compute through smarter methods.

    6. Structured Output: The Last Mile That Determines Downstream Usability

    Getting recognition right is not enough; how the output is organized equally determines success. Downstream systems need structured data that machines can consume, so standard formats such as Markdown, LaTeX, and JSON have become the de facto best carriers. An ideal parsing result roughly looks like this:

    1. # Section Title
    2. This is a body paragraph...
    3. Inline formula like $E = mc^2$, and display formulas stand alone as a block:
    4. $
    5. \int_{0}^{1} x^2 \, dx = \frac{1}{3}
    6. $
    7. | Col 1 | Col 2 | Col 3 |
    8. |-------|-------|-------|
    9. | a | b | c |
    10. ![Figure 1: System Architecture Diagram](image_ref_01)

    A quick comparison reveals the gap: plain text extraction gives you “a b c scattered across a paragraph,” while structured output preserves table semantics, formula compilability, and image reference relationships. The finer the element granularity and the higher the structural fidelity, the less work downstream systems face. So when evaluating a parsing solution, you cannot look only at character recognition accuracy; you must also look at the completeness of structural reconstruction and the standardization of the output format.

    7. RAG-Oriented: Why Parsing Quality Caps the Quality of Your Q&A

    Retrieval-Augmented Generation (RAG) has become the mainstream architecture for enterprise knowledge bases. But many people overlook a precondition: the ceiling of a RAG system is determined by the quality of the retrieved context, which is in turn determined by document parsing quality.

    If the parsing stage shreds a two-column paper into pieces or turns a table into garbled text, then:

    • During embedding, vectors generated from incomplete text have distorted semantics, and the relevant snippets retrieved are themselves off-topic;
    • No matter how smart the large model is, it receives incomplete context, and hallucination and off-topic answers become almost unavoidable;
    • Q&A involving numbers and structure (e.g., “what is the total of the third row in the table”) fails outright.

    Therefore, moving high-quality parsing to the front of the RAG data ingestion stage is the most cost-effective link for improving final answer quality. A well-designed ingestion pipeline typically looks like this:

    1. # RAG document ingestion pipeline (parsing up front)
    2. for doc in document_source:
    3. structured = parser.parse(doc) # high-quality structured parsing
    4. chunks = semantic_chunker(structured) # chunk by semantics/structure, not fixed length
    5. vectors = embedder.encode(chunks) # vectorization
    6. index.upsert(chunks, vectors) # write to vector store

    The key design here is structure-aware (semantic) chunking rather than naive fixed-length splitting: keeping a table or a formula unit intact, avoiding having key information cut off in the middle.

    8. The Value of Lightweight Open-Source Models and Engineering Recommendations

    The open-sourcing of such vertical-domain small models differs in meaning from the open-sourcing of general-purpose large language models. Large-model open-sourcing is more about democratizing capability and ecosystem, whereas open-sourcing a lightweight model focused on document parsing directly lowers the barrier for enterprises to build their own document processing pipelines — no reliance on expensive closed-source APIs, and no per-page paid inference costs. Data can be kept entirely within a private environment, which is especially critical for strongly regulated scenarios such as finance, healthcare, and government.

    Some pragmatic recommendations for deployment:

    • Match compute to throughput: Lightweight models can achieve relatively high batch parsing throughput on a single mainstream inference card. When evaluating, focus on “pages/second under a given unit of compute” rather than single-request latency.
    • Layer parsing and review: Automate the 80% of well-structured documents, and route low-confidence, complex-layout samples to manual review, balancing efficiency and quality.
    • Build an offline evaluation set: Construct a small evaluation set from real documents in your own business and continuously monitor parsing quality, guarding against silent degradation caused by model iteration or data distribution shifts.
    • Leverage general cloud capabilities: Parsing tasks can rely on object storage to hold raw documents, container platforms for elastic inference, and logging and monitoring/alerting services for quality tracking, scaling up and down on demand to control costs. For example, with general-purpose container and storage products from Baidu AI Cloud, you can quickly stand up an elastic, observable document parsing service.

    At the end of the day, the value of document parsing lies not in how “flashy” the recognition is, but in reliably converting real-world documents of wildly varying shapes into high-quality, machine-usable context. When recursive self-improvement significantly lowers annotation costs and lightweight models lower the deployment barrier, document parsing is shifting from a “manual craft” for a few teams into an infrastructure capability that every data-driven application can afford. Whoever connects this data entry point first will gain an edge in the competition over RAG and knowledge applications.