Autoregressive Blank Infilling: A Unified Pretraining Paradigm for Understanding and Generation in Large Language Models

    This article explains how autoregressive blank infilling unifies masked bidirectional modeling and autoregressive generation in a single pretraining objective. It covers the core idea, key mechanisms, engineering considerations, and practical deployment guidance for building large language models that need both understanding and generation capabilities.

    1. Two Main Lines of Pretraining and Their Limitations

    Large language model pretraining has evolved along two main technical lines.

    The first is the autoregressive generation line. In this paradigm, the model predicts the next token in a left-to-right order. Its training objective naturally aligns with the text generation process, making it effective for generation-oriented tasks such as writing, dialogue, and code completion. However, it struggles to fully leverage the “right-side” context in a single prediction step.

    The second is the masked language modeling line. Here, the model uses both the left and right context of masked positions to perform bidirectional modeling. This gives it stronger understanding and classification capabilities. However, it predicts discrete tokens during training but often requires autoregressive decoding during inference, creating a mismatch between the training objective and actual usage scenarios. In addition, when used directly for generation tasks, the user experience is often suboptimal.

    Since each line has clear limitations, the question of how to use a single pretraining objective to support both understanding and generation has become a long-standing topic in the field. Autoregressive blank infilling is one general solution proposed in this context. Its core contribution is to combine the ideas of “blank infilling” and “autoregressive generation” into one pretraining objective.

    2. Core Idea of Autoregressive Blank Infilling

    The idea of autoregressive blank infilling can be summarized in one sentence: first create blanks, then fill them in order.

    More specifically, during pretraining, several continuous spans are randomly selected from the original text and masked as a whole. The model must then regenerate the masked content in an autoregressive manner, using the surrounding context before and after the masked spans.

    The “autoregressive” part means that generation inside the blank still proceeds from left to right. As a result, this paradigm preserves the ability of masked models to use bidirectional context, while also retaining the generation advantages of autoregressive models.

    Compared with random token-level masking, using continuous spans as masking units is closer to real language structures, such as phrases, named entities, and sentence fragments. The model must understand semantic coherence in order to fill them correctly, which also encourages it to learn longer-range dependencies.

    3. Key Differences from Pure Autoregressive Modeling and Masked Modeling

    • Different masking units: Masked modeling usually masks individual tokens at random, while blank infilling masks continuous spans, making the masked units more semantically complete.
    • Different generation methods: Masked modeling predicts positions independently, while blank infilling generates the masked regions autoregressively, preserving causal structure.
    • Different task coverage: Pure autoregressive modeling is biased toward generation, while masked modeling is biased toward understanding. Blank infilling uses a unified objective to allow the same model to adapt to both types of tasks, reducing the cost and maintenance burden of running multiple models.

    4. Key Mechanisms Explained

    1. Span masking strategy.
    It is necessary to decide the masking ratio and the length distribution of masked spans. A common practice is to set an overall masking ratio, such as about 15%, and make span lengths follow a certain distribution, such as short spans with a mean length around 3. This helps balance two risks: masking too little makes the task too easy, while masking too much makes it too difficult.

    2. Distinguishable position encoding design.
    Because the text contains both preserved context and placeholders for missing content, the model needs to distinguish between them. A common approach is to assign distinguishable position information to different intervals. For example, two sets of position numbering can be built separately for masked regions and visible regions, allowing the model to correctly align context with infilling targets.

    Illustrative pseudocode is shown below:

    1. def build_infilling_sample(tokens, mask_ratio=0.15, mean_span=3):
    2. """Illustrative pseudocode: continuous span masking with distinguishable position encodings"""
    3. spans = select_random_spans(tokens, mask_ratio, mean_span)
    4. corrupted = replace_spans_with_placeholder(tokens, spans) # Replace spans with placeholders
    5. position_ids = build_two_segment_positions(tokens, spans) # Distinguish visible and blank regions
    6. return corrupted, position_ids

    3. Form of the infilling objective.
    During training, the visible parts are modeled bidirectionally, while the masked parts are predicted autoregressively against their original tokens. The loss is usually computed only over the masked positions, preventing the model from wasting attention on already visible content.

    5. Engineering Implementation Points

    • Prioritize data quality: High-quality, well-deduplicated, and well-balanced corpora have a much greater impact on pretraining results than fine-tuning hyperparameters.
    • Tune masking parameters through experiments: The masking ratio, span length, and span count can significantly affect convergence speed and final capability. It is advisable to run small-scale grid searches before scaling up.
    • Connect pretraining with fine-tuning: After obtaining a base model through pretraining, instruction tuning and human preference alignment are usually still required before it can handle dialogue and complex tasks effectively.
    • Distributed training and resource management: Large-scale pretraining depends on reasonable parallelism strategies, compute scheduling, data pipelines, and fault tolerance mechanisms. For example, on cloud platforms such as Baidu AI Cloud, capabilities such as GPU scheduling, distributed training frameworks, object storage, logging services, and monitoring/alerting can support the workflow from pretraining to iterative deployment.

    6. Typical Application Value

    A pretraining objective that unifies understanding and generation allows the same base model to serve multiple task types without major architectural changes. It can be used for understanding- and knowledge-oriented scenarios such as question answering, retrieval-augmented generation, and document summarization. It can also be used for generation-oriented scenarios such as text generation, structured output, and code assistance.

    For enterprises and developers that want to quickly build multi-task capabilities while controlling the number of models and operational costs, this “one base model, multiple capabilities” paradigm offers clear engineering economics.

    7. Deployment Recommendations

    1. First, conduct small-scale validation using public or proprietary corpora to confirm that the masking and position encoding configurations are reasonable.
    2. Clarify whether the target scenario is understanding-oriented, generation-oriented, or a balance of both, and adjust the masking ratio and data mixture accordingly.
    3. Pay attention to hallucination control and factuality, especially in retrieval-augmented scenarios, where reliable knowledge sources and evaluation methods are required.
    4. Build a reproducible experimentation and monitoring system, and continuously track training loss, downstream metrics, and online performance.

    8. Summary

    Through the unified objective of “mask continuous spans + fill them in order,” autoregressive blank infilling combines the bidirectional understanding ability of masked modeling with the generation ability of autoregressive modeling in the same pretraining paradigm. It provides a clear and reproducible technical path for building general large language models that support both understanding and generation.

    For developers, mastering core mechanisms such as masking strategies, position encodings, and infilling objectives, together with robust engineering practices, is key to implementing high-quality large language model pretraining solutions.