Autoregressive Blank Infilling: Principles, Architecture, and Engineering Practices

    Autoregressive blank infilling combines bidirectional context understanding with autoregressive generation in a unified pretraining objective. This article explains its core workflow, comparison with conventional language-modeling paradigms, two-dimensional positional encoding, attention-mask design, span-masking strategies, loss calculation, tuning considerations, and representative application scenarios.

    1. Why a New Pretraining Paradigm Is Needed

    Pretrained language models generally use one of two modeling objectives, each with distinct trade-offs:

    • Unidirectional autoregressive modeling (next-token prediction): The model predicts the next token one at a time. This provides strong generation capabilities and good contextual consistency, but each position can only access information on its left. As a result, it has an inherent disadvantage in tasks that require bidirectional semantic understanding, such as classification, extraction, and retrieval.
    • Bidirectional masked language modeling (MLM): The model randomly masks some tokens and reconstructs them using the full context. This makes better use of bidirectional information, but predictions at different masked positions are conditionally independent. This limits direct support for high-quality free-form generation and creates a mismatch between pretraining and fine-tuning objectives.

    In practice, many applications require both strong understanding and strong generation. The motivation behind autoregressive blank infilling is to cover both capabilities within a unified pretraining objective, reducing reliance on multi-stage pipelines that combine different objectives and architectures.

    2. The Core Idea of Autoregressive Blank Infilling

    Autoregressive blank infilling combines span completion with autoregressive generation. The process can be summarized in three steps.

    2.1 Span Masking

    Instead of masking isolated random tokens, the model masks several contiguous spans sampled according to a predefined length distribution. Contiguous spans are closer to real phrases, clauses, and sentence boundaries. They encourage the model to learn more complete semantic structures rather than relying only on local token co-occurrence.

    2.2 Sequence Rearrangement

    The original text is divided into two parts:

    • Part A: The visible, unmasked context
    • Part B: The masked content that must be reconstructed

    The model is trained to restore Part B in an autoregressive manner.

    2.3 Autoregressive Reconstruction

    Within Part B, prediction follows a causal order. When predicting the i-th masked token, the model can only use information that precedes it in the reconstruction sequence. Part A, however, remains fully visible in both directions.

    In short, the paradigm enables bidirectional understanding of visible content and sequential reconstruction of masked content. It preserves the bidirectional understanding benefits of masked language modeling while retaining the generation capabilities of autoregressive modeling.

    3. Comparison with Conventional Paradigms

    Dimension Unidirectional Autoregressive Modeling Bidirectional Masked Modeling Autoregressive Blank Infilling
    Prediction target The next token A masked token A masked contiguous span
    Context direction Unidirectional Bidirectional Bidirectional in the visible region and unidirectional in the generation region
    Generation capability Strong Limited Strong
    Understanding capability Moderate Strong Strong
    Pretraining/fine-tuning alignment High Relatively low Relatively high

    No single paradigm is optimal for every task. The value of autoregressive blank infilling lies in using one objective to cover a broader range of tasks, thereby reducing the cost of switching between architectures and maintaining multiple training pipelines.

    However, it is not a silver bullet. Its actual benefits still depend on the data, hyperparameters, and distribution of downstream tasks.

    4. Key Architectural Design

    4.1 Two-Dimensional Positional Encoding

    Because the sequence is rearranged into Part A followed by Part B, a single position index cannot simultaneously represent the original token order and the relationship between visible and generated segments.

    A common solution is to assign each token a two-dimensional positional representation:

    • First dimension: The token’s original position in the text, used to model token order and distance.
    • Second dimension: The token’s position or role in the rearranged sequence, used to distinguish visible context from the reconstructed span.

    Although Part B is physically moved to the end of the sequence, the first position dimension allows the model to determine where each token originally appeared. This helps the model represent long-range dependencies correctly.

    4.2 Attention-Mask Design

    The attention mask controls information flow and is central to combining understanding and generation:

    • Tokens in Part A use bidirectional attention within Part A.
    • The i-th token in Part B can attend only to preceding tokens in Part B, enforcing a causal constraint.
    • Part B can attend to all tokens in Part A.
    • Part A must not attend to Part B, preventing information leakage.

    A typical implementation is shown below:

    1. def build_attention_mask(seq_len, part_b_start):
    2. """
    3. seq_len: Total length of the rearranged sequence
    4. part_b_start: Starting index of Part B in the rearranged sequence
    5. Returns a Boolean mask of shape [seq_len, seq_len].
    6. True indicates that the position is visible.
    7. """
    8. mask = torch.zeros(seq_len, seq_len, dtype=torch.bool)
    9. # 1) Bidirectional visibility within Part A
    10. mask[:part_b_start, :part_b_start] = True
    11. # 2) Part B can attend to all of Part A
    12. mask[part_b_start:, :part_b_start] = True
    13. # 3) Causal visibility within Part B, including the diagonal
    14. n_b = seq_len - part_b_start
    15. causal = torch.tril(torch.ones(n_b, n_b, dtype=torch.bool))
    16. mask[part_b_start:, part_b_start:] = causal
    17. # 4) Part A cannot attend to Part B, preventing information leakage
    18. mask[:part_b_start, part_b_start:] = False
    19. return mask

    The fourth rule is particularly important. The visible context must never “peek” at the masked content. Otherwise, the answer is leaked during training and the learning objective becomes invalid.

    5. Masking Strategy and Data Construction

    The length distribution of masked spans directly affects task difficulty and training-signal density. In practice, span lengths are often sampled from a geometric or Poisson distribution, while an upper bound is placed on the overall masking ratio.

    Masking too much content can damage the integrity of the context. Masking too little can make the training signal too sparse. A simplified example is shown below:

    1. def span_mask(tokens, mask_ratio=0.15, mean_span=3, mask_id=0):
    2. n = len(tokens)
    3. n_mask = max(1, int(n * mask_ratio))
    4. spans, covered = [], 0
    5. while covered < n_mask:
    6. # Sample the current span length from a geometric distribution
    7. span_len = int(np.random.geometric(1.0 / mean_span))
    8. start = np.random.randint(0, n - span_len)
    9. spans.append((start, start + span_len))
    10. covered += span_len
    11. # Merge overlapping spans and construct the mask tensor
    12. masked = tokens.copy()
    13. for s, e in spans:
    14. masked[s:e] = mask_id
    15. return masked, spans

    In production implementations, overlapping spans are usually deduplicated and merged to prevent the actual masking ratio from deviating from the intended value.

    6. Training Objective and Loss

    The loss is computed only for Part B, the masked content being reconstructed. Part A does not contribute to the reconstruction objective.

    The loss can be implemented as a joint cross-entropy loss over the entire reconstructed span. Alternatively, different positions can be assigned different weights to emphasize span-level coherence:

    1. # Compute the language-modeling loss only at masked positions
    2. logits = model(masked_input, position_ids_2d, attn_mask)
    3. loss = cross_entropy(
    4. logits[part_b_positions], # Predicted distributions
    5. targets[part_b_positions], # Ground-truth masked tokens
    6. ignore_index=-100,
    7. )

    Because Part B uses causal modeling, the loss naturally supports teacher forcing. Training efficiency is therefore comparable to that of unidirectional autoregressive modeling.

    7. Engineering Practices and Tuning Recommendations

    1. The masking ratio is the primary hyperparameter.
      A ratio that is too high leaves insufficient context, while a ratio that is too low provides too little training signal. A practical starting point is around 15%, followed by adjustment based on validation loss.

    2. The span-length distribution affects downstream performance.
      Short spans emphasize local modeling, while longer spans are closer to sentence-level understanding and generation. The sampling distribution can be adjusted according to the target task, such as generation or understanding.

    3. The positional-encoding dimensions must match the context length.
      Two-dimensional position indices should cover the expected maximum sequence length. Long-context applications require sufficient room for future expansion.

    4. Watch for information leakage.
      Always verify that the attention mask does not accidentally allow Part A to see Part B.

    5. Align the pretraining and fine-tuning objectives.
      If the primary downstream use case is conversational generation, instruction-style samples can be mixed into continued pretraining to reduce the format gap between pretraining and inference.

    8. Typical Application Scenarios

    This paradigm is naturally suited to tasks that require local generation combined with global understanding, including:

    • Long-text completion and summarization: Span reconstruction can transfer directly to infilling and continuation tasks.
    • Information extraction and structured generation: The model can understand the surrounding context bidirectionally and then generate structured fields sequentially.
    • Retrieval-augmented generation and question answering: Multiple retrieved context segments can be encoded bidirectionally while generation remains coherent.
    • Code generation: Span masking can cover continuous structures such as function bodies and statement blocks.

    9. Summary

    The essence of autoregressive blank infilling is to unify understanding and generation under a single pretraining objective. Contiguous span masking strengthens holistic semantic modeling, autoregressive reconstruction preserves generation capabilities, and carefully designed two-dimensional positional encoding and attention masks address the positional and information-flow challenges introduced by sequence rearrangement.

    For developers, understanding the objective function, mask construction, and positional-encoding design is sufficient to make informed trade-offs when building a custom pretraining pipeline. The resulting foundation model can support both language understanding and generation, but its actual performance still depends on systematic tuning of the masking strategy, sequence length, and downstream task distribution.