Autoregressive Blank Infilling: Principles, Architecture, and Engineering Practices
Autoregressive blank infilling combines bidirectional context understanding with autoregressive generation in a unified pretraining objective. This article explains its core workflow, comparison with conventional language-modeling paradigms, two-dimensional positional encoding, attention-mask design, span-masking strategies, loss calculation, tuning considerations, and representative application scenarios.
1. Why a New Pretraining Paradigm Is Needed
Pretrained language models generally use one of two modeling objectives, each with distinct trade-offs:
- Unidirectional autoregressive modeling (next-token prediction): The model predicts the next token one at a time. This provides strong generation capabilities and good contextual consistency, but each position can only access information on its left. As a result, it has an inherent disadvantage in tasks that require bidirectional semantic understanding, such as classification, extraction, and retrieval.
- Bidirectional masked language modeling (MLM): The model randomly masks some tokens and reconstructs them using the full context. This makes better use of bidirectional information, but predictions at different masked positions are conditionally independent. This limits direct support for high-quality free-form generation and creates a mismatch between pretraining and fine-tuning objectives.
In practice, many applications require both strong understanding and strong generation. The motivation behind autoregressive blank infilling is to cover both capabilities within a unified pretraining objective, reducing reliance on multi-stage pipelines that combine different objectives and architectures.
2. The Core Idea of Autoregressive Blank Infilling
Autoregressive blank infilling combines span completion with autoregressive generation. The process can be summarized in three steps.
2.1 Span Masking
Instead of masking isolated random tokens, the model masks several contiguous spans sampled according to a predefined length distribution. Contiguous spans are closer to real phrases, clauses, and sentence boundaries. They encourage the model to learn more complete semantic structures rather than relying only on local token co-occurrence.
2.2 Sequence Rearrangement
The original text is divided into two parts:
- Part A: The visible, unmasked context
- Part B: The masked content that must be reconstructed
The model is trained to restore Part B in an autoregressive manner.
2.3 Autoregressive Reconstruction
Within Part B, prediction follows a causal order. When predicting the i-th masked token, the model can only use information that precedes it in the reconstruction sequence. Part A, however, remains fully visible in both directions.
In short, the paradigm enables bidirectional understanding of visible content and sequential reconstruction of masked content. It preserves the bidirectional understanding benefits of masked language modeling while retaining the generation capabilities of autoregressive modeling.
3. Comparison with Conventional Paradigms
| Dimension | Unidirectional Autoregressive Modeling | Bidirectional Masked Modeling | Autoregressive Blank Infilling |
|---|---|---|---|
| Prediction target | The next token | A masked token | A masked contiguous span |
| Context direction | Unidirectional | Bidirectional | Bidirectional in the visible region and unidirectional in the generation region |
| Generation capability | Strong | Limited | Strong |
| Understanding capability | Moderate | Strong | Strong |
| Pretraining/fine-tuning alignment | High | Relatively low | Relatively high |
No single paradigm is optimal for every task. The value of autoregressive blank infilling lies in using one objective to cover a broader range of tasks, thereby reducing the cost of switching between architectures and maintaining multiple training pipelines.
However, it is not a silver bullet. Its actual benefits still depend on the data, hyperparameters, and distribution of downstream tasks.
4. Key Architectural Design
4.1 Two-Dimensional Positional Encoding
Because the sequence is rearranged into Part A followed by Part B, a single position index cannot simultaneously represent the original token order and the relationship between visible and generated segments.
A common solution is to assign each token a two-dimensional positional representation:
- First dimension: The token’s original position in the text, used to model token order and distance.
- Second dimension: The token’s position or role in the rearranged sequence, used to distinguish visible context from the reconstructed span.
Although Part B is physically moved to the end of the sequence, the first position dimension allows the model to determine where each token originally appeared. This helps the model represent long-range dependencies correctly.
4.2 Attention-Mask Design
The attention mask controls information flow and is central to combining understanding and generation:
- Tokens in Part A use bidirectional attention within Part A.
- The i-th token in Part B can attend only to preceding tokens in Part B, enforcing a causal constraint.
- Part B can attend to all tokens in Part A.
- Part A must not attend to Part B, preventing information leakage.
A typical implementation is shown below:
def build_attention_mask(seq_len, part_b_start):"""seq_len: Total length of the rearranged sequencepart_b_start: Starting index of Part B in the rearranged sequenceReturns a Boolean mask of shape [seq_len, seq_len].True indicates that the position is visible."""mask = torch.zeros(seq_len, seq_len, dtype=torch.bool)# 1) Bidirectional visibility within Part Amask[:part_b_start, :part_b_start] = True# 2) Part B can attend to all of Part Amask[part_b_start:, :part_b_start] = True# 3) Causal visibility within Part B, including the diagonaln_b = seq_len - part_b_startcausal = torch.tril(torch.ones(n_b, n_b, dtype=torch.bool))mask[part_b_start:, part_b_start:] = causal# 4) Part A cannot attend to Part B, preventing information leakagemask[:part_b_start, part_b_start:] = Falsereturn mask
The fourth rule is particularly important. The visible context must never “peek” at the masked content. Otherwise, the answer is leaked during training and the learning objective becomes invalid.
5. Masking Strategy and Data Construction
The length distribution of masked spans directly affects task difficulty and training-signal density. In practice, span lengths are often sampled from a geometric or Poisson distribution, while an upper bound is placed on the overall masking ratio.
Masking too much content can damage the integrity of the context. Masking too little can make the training signal too sparse. A simplified example is shown below:
def span_mask(tokens, mask_ratio=0.15, mean_span=3, mask_id=0):n = len(tokens)n_mask = max(1, int(n * mask_ratio))spans, covered = [], 0while covered < n_mask:# Sample the current span length from a geometric distributionspan_len = int(np.random.geometric(1.0 / mean_span))start = np.random.randint(0, n - span_len)spans.append((start, start + span_len))covered += span_len# Merge overlapping spans and construct the mask tensormasked = tokens.copy()for s, e in spans:masked[s:e] = mask_idreturn masked, spans
In production implementations, overlapping spans are usually deduplicated and merged to prevent the actual masking ratio from deviating from the intended value.
6. Training Objective and Loss
The loss is computed only for Part B, the masked content being reconstructed. Part A does not contribute to the reconstruction objective.
The loss can be implemented as a joint cross-entropy loss over the entire reconstructed span. Alternatively, different positions can be assigned different weights to emphasize span-level coherence:
# Compute the language-modeling loss only at masked positionslogits = model(masked_input, position_ids_2d, attn_mask)loss = cross_entropy(logits[part_b_positions], # Predicted distributionstargets[part_b_positions], # Ground-truth masked tokensignore_index=-100,)
Because Part B uses causal modeling, the loss naturally supports teacher forcing. Training efficiency is therefore comparable to that of unidirectional autoregressive modeling.
7. Engineering Practices and Tuning Recommendations
The masking ratio is the primary hyperparameter.
A ratio that is too high leaves insufficient context, while a ratio that is too low provides too little training signal. A practical starting point is around 15%, followed by adjustment based on validation loss.The span-length distribution affects downstream performance.
Short spans emphasize local modeling, while longer spans are closer to sentence-level understanding and generation. The sampling distribution can be adjusted according to the target task, such as generation or understanding.The positional-encoding dimensions must match the context length.
Two-dimensional position indices should cover the expected maximum sequence length. Long-context applications require sufficient room for future expansion.Watch for information leakage.
Always verify that the attention mask does not accidentally allow Part A to see Part B.Align the pretraining and fine-tuning objectives.
If the primary downstream use case is conversational generation, instruction-style samples can be mixed into continued pretraining to reduce the format gap between pretraining and inference.
8. Typical Application Scenarios
This paradigm is naturally suited to tasks that require local generation combined with global understanding, including:
- Long-text completion and summarization: Span reconstruction can transfer directly to infilling and continuation tasks.
- Information extraction and structured generation: The model can understand the surrounding context bidirectionally and then generate structured fields sequentially.
- Retrieval-augmented generation and question answering: Multiple retrieved context segments can be encoded bidirectionally while generation remains coherent.
- Code generation: Span masking can cover continuous structures such as function bodies and statement blocks.
9. Summary
The essence of autoregressive blank infilling is to unify understanding and generation under a single pretraining objective. Contiguous span masking strengthens holistic semantic modeling, autoregressive reconstruction preserves generation capabilities, and carefully designed two-dimensional positional encoding and attention masks address the positional and information-flow challenges introduced by sequence rearrangement.
For developers, understanding the objective function, mask construction, and positional-encoding design is sufficient to make informed trade-offs when building a custom pretraining pipeline. The resulting foundation model can support both language understanding and generation, but its actual performance still depends on systematic tuning of the masking strategy, sequence length, and downstream task distribution.
