Autoregressive Blank Infilling Pretraining: Masks, Positions, and a Reproducible Engineering Loop
This article explains autoregressive blank infilling pretraining, an objective that combines bidirectional understanding of the visible context with sequential, in-order generation of masked spans. It walks through a reproducible engineering loop: a unified token id interface, per-span start tokens with right-shifted teacher-forcing inputs, 2D position encoding that separates original position from intra-span order, a span-blocked boolean attention mask, and loss alignment. It also covers an inference decoding strategy consistent with the training isolation, key parameters such as mask ratio and span length, context-length and cost mitigation, an evaluation loop with explicit metrics, and a checklist of common mistakes.}
Introduction: The Pretraining Objective Shapes the Whole Training Loop
In LLM engineering, model capability is determined not only by parameter count but also by the pretraining objective. Autoregressive language modeling excels at left-to-right generation, while autoencoding masked modeling excels at bidirectional understanding. Autoregressive Blank Infilling tries to put both “understanding context” and “generating content in order” into the same pretraining objective: the model must read the visible context and, within the masked spans, generate content in sequence.
But autoregressive blank infilling is not simply a matter of replacing [MASK] with a generation task. It involves four engineering problems that must be aligned simultaneously: how to sample masked spans, how to construct training inputs, how to isolate attention masks, and how to align the loss with positions. If these details are inconsistent, the model may suffer cross-span leakage, positional misalignment, or a mismatch between training and inference behavior.
This article presents a reproducible engineering loop: a unified token id interface, per-span start tokens, teacher-forcing inputs, 2D position encoding, a span-blocked boolean attention mask, and a span isolation strategy at inference that stays consistent with training.
1. Paradigm Choice: Why “Intra-Span Autoregression”
Autoregressive language modeling has the form:
$
P(x) = \prod{t=1}^{T} P(x_t \mid x{<t})
$
It predicts the next token from left to right; during training the loss can be computed in parallel via teacher forcing, and during inference it typically decodes sequentially. Autoencoding masked modeling randomly masks tokens and reconstructs them using bidirectional context. It has strong understanding ability, but its training objective is not an autoregressive decomposition, so using it directly for open-ended generation can create a pretraining-inference mismatch.
The key difference in autoregressive blank infilling is that the visible region can be understood bidirectionally, the masked region is generated in autoregressive order internally, and spans are isolated by default. That is, one span cannot peek at another span’s generation results unless cross-span mode is explicitly enabled.
| Dimension | AR (autoregressive) | AE (autoencoding) | Prefix LM | Span Corruption | Autoregressive Blank Infilling |
|---|---|---|---|---|---|
| Modeling objective | Predict the next token left to right | Bidirectionally reconstruct masked tokens | Bidirectional prefix, causal generation for suffix | Generate spans separated by sentinels | Bidirectional visible region, intra-span autoregressive reconstruction |
| Attention structure | Causal | Bidirectional | Bidirectional prefix, causal suffix | Causal over the whole target region | Bidirectional visible region, intra-span causal |
| Span isolation | None | None | Single suffix | Target region can cross spans | Span-blocked isolation by default |
| Position encoding | 1D | 1D | 1D | 1D or sentinel | 2D positions more natural |
| Inference method | Sequential generation | Usually no direct generation | Prefix continuation | Generate spans sequentially | Fill spans back into their original positions |
So the essence of autoregressive blank infilling is not the “infilling” itself, but how to maintain a strict boundary between bidirectional understanding of the visible region and sequential completion in the generation region.
2. Constructing Training Samples: Unified token id, Span Sampling, and Teacher Forcing
A reproducible training sample must pin down five things at once: the token interface, span sampling, training inputs, 2D positions, and the attention mask with loss alignment.
2.1 Unified token id Interface
To avoid mixing strings and integers, training samples should use token ids uniformly. [START] should not be mixed directly into the sequence as a string but should be registered as a special token with a fixed integer id.
START_TOKEN = "[START]"START_ID = 100000 # should stay consistent with the special token id registered in the tokenizer
The recommended model input is unified as:
input_ids: List[int]position_ids: List[Tuple[int, int]] # 2D position: original position + intra-span orderattention_mask: torch.Tensor # boolean mask, True means attendablelabels: List[int] # same length as input_ids, -100 in the visible region
This way training, inference, and evaluation can all revolve around the same interface.
2.2 Span Sampling
During training, contiguous spans are randomly sampled by token length as masking intervals. The cumulative mask ratio is commonly around 10%–30%. The following is a teaching-simplified version: it samples approximately fixed-length spans and controls the total via cur_len = min(span_len, target - masked) so it does not significantly exceed the target ratio.
import randomdef sample_spans(n_tokens, mask_ratio=0.15, span_len=4, min_gap=2):"""Teaching-version span sampling: cumulative mask ratio is roughly mask_ratio."""target = int(n_tokens * mask_ratio)spans = []masked = 0i = 0while i < n_tokens and masked < target:cur_len = min(span_len, target - masked)if cur_len >= 1 and i + cur_len <= n_tokens and random.random() < 0.35:s = ie = i + cur_len - 1spans.append((s, e))masked += cur_leni = e + 1 + min_gapelse:i += 1return spans
In production, you can use a Poisson distribution, a geometric distribution, or variable-length spans sampled by task unit. This example mainly illustrates the training loop and does not aim to cover all sampling strategies.
2.3 Key Correction: Each Span Uses Its Own Start Token
Autoregressive blank infilling cannot concatenate all masked spans into one continuous generation sequence and place a single [START] at the very front. Otherwise later spans would inherit the last token of the preceding span as their initial input, breaking the “no cross-contamination” semantics.
The correct approach is that every span has its own start token, and each span’s input is “start token + that span’s real tokens shifted right by one”.
def build_span_info(n_tokens, spans):span_id = [-1] * n_tokensspan_rank = [0] * n_tokensfor s, e in spans:for rank, pos in enumerate(range(s, e + 1), start=1):span_id[pos] = sspan_rank[pos] = rankreturn span_id, span_rankdef build_infilling_samples(token_ids, spans, start_id=START_ID):"""token_ids: List[int]spans: List[Tuple[int, int]], closed intervals, 0-basedreturns: input_ids, position_ids, labels, span_lens"""n = len(token_ids)spans = sorted(spans, key=lambda x: x[0])span_id, _ = build_span_info(n, spans)visible_ids = []visible_pos = []input_ids = []position_ids = []labels = []span_lens = []for i, tid in enumerate(token_ids):if span_id[i] == -1:visible_ids.append(tid)visible_pos.append((i, 0))# visible region enters the sequence firstinput_ids.extend(visible_ids)position_ids.extend(visible_pos)labels.extend([-100] * len(visible_ids))# each span is built independently: START + shifted right by onefor s, e in spans:span_tokens = [token_ids[i] for i in range(s, e + 1)]span_len = len(span_tokens)span_lens.append(span_len)span_input = [start_id] + span_tokens[:-1]span_pos = [(s + k, k + 1) for k in range(span_len)]input_ids.extend(span_input)position_ids.extend(span_pos)labels.extend(span_tokens)return input_ids, position_ids, labels, span_lens
The core here is that labels and input_ids are already explicitly aligned. The labels[i] at the i-th input position indicates the token that position needs to predict; no additional left shift is applied.
For example, if a span’s real tokens are [m0, m1, m2], then the span input is:
[START, m0, m1]
and the corresponding labels are:
[m0, m1, m2]
That is:
- the
STARTposition predictsm0 - the
m0position predictsm1 - the
m1position predictsm2
This preserves teacher forcing while avoiding cross-span seeding.
2.4 2D Position Encoding
In autoregressive blank infilling, positional information carries at least two meanings:
- Original sequence position: where the token is in the original text;
- Intra-span generation order: which generated target the token is within the current span.
So engineering practice commonly uses 2D positions:
- 1st dimension: original position, or the span’s position in the original text;
- 2nd dimension: intra-span generation order;
- 2nd dimension of the visible region is always 0;
- 2nd dimension of the generation region starts at 1 and increments.
For example, given the original tokens:
Index: 0 1 2 3 4 5 6 7 8Token: t0 t1 t2 t3 t4 t5 t6 t7 t8
With masked spans (4, 6) and (7, 8), the 2D positions of the visible and generation regions can be represented as:
| Region | Token | 2D Position |
|---|---|---|
| Visible | t0 | (0, 0) |
| Visible | t1 | (1, 0) |
| Visible | t2 | (2, 0) |
| Visible | t3 | (3, 0) |
| Generation | t4 | (4, 1) |
| Generation | t5 | (5, 2) |
| Generation | t6 | (6, 3) |
| Generation | t7 | (7, 1) |
| Generation | t8 | (8, 2) |
Under the setting of “bidirectional visible region + sequential generation region”, a single dimension of position is not natural enough, and 2D positions are more common in engineering. If the underlying model only supports 1D positions, 2D positions can be mapped to a 1D index through a position adapter layer or a learnable embedding, but training and inference must use the same mapping rule.
2.5 Attention Mask: Span-Blocked by Default
The default strategy is span isolation:
- the visible region is bidirectionally visible internally;
- the generation region can see all of the visible region;
- each generation span is causal internally;
- different generation spans cannot see each other.
In a boolean mask, True means visible or attendable. If some frameworks use additive masks, they typically use -inf to mean not visible, which is the opposite semantic direction and needs special attention.
import torchdef build_attention_mask(n_visible, n_span, span_lens, device="cpu"):"""Boolean mask: True means visible/attendable.Generation region is span-blocked and causal, not across spans."""total = n_visible + n_spanmask = torch.zeros(total, total, dtype=torch.bool, device=device)# visible region is bidirectionally visible internallymask[:n_visible, :n_visible] = True# generation region can see all of the visible regionmask[n_visible:, :n_visible] = Trueoffset = 0for length in span_lens:s = n_visible + offsete = s + length# each span is causal internallymask[s:e, s:e] = torch.tril(torch.ones(length, length, dtype=torch.bool, device=device))offset += lengthreturn mask
If cross-span causality is truly needed, you can explicitly treat all spans as one large span, or modify the mask so that later spans can see earlier ones. In this case training and inference must be changed together; you cannot loosen it only temporarily at inference.
2.6 Loss Alignment
All visible-region positions are set to -100 and do not participate in the loss. Gradients come only from the generation region.
import torch.nn.functional as F# logits: [batch, seq_len, vocab]# labels: [batch, seq_len], -100 in the visible regionloss = F.cross_entropy(logits.reshape(-1, logits.size(-1)),torch.tensor(labels, device=logits.device).reshape(-1),ignore_index=-100,)
If the framework you use performs shifting automatically—for example, some CausalLM interfaces internally align logits[..., :-1] with labels[..., 1:]—then do not shift right by hand or align again. Explicit alignment and framework auto-shift are mutually exclusive; you can only choose one.
2.7 Minimal Numerical Walkthrough
Suppose the original token ids are:
token_ids = [10, 11, 12, 13]spans = [(1, 2)] # mask 11 and 12
Then:
- visible region:
10,13 - masked tokens:
11,12 - the span input:
[START_ID, 11] - the span labels:
[11, 12] - the span’s 2D positions:
[(1, 1), (2, 2)]
The final training input is:
input_ids = [10, 13, START_ID, 11]position_ids = [(0, 0), (3, 0), (1, 1), (2, 2)]labels = [-100, -100, 11, 12]
where:
- input position 2 is
START_ID, supervised target11 - input position 3 is
11, supervised target12
This avoids a common off-by-one error.
3. Inference Decoding: Consistent with the Training Isolation Strategy by Default
At inference, you cannot use the cross-span all-visible approach of “one START plus all generated tokens”, or it would be inconsistent with the default span-blocked training.
The recommended approach: each span is generated independently, only allowing the current span to see its own START and its generated prefix, while still seeing all of the visible region. Different spans do not depend on each other by default.
import torch@torch.no_grad()def autoregressive_infill(model, token_ids, spans, start_id=START_ID, device="cpu"):result = list(token_ids)n = len(token_ids)span_id, _ = build_span_info(n, spans)visible_ids = []visible_pos = []for i, tid in enumerate(token_ids):if span_id[i] == -1:visible_ids.append(tid)visible_pos.append((i, 0))for s, e in spans:generated = []span_len = e - s + 1for step in range(span_len):# the current span always starts independently with STARTspan_input = [start_id] + generated# each input position corresponds to the target position it predictsspan_pos = [(s + k, k + 1) for k in range(step + 1)]input_ids = visible_ids + span_inputposition_ids = visible_pos + span_pos# visible region + current span prefix; different spans are invisible to each otherattention_mask = build_attention_mask(len(visible_ids),step + 1,[step + 1],device=device,)outputs = model(input_ids=torch.tensor([input_ids], device=device),position_ids=torch.tensor([position_ids], device=device),attention_mask=attention_mask.unsqueeze(0),)logits = outputs.logits[0, -1] if hasattr(outputs, "logits") else outputs[0, -1]token_id = int(logits.argmax(-1))generated.append(token_id)result[s + step] = token_idreturn result
In production, cache the Key/Value of the visible region to avoid recomputing the visible context at every step. When handling multiple spans, the visible-region cache can be reused, but you must ensure the current span cannot see other spans through the mask or explicit context.
If the business genuinely needs cross-span completion—for example, contiguously masking several adjacent spans and wanting the later span to condition on the earlier one—you should explicitly merge them into one large span, or enable cross-span mode, and update the training mask, position encoding, and inference logic in sync.
4. Key Engineering Parameters and Cost Mitigation
4.1 Mask Ratio (mask_ratio)
The mask ratio is commonly around 10%–30%.
- For understanding-type tasks such as entity extraction and relation judgment, choose the lower end;
- For generation or completion-type tasks such as code completion and summarization continuation, choose the higher end;
- If too much is masked, there is insufficient visible context and the model may be forced to guess;
- If too little is masked, the generation targets are undertrained.
It is recommended to evaluate at least three mask ratios—10%, 15%, 30%—and record performance for short, medium, and long spans separately.
4.2 Span Length
Span length should match the task unit:
- code repair and sentence-level completion suit longer spans;
- phrase correction and entity completion suit shorter spans;
- if spans are too long, intra-span autoregressive difficulty rises;
- if spans are too short, the model struggles to learn cross-token generation dependencies.
4.3 Context Length and Inference Cost
Long context can improve document-level understanding but brings pressure on memory, throughput, and first-token latency. Common mitigation approaches include:
- caching and reusing the visible-region KV;
- chunking long inputs or using sliding-window attention;
- using quantization to reduce memory footprint;
- tuning batch size and concurrency strategy;
- setting separate thresholds for first-token latency and P95 latency;
- making the hardware, batch size, input length, and other parameters explicit during evaluation.
Avoid only pursuing “it can train” while ignoring “it can’t run” or “latency is unacceptable”.
5. Evaluation Loop and a Checklist of Common Mistakes
Before launch, establish an evaluation loop. Each metric needs to specify its measurement definition, sample size, evaluation-set construction, and calibration boundaries; avoid giving numbers without conditions.
| Metric | Measurement definition | Sample size / evaluation-set construction | Example threshold | Calibration boundary |
|---|---|---|---|---|
| Span filling accuracy | Match rate between the generated and original masked spans, split into token-level or span-level exact match | At least 500 per mask_ratio level, covering short, medium, and long spans | ≥ 90% | Entities and code require exact match; summaries can relax to semantic match |
| Hallucination rate | Fraction of generated content not supported by visible context or external knowledge | At least 300, dual-annotated with inter-annotator agreement computed | ≤ 5% | High-risk scenarios should be lower; threshold adjusts with business tolerance |
| P95 latency | 95th-percentile time from server-side request to complete response | Fixed hardware, batch size, and input length | ≤ 2 seconds | Long context or large models need relaxation |
| Manual review rate | Fraction of outputs requiring human intervention | Full online sampling statistics | ≤ 5% | Tied to the business risk level |
Common mistakes checklist:
- Cross-span leakage: no isolation between spans, so later spans see the generation results of earlier spans.
- Concatenating multiple spans with a single start token: all spans share one
[START], causing later spans to inherit the last token of the preceding span. - 2D position misalignment: the 2nd dimension of the generation region does not increment from 1, or the 2nd dimension of the visible region is not set to 0.
- Training inputs not shifted right for teacher forcing: the generation-region input is still the real token itself or all
[MASK], inconsistent with autoregressive training semantics. - Wrong loss position: computing loss on the visible region by mistake, or stacking framework auto-shift on top of manual alignment, causing off-by-one.
- Boolean mask direction reversed: in a boolean mask
Truemeans attendable, while in an additive mask-infmeans not attendable; the two directions are opposite.
Conclusion
The key to autoregressive blank infilling pretraining is organizing “bidirectional understanding of the visible region” and “sequential completion of the generation region” under a unified objective. What truly determines engineering success is not any single module but whether the following links are consistent:
- is span sampling stable and controllable;
- does each span use its own start token;
- is the teacher-forcing input correctly shifted right;
- does 2D position encoding distinguish original position from intra-span order;
- is the attention mask span-blocked and isolated by default;
- is the loss strictly aligned with the input positions;
- does inference use the same isolation strategy as training.
Combining these mechanisms with mask ratio, span length, context length, cost control, and an evaluation loop with explicit definitions yields a reproducible, debuggable, and deployable autoregressive blank infilling pretraining engineering solution.
