Autoregressive Blank Infilling Pretraining: Masks, Positions, and a Reproducible Engineering Loop

    This article explains autoregressive blank infilling pretraining, an objective that combines bidirectional understanding of the visible context with sequential, in-order generation of masked spans. It walks through a reproducible engineering loop: a unified token id interface, per-span start tokens with right-shifted teacher-forcing inputs, 2D position encoding that separates original position from intra-span order, a span-blocked boolean attention mask, and loss alignment. It also covers an inference decoding strategy consistent with the training isolation, key parameters such as mask ratio and span length, context-length and cost mitigation, an evaluation loop with explicit metrics, and a checklist of common mistakes.}

    Introduction: The Pretraining Objective Shapes the Whole Training Loop

    In LLM engineering, model capability is determined not only by parameter count but also by the pretraining objective. Autoregressive language modeling excels at left-to-right generation, while autoencoding masked modeling excels at bidirectional understanding. Autoregressive Blank Infilling tries to put both “understanding context” and “generating content in order” into the same pretraining objective: the model must read the visible context and, within the masked spans, generate content in sequence.

    But autoregressive blank infilling is not simply a matter of replacing [MASK] with a generation task. It involves four engineering problems that must be aligned simultaneously: how to sample masked spans, how to construct training inputs, how to isolate attention masks, and how to align the loss with positions. If these details are inconsistent, the model may suffer cross-span leakage, positional misalignment, or a mismatch between training and inference behavior.

    This article presents a reproducible engineering loop: a unified token id interface, per-span start tokens, teacher-forcing inputs, 2D position encoding, a span-blocked boolean attention mask, and a span isolation strategy at inference that stays consistent with training.

    1. Paradigm Choice: Why “Intra-Span Autoregression”

    Autoregressive language modeling has the form:

    $
    P(x) = \prod{t=1}^{T} P(x_t \mid x{<t})
    $

    It predicts the next token from left to right; during training the loss can be computed in parallel via teacher forcing, and during inference it typically decodes sequentially. Autoencoding masked modeling randomly masks tokens and reconstructs them using bidirectional context. It has strong understanding ability, but its training objective is not an autoregressive decomposition, so using it directly for open-ended generation can create a pretraining-inference mismatch.

    The key difference in autoregressive blank infilling is that the visible region can be understood bidirectionally, the masked region is generated in autoregressive order internally, and spans are isolated by default. That is, one span cannot peek at another span’s generation results unless cross-span mode is explicitly enabled.

    Dimension AR (autoregressive) AE (autoencoding) Prefix LM Span Corruption Autoregressive Blank Infilling
    Modeling objective Predict the next token left to right Bidirectionally reconstruct masked tokens Bidirectional prefix, causal generation for suffix Generate spans separated by sentinels Bidirectional visible region, intra-span autoregressive reconstruction
    Attention structure Causal Bidirectional Bidirectional prefix, causal suffix Causal over the whole target region Bidirectional visible region, intra-span causal
    Span isolation None None Single suffix Target region can cross spans Span-blocked isolation by default
    Position encoding 1D 1D 1D 1D or sentinel 2D positions more natural
    Inference method Sequential generation Usually no direct generation Prefix continuation Generate spans sequentially Fill spans back into their original positions

    So the essence of autoregressive blank infilling is not the “infilling” itself, but how to maintain a strict boundary between bidirectional understanding of the visible region and sequential completion in the generation region.

    2. Constructing Training Samples: Unified token id, Span Sampling, and Teacher Forcing

    A reproducible training sample must pin down five things at once: the token interface, span sampling, training inputs, 2D positions, and the attention mask with loss alignment.

    2.1 Unified token id Interface

    To avoid mixing strings and integers, training samples should use token ids uniformly. [START] should not be mixed directly into the sequence as a string but should be registered as a special token with a fixed integer id.

    1. START_TOKEN = "[START]"
    2. START_ID = 100000 # should stay consistent with the special token id registered in the tokenizer

    The recommended model input is unified as:

    1. input_ids: List[int]
    2. position_ids: List[Tuple[int, int]] # 2D position: original position + intra-span order
    3. attention_mask: torch.Tensor # boolean mask, True means attendable
    4. labels: List[int] # same length as input_ids, -100 in the visible region

    This way training, inference, and evaluation can all revolve around the same interface.

    2.2 Span Sampling

    During training, contiguous spans are randomly sampled by token length as masking intervals. The cumulative mask ratio is commonly around 10%–30%. The following is a teaching-simplified version: it samples approximately fixed-length spans and controls the total via cur_len = min(span_len, target - masked) so it does not significantly exceed the target ratio.

    1. import random
    2. def sample_spans(n_tokens, mask_ratio=0.15, span_len=4, min_gap=2):
    3. """Teaching-version span sampling: cumulative mask ratio is roughly mask_ratio."""
    4. target = int(n_tokens * mask_ratio)
    5. spans = []
    6. masked = 0
    7. i = 0
    8. while i < n_tokens and masked < target:
    9. cur_len = min(span_len, target - masked)
    10. if cur_len >= 1 and i + cur_len <= n_tokens and random.random() < 0.35:
    11. s = i
    12. e = i + cur_len - 1
    13. spans.append((s, e))
    14. masked += cur_len
    15. i = e + 1 + min_gap
    16. else:
    17. i += 1
    18. return spans

    In production, you can use a Poisson distribution, a geometric distribution, or variable-length spans sampled by task unit. This example mainly illustrates the training loop and does not aim to cover all sampling strategies.

    2.3 Key Correction: Each Span Uses Its Own Start Token

    Autoregressive blank infilling cannot concatenate all masked spans into one continuous generation sequence and place a single [START] at the very front. Otherwise later spans would inherit the last token of the preceding span as their initial input, breaking the “no cross-contamination” semantics.

    The correct approach is that every span has its own start token, and each span’s input is “start token + that span’s real tokens shifted right by one”.

    1. def build_span_info(n_tokens, spans):
    2. span_id = [-1] * n_tokens
    3. span_rank = [0] * n_tokens
    4. for s, e in spans:
    5. for rank, pos in enumerate(range(s, e + 1), start=1):
    6. span_id[pos] = s
    7. span_rank[pos] = rank
    8. return span_id, span_rank
    9. def build_infilling_samples(token_ids, spans, start_id=START_ID):
    10. """
    11. token_ids: List[int]
    12. spans: List[Tuple[int, int]], closed intervals, 0-based
    13. returns: input_ids, position_ids, labels, span_lens
    14. """
    15. n = len(token_ids)
    16. spans = sorted(spans, key=lambda x: x[0])
    17. span_id, _ = build_span_info(n, spans)
    18. visible_ids = []
    19. visible_pos = []
    20. input_ids = []
    21. position_ids = []
    22. labels = []
    23. span_lens = []
    24. for i, tid in enumerate(token_ids):
    25. if span_id[i] == -1:
    26. visible_ids.append(tid)
    27. visible_pos.append((i, 0))
    28. # visible region enters the sequence first
    29. input_ids.extend(visible_ids)
    30. position_ids.extend(visible_pos)
    31. labels.extend([-100] * len(visible_ids))
    32. # each span is built independently: START + shifted right by one
    33. for s, e in spans:
    34. span_tokens = [token_ids[i] for i in range(s, e + 1)]
    35. span_len = len(span_tokens)
    36. span_lens.append(span_len)
    37. span_input = [start_id] + span_tokens[:-1]
    38. span_pos = [(s + k, k + 1) for k in range(span_len)]
    39. input_ids.extend(span_input)
    40. position_ids.extend(span_pos)
    41. labels.extend(span_tokens)
    42. return input_ids, position_ids, labels, span_lens

    The core here is that labels and input_ids are already explicitly aligned. The labels[i] at the i-th input position indicates the token that position needs to predict; no additional left shift is applied.

    For example, if a span’s real tokens are [m0, m1, m2], then the span input is:

    1. [START, m0, m1]

    and the corresponding labels are:

    1. [m0, m1, m2]

    That is:

    • the START position predicts m0
    • the m0 position predicts m1
    • the m1 position predicts m2

    This preserves teacher forcing while avoiding cross-span seeding.

    2.4 2D Position Encoding

    In autoregressive blank infilling, positional information carries at least two meanings:

    1. Original sequence position: where the token is in the original text;
    2. Intra-span generation order: which generated target the token is within the current span.

    So engineering practice commonly uses 2D positions:

    • 1st dimension: original position, or the span’s position in the original text;
    • 2nd dimension: intra-span generation order;
    • 2nd dimension of the visible region is always 0;
    • 2nd dimension of the generation region starts at 1 and increments.

    For example, given the original tokens:

    1. Index: 0 1 2 3 4 5 6 7 8
    2. Token: t0 t1 t2 t3 t4 t5 t6 t7 t8

    With masked spans (4, 6) and (7, 8), the 2D positions of the visible and generation regions can be represented as:

    Region Token 2D Position
    Visible t0 (0, 0)
    Visible t1 (1, 0)
    Visible t2 (2, 0)
    Visible t3 (3, 0)
    Generation t4 (4, 1)
    Generation t5 (5, 2)
    Generation t6 (6, 3)
    Generation t7 (7, 1)
    Generation t8 (8, 2)

    Under the setting of “bidirectional visible region + sequential generation region”, a single dimension of position is not natural enough, and 2D positions are more common in engineering. If the underlying model only supports 1D positions, 2D positions can be mapped to a 1D index through a position adapter layer or a learnable embedding, but training and inference must use the same mapping rule.

    2.5 Attention Mask: Span-Blocked by Default

    The default strategy is span isolation:

    • the visible region is bidirectionally visible internally;
    • the generation region can see all of the visible region;
    • each generation span is causal internally;
    • different generation spans cannot see each other.

    In a boolean mask, True means visible or attendable. If some frameworks use additive masks, they typically use -inf to mean not visible, which is the opposite semantic direction and needs special attention.

    1. import torch
    2. def build_attention_mask(n_visible, n_span, span_lens, device="cpu"):
    3. """
    4. Boolean mask: True means visible/attendable.
    5. Generation region is span-blocked and causal, not across spans.
    6. """
    7. total = n_visible + n_span
    8. mask = torch.zeros(total, total, dtype=torch.bool, device=device)
    9. # visible region is bidirectionally visible internally
    10. mask[:n_visible, :n_visible] = True
    11. # generation region can see all of the visible region
    12. mask[n_visible:, :n_visible] = True
    13. offset = 0
    14. for length in span_lens:
    15. s = n_visible + offset
    16. e = s + length
    17. # each span is causal internally
    18. mask[s:e, s:e] = torch.tril(
    19. torch.ones(length, length, dtype=torch.bool, device=device)
    20. )
    21. offset += length
    22. return mask

    If cross-span causality is truly needed, you can explicitly treat all spans as one large span, or modify the mask so that later spans can see earlier ones. In this case training and inference must be changed together; you cannot loosen it only temporarily at inference.

    2.6 Loss Alignment

    All visible-region positions are set to -100 and do not participate in the loss. Gradients come only from the generation region.

    1. import torch.nn.functional as F
    2. # logits: [batch, seq_len, vocab]
    3. # labels: [batch, seq_len], -100 in the visible region
    4. loss = F.cross_entropy(
    5. logits.reshape(-1, logits.size(-1)),
    6. torch.tensor(labels, device=logits.device).reshape(-1),
    7. ignore_index=-100,
    8. )

    If the framework you use performs shifting automatically—for example, some CausalLM interfaces internally align logits[..., :-1] with labels[..., 1:]—then do not shift right by hand or align again. Explicit alignment and framework auto-shift are mutually exclusive; you can only choose one.

    2.7 Minimal Numerical Walkthrough

    Suppose the original token ids are:

    1. token_ids = [10, 11, 12, 13]
    2. spans = [(1, 2)] # mask 11 and 12

    Then:

    • visible region: 10, 13
    • masked tokens: 11, 12
    • the span input: [START_ID, 11]
    • the span labels: [11, 12]
    • the span’s 2D positions: [(1, 1), (2, 2)]

    The final training input is:

    1. input_ids = [10, 13, START_ID, 11]
    2. position_ids = [(0, 0), (3, 0), (1, 1), (2, 2)]
    3. labels = [-100, -100, 11, 12]

    where:

    • input position 2 is START_ID, supervised target 11
    • input position 3 is 11, supervised target 12

    This avoids a common off-by-one error.

    3. Inference Decoding: Consistent with the Training Isolation Strategy by Default

    At inference, you cannot use the cross-span all-visible approach of “one START plus all generated tokens”, or it would be inconsistent with the default span-blocked training.

    The recommended approach: each span is generated independently, only allowing the current span to see its own START and its generated prefix, while still seeing all of the visible region. Different spans do not depend on each other by default.

    1. import torch
    2. @torch.no_grad()
    3. def autoregressive_infill(model, token_ids, spans, start_id=START_ID, device="cpu"):
    4. result = list(token_ids)
    5. n = len(token_ids)
    6. span_id, _ = build_span_info(n, spans)
    7. visible_ids = []
    8. visible_pos = []
    9. for i, tid in enumerate(token_ids):
    10. if span_id[i] == -1:
    11. visible_ids.append(tid)
    12. visible_pos.append((i, 0))
    13. for s, e in spans:
    14. generated = []
    15. span_len = e - s + 1
    16. for step in range(span_len):
    17. # the current span always starts independently with START
    18. span_input = [start_id] + generated
    19. # each input position corresponds to the target position it predicts
    20. span_pos = [(s + k, k + 1) for k in range(step + 1)]
    21. input_ids = visible_ids + span_input
    22. position_ids = visible_pos + span_pos
    23. # visible region + current span prefix; different spans are invisible to each other
    24. attention_mask = build_attention_mask(
    25. len(visible_ids),
    26. step + 1,
    27. [step + 1],
    28. device=device,
    29. )
    30. outputs = model(
    31. input_ids=torch.tensor([input_ids], device=device),
    32. position_ids=torch.tensor([position_ids], device=device),
    33. attention_mask=attention_mask.unsqueeze(0),
    34. )
    35. logits = outputs.logits[0, -1] if hasattr(outputs, "logits") else outputs[0, -1]
    36. token_id = int(logits.argmax(-1))
    37. generated.append(token_id)
    38. result[s + step] = token_id
    39. return result

    In production, cache the Key/Value of the visible region to avoid recomputing the visible context at every step. When handling multiple spans, the visible-region cache can be reused, but you must ensure the current span cannot see other spans through the mask or explicit context.

    If the business genuinely needs cross-span completion—for example, contiguously masking several adjacent spans and wanting the later span to condition on the earlier one—you should explicitly merge them into one large span, or enable cross-span mode, and update the training mask, position encoding, and inference logic in sync.

    4. Key Engineering Parameters and Cost Mitigation

    4.1 Mask Ratio (mask_ratio)

    The mask ratio is commonly around 10%–30%.

    • For understanding-type tasks such as entity extraction and relation judgment, choose the lower end;
    • For generation or completion-type tasks such as code completion and summarization continuation, choose the higher end;
    • If too much is masked, there is insufficient visible context and the model may be forced to guess;
    • If too little is masked, the generation targets are undertrained.

    It is recommended to evaluate at least three mask ratios—10%, 15%, 30%—and record performance for short, medium, and long spans separately.

    4.2 Span Length

    Span length should match the task unit:

    • code repair and sentence-level completion suit longer spans;
    • phrase correction and entity completion suit shorter spans;
    • if spans are too long, intra-span autoregressive difficulty rises;
    • if spans are too short, the model struggles to learn cross-token generation dependencies.

    4.3 Context Length and Inference Cost

    Long context can improve document-level understanding but brings pressure on memory, throughput, and first-token latency. Common mitigation approaches include:

    • caching and reusing the visible-region KV;
    • chunking long inputs or using sliding-window attention;
    • using quantization to reduce memory footprint;
    • tuning batch size and concurrency strategy;
    • setting separate thresholds for first-token latency and P95 latency;
    • making the hardware, batch size, input length, and other parameters explicit during evaluation.

    Avoid only pursuing “it can train” while ignoring “it can’t run” or “latency is unacceptable”.

    5. Evaluation Loop and a Checklist of Common Mistakes

    Before launch, establish an evaluation loop. Each metric needs to specify its measurement definition, sample size, evaluation-set construction, and calibration boundaries; avoid giving numbers without conditions.

    Metric Measurement definition Sample size / evaluation-set construction Example threshold Calibration boundary
    Span filling accuracy Match rate between the generated and original masked spans, split into token-level or span-level exact match At least 500 per mask_ratio level, covering short, medium, and long spans ≥ 90% Entities and code require exact match; summaries can relax to semantic match
    Hallucination rate Fraction of generated content not supported by visible context or external knowledge At least 300, dual-annotated with inter-annotator agreement computed ≤ 5% High-risk scenarios should be lower; threshold adjusts with business tolerance
    P95 latency 95th-percentile time from server-side request to complete response Fixed hardware, batch size, and input length ≤ 2 seconds Long context or large models need relaxation
    Manual review rate Fraction of outputs requiring human intervention Full online sampling statistics ≤ 5% Tied to the business risk level

    Common mistakes checklist:

    1. Cross-span leakage: no isolation between spans, so later spans see the generation results of earlier spans.
    2. Concatenating multiple spans with a single start token: all spans share one [START], causing later spans to inherit the last token of the preceding span.
    3. 2D position misalignment: the 2nd dimension of the generation region does not increment from 1, or the 2nd dimension of the visible region is not set to 0.
    4. Training inputs not shifted right for teacher forcing: the generation-region input is still the real token itself or all [MASK], inconsistent with autoregressive training semantics.
    5. Wrong loss position: computing loss on the visible region by mistake, or stacking framework auto-shift on top of manual alignment, causing off-by-one.
    6. Boolean mask direction reversed: in a boolean mask True means attendable, while in an additive mask -inf means not attendable; the two directions are opposite.

    Conclusion

    The key to autoregressive blank infilling pretraining is organizing “bidirectional understanding of the visible region” and “sequential completion of the generation region” under a unified objective. What truly determines engineering success is not any single module but whether the following links are consistent:

    • is span sampling stable and controllable;
    • does each span use its own start token;
    • is the teacher-forcing input correctly shifted right;
    • does 2D position encoding distinguish original position from intra-span order;
    • is the attention mask span-blocked and isolated by default;
    • is the loss strictly aligned with the input positions;
    • does inference use the same isolation strategy as training.

    Combining these mechanisms with mask ratio, span length, context length, cost control, and an evaluation loop with explicit definitions yields a reproducible, debuggable, and deployable autoregressive blank infilling pretraining engineering solution.