Autoregressive Span Infilling: Principles and Engineering Practices of a Universal Pretraining Paradigm

    This article explains autoregressive span infilling, a unified pretraining paradigm that combines the bidirectional understanding of autoencoding with the generative capability of autoregressive modeling. It compares the three dominant pretraining approaches, breaks down the core mechanisms—mask span design, two-dimensional positional encoding, and the training objective—and explains how the paradigm fits into inference, Model-as-a-Service (MaaS) platforms, and real-world industry applications. It closes with practical guidance on choosing a paradigm and applying engineering best practices.

    1. Introduction: The Evolution of Pretraining Paradigms

    During the pretraining stage of general-purpose language models, researchers have long faced a core tension: a model must both “read” its context and be able to “generate” text. Two main lines of work have emerged around this tension.

    One is the autoregressive approach. It is trained to predict the next token and generates text token by token, which makes it naturally suited to generative tasks such as dialogue and writing. However, its use of context is unidirectional—each token can only attend to the information to its left. The other is the autoencoding approach. It randomly masks part of the text and asks the model to reconstruct it, so every position can attend to both the left and right context. This performs well on understanding-oriented tasks but produces weaker generative coherence.

    Both approaches have their strengths, yet neither manages to cover everything. It is against this backdrop that autoregressive span infilling was proposed as a hybrid pretraining paradigm: it borrows the bidirectional understanding mechanism of autoencoding while retaining the generative capability of autoregression, aiming to cover both natural language understanding and generation within a single unified architecture.

    2. What Is Autoregressive Span Infilling

    The core idea of autoregressive span infilling can be summarized in one sentence: randomly mask continuous spans of text, then reconstruct the masked content in an autoregressive manner.

    Unlike classic autoencoding, which typically masks only discrete single tokens, autoregressive span infilling usually masks a span—a continuous segment of text. When reconstructing these masked spans, the model does not predict all masked positions in parallel at once; instead, it generates them one by one in positional order within the span. As a result:

    • For the context outside the masked span, the model can perceive it bidirectionally and therefore has understanding capability;
    • For the content inside the masked span, the model generates in order and therefore has generation capability;
    • The training objective and the inference objective are unified under the same goal, avoiding the mismatch between pretraining and fine-tuning.

    It is precisely this characteristic of “bidirectional outside, autoregressive inside” that makes it a highly general pretraining paradigm.

    3. A Side-by-Side Comparison of the Three Paradigms

    Dimension Autoregressive Autoencoding Autoregressive Span Infilling
    Training objective Predict the next token Reconstruct masked tokens Autoregressively reconstruct masked spans
    Context awareness Unidirectional (left side) Bidirectional Bidirectional outside, ordered inside
    Generative coherence Strong Weak Strong
    Understanding tasks Moderate Strong Strong
    Task unification Fairly unified Requires conversion Highly unified

    The table shows the trade-off logic of autoregressive span infilling: it uses span masking to obtain bidirectional context and inner autoregression to preserve generative capability, thereby striking a balance between understanding and generation.

    4. A Closer Look at the Core Mechanisms

    4.1 Mask Span Design

    The span masking strategy directly determines the quality of the training signal. Common practices include:

    • Ratio-based contiguous masking: Mask the text in blocks with a certain probability, where span length follows some distribution (for example, biased toward medium length). This avoids spans that are too short—making the task too easy—or too long—removing too much context.
    • Positional randomization: The number of masked spans is not fixed, so the model adapts to different masking ratios.
    • Mixed objectives: Some training samples mask contiguous spans while others mask single tokens, improving robustness.

    4.2 Two-Dimensional Positional Encoding

    When autoregressive generation is needed within a span, reusing an ordinary one-dimensional positional encoding is problematic: the masked positions have effectively “disappeared” from the original sequence, so their relative positional information is hard to express. A common solution is two-dimensional positional encoding: one dimension represents a token’s absolute position in the original text, and the other represents its internal order within its span. This lets the model know both “roughly where this content sits” and “its rank inside the span,” enabling stable, ordered generation.

    4.3 The Objective Function

    Its optimization objective is essentially still to maximize the conditional likelihood of the masked content. The following illustrative pseudocode describes the training flow:

    1. # Illustration: constructing a span-masked sample and computing the objective
    2. def build_span_masked_sample(tokens, mask_ratio=0.15):
    3. # 1. Randomly select several contiguous spans to mask
    4. spans = sample_spans(tokens, mask_ratio) # returns [(start, end), ...]
    5. labels = [None] * len(tokens)
    6. # 2. Replace masked regions with placeholders and record the targets to predict
    7. for (start, end) in spans:
    8. labels[start:end] = tokens[start:end] # record ground-truth
    9. tokens = tokens[:start] + [MASK] * len(range(start, end)) + tokens[end:]
    10. return tokens, labels, spans
    11. def span_filling_loss(model, tokens, labels, spans):
    12. # 3. Compute negative log-likelihood autoregressively within each span, in order
    13. total_loss = 0.0
    14. for (start, end) in spans:
    15. logits = model(tokens, span_pos=(start, end)) # introduce two-dimensional positional encoding
    16. for i in range(start, end):
    17. total_loss += cross_entropy(logits[i], labels[i])
    18. return total_loss

    Note that the code above is an illustrative implementation provided to aid understanding. A real system involves batch parallelism, the specific tensor layout of the positional encoding, and engineering-level tuning of the masking distribution, so it should not be used directly in production.

    5. From Pretraining to Inference Adaptation

    Models built on the autoregressive span infilling paradigm are naturally suited to two usage patterns: context continuation and instruction following. In practical integration, an additional alignment stage is usually involved: instruction tuning teaches the model to follow human instructions, and preference alignment further improves the usefulness and safety of its outputs. For developers, this means there is no need to switch back and forth between an “understanding-oriented interface” and a “generation-oriented interface”—a single conversational calling style can cover summarization, question answering, translation, code generation, and many other scenarios.

    6. Platformization and Model-as-a-Service

    For a single pretraining model to deliver value at scale, it needs an engineered service platform. The industry commonly adopts a Model-as-a-Service (MaaS) approach, exposing model capabilities through standardized interfaces. A typical platform architecture usually includes the following general-purpose components:

    • API gateway: unified authentication, rate limiting, and metering, abstracting away differences between backend models;
    • Container platform: elastic scaling and rolling releases for inference services;
    • Message queue: smoothing out traffic for asynchronous, batch, and long-running tasks;
    • Object storage: persisting training data, model weights, and invocation logs;
    • Logging, monitoring, and alerting: tracking invocation latency, success rate, and resource utilization to safeguard service quality.

    Under this architecture, enterprises and developers do not need to build expensive training and inference clusters on their own; they can obtain model capabilities through standard interfaces, significantly lowering the cost of experimentation. When built on top of platforms such as Baidu AI Cloud, mature general-purpose cloud product capabilities can also be reused, letting teams focus their energy on business logic itself.

    7. Industry Practices

    The value of general-purpose models based on autoregressive span infilling is ultimately realized in how they combine with real business. A few typical directions include:

    • Finance: intelligent customer-service Q&A, summarization and extraction from contracts and research reports, and classification-based review of risk-related text;
    • Industrial manufacturing: semantic retrieval of equipment manuals, and generation and verification of process documentation;
    • Energy and power: intelligent categorization of operations and maintenance records, and distillation of key points from dispatch logs.

    The common thread across these scenarios is “large volumes of unstructured text that require both understanding and generation”—which aligns well with the “understands and writes” characteristic of the autoregressive span infilling paradigm.

    8. Engineering Choices and Best Practices

    1. Choose the paradigm to fit the task: For generation-heavy tasks that emphasize coherence, autoregression remains the most reliable starting point. For understanding-intensive tasks, autoencoding is worth focusing on. When you need both and want architectural unification, autoregressive span infilling is more worth evaluating.
    2. Tune the masking strategy: The masking ratio and span length distribution have a significant impact on final results. It is advisable to run small-scale ablation experiments on your business corpus.
    3. Control hallucination: In scenarios such as retrieval-augmented generation (RAG), prioritize constraining the model to “answer based on the given context,” and combine this with citation tracing to reduce the risk of fabrication.
    4. Leverage platform capabilities: Delegate deployment, monitoring, and rate limiting to mature cloud infrastructure, and drive continuous optimization with logs and metrics.

    9. Conclusion

    The value of autoregressive span infilling lies in its ability to resolve the long-standing separation between “understanding” and “generation” through a single unified architecture. For developers, grasping its three key elements—mask span design, two-dimensional positional encoding, and the objective function—leads to better-grounded decisions when selecting a paradigm and putting it into practice. For enterprises, combining this with a platform-oriented Model-as-a-Service approach turns cutting-edge pretraining capabilities into real business value faster and more reliably.