Autoregressive Blank Infilling LLMs: Principles, Evaluation, and Deployment

    This article explains the autoregressive blank-infilling paradigm for large language models: how it differs from bidirectional encoders and left-to-right decoders, how training and inference work, what to check when using MaaS APIs, how to read benchmarks, how to reduce RAG hallucinations, and practical constraints for finance and industrial use.

    Large language models are a core capability for many organizations. The underlying architecture affects what a model can do, how much it costs, and how hard it is to deploy. This article focuses on the autoregressive blank-infilling paradigm: its mechanics, and how to decide whether it fits a given product.

    Limitations of two common architectures

    Pretrained models typically fall into two families. Bidirectional encoders such as BERT see left and right context around a target position, which helps classification, entity recognition, and other understanding tasks. Their training objective is not token-by-token generation, so long-form generation usually needs extra adaptation. Autoregressive decoders such as GPT predict the next token left to right and produce fluent text, but each prediction uses only the prefix.

    Teams that need both understanding and generation often train and serve two stacks, which raises inference cost and operational load.

    The table below compares three paradigms:

    Dimension Bidirectional encoder Autoregressive decoder Autoregressive blank infilling
    Visible context Bidirectional Left only Bidirectional context around masked spans plus already generated spans
    Training objective Recover masked tokens Predict the next token Autoregressively generate masked spans in order
    Strengths Classification, extraction, matching Continuation, dialogue Completion, rewriting, and mixed understanding–generation edits
    Main limitation Generation needs extra design Limited use of global context More complex training and inference

    How autoregressive blank infilling works

    The idea is to treat filling blanks as generation. During training, contiguous spans are sampled and replaced with placeholders. The model then generates each masked span autoregressively. When generating a span, it sees unmasked context plus spans already produced.

    Example: “I went to [MASK] today and bought a new computer.” The model reads surrounding text and generates a span such as “the electronics market.” The same structure can complete a missing middle of a sentence or rewrite a selected sentence in existing text.

    A representative paper is GLM (General Language Model Pretraining with Autoregressive Blank Infilling, Du et al., ACL 2022). It uses random-length span masking and position encodings that separate context positions from positions inside a span. Hyperparameters should follow the paper and the matching code repository; they can differ by model version.

    At serving time, prompt format is platform-specific. Placeholders and special tokens must match the official documentation, or outputs can be wrong.

    Model evolution and MaaS access

    Models in this family typically scale along parameter count and training data. When choosing a version, check parameter size, data coverage, context length, modalities, and how the model is released—not only the version number.

    MaaS (Model as a Service) exposes the model as an API so teams do not operate training or inference clusters. A typical path:

    1. Create an application and obtain access keys. Store keys in environment variables or a secrets manager, not in source control.
    2. Read the API documentation for request format, max input length, billing units, and rate limits.
    3. Validate on a fixed test set; record latency, error rate, and cost per call.
    4. Plan retries, timeouts, and fallbacks.

    Compute clusters, object storage, and container platforms affect elasticity and availability. Check regions, SLAs, and data residency against compliance needs.

    Evaluation: metrics and reporting

    Leaderboards are a starting point, not a substitute for product tests. When reading results, confirm:

    • Benchmark name, version, and date.
    • Metric definitions (for example, whether code pass rate is one sample or best-of-n).
    • Model version and decoding settings, including temperature and max generation length.
    • Test-set source and size, and possible overlap with training data.

    Code-generation evals need execution safety. Run generated code in isolation with limited network and filesystem access; do not execute it in production.

    Controlling hallucinations in RAG

    Retrieval-augmented generation (RAG) supplies external documents as context. Hallucination here means fluent text that does not match the given sources.

    Common mitigations:

    • Instruct the model to answer only from the provided material and to say when the answer is not there.
    • Require citations to document chunk IDs for human review.
    • Measure hallucination rate on a labeled set and record sample size and judging criteria.

    Context constraints reduce unsupported answers but do not guarantee correctness. Report numbers and test conditions rather than unverifiable phrases such as “almost none.”

    Industry use: finance and industrial operations

    In finance, models can extract and summarize research reports, filings, and news. Inputs are source documents; outputs are structured fields and summaries. Human review is required, especially for amounts, dates, and conclusions. Code for risk prototypes still needs independent backtesting and approval before production.

    In industrial and energy settings, logs and sensor data often live in different systems. Models can turn unstructured maintenance notes into structured records and help retrieve past repairs and procedures. Emergency guidance is high risk and should be confirmed by qualified engineers before action.

    Before launch, at least decide: whether data may leave the organization, who reviews outputs, the impact of errors, and how to roll back on failure.

    Conclusion

    Autoregressive blank infilling combines bidirectional context with span-level generation, which fits completion, rewriting, and mixed understanding–generation work. Selection should weigh the specific model version, evaluation protocol, and compliance. Validate on real data at small scale before wider rollout.

    References

    Du, Z., Qian, Y., Liu, X., et al. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. Proceedings of ACL 2022.