From Autoregressive Infilling to Production: Architecture Evolution and Engineering Practices for General-Purpose Large Language Models

    General-purpose large language models have moved from isolated technical breakthroughs toward systematic engineering. This article takes a neutral look at how a mature LLM family addresses pretraining objectives, model size tiers, capability evaluation, multimodal extension, agent tool use, open-source distribution, hardware adaptation, and industry governance. Developers and enterprises will find reusable methods for model selection, deployment, and evaluation.

    General-purpose large language models have moved beyond single-point technical breakthroughs into systematic engineering. A mature model family typically needs to solve pretraining objective design, parameter-scale tiering, capability evaluation, multimodal extension, agent tool calling, open-source distribution, chip adaptation, and industry governance at the same time. This article reviews the technical evolution and engineering practices behind general-purpose LLM families from a neutral perspective, helping developers and enterprises build reusable methods for selection, deployment, and evaluation.

    1. Understanding Pretraining Architectures Through Autoregressive Span Infilling

    Autoregressive span infilling is a pretraining objective that balances understanding and generation. The basic approach: sample several contiguous segments from the input text as “blanks,” and the model autoregressively generates the content of each blank based on the unmasked context. Compared with pure causal language modeling that predicts one token at a time, span infilling lets the model leverage more complete context during generation. Compared with traditional masked modeling that predicts masked positions independently, it generates contiguous segments, which better approximates real text generation—making it well suited to unifying understanding and generation capabilities.

    In engineering practice, several areas deserve close attention:

    1. Span sampling strategy: Short and long spans should be mixed. Short spans train local semantic completion; long spans train paragraph-level coherent generation. An overly homogeneous sampling distribution biases the model toward short-text completion.
    2. Attention mask design: Visibility rules must be explicit for unmasked context, within a span, and across different spans, to prevent information leakage while preserving training parallelism.
    3. Positional encoding and length extrapolation: For long-context scenarios, positional encoding design directly affects long-document retrieval and cross-paragraph reasoning stability, and should be paired with continued long-context training and evaluation.
    4. Data mixture and deduplication: Pretraining data quality matters more than simply scaling up. Establish mechanisms for deduplication, cleaning, quality grading, and domain mixing ratios.
    5. Training stability: Large-scale training requires monitoring loss spikes, gradient norms, mixed-precision overflow, and checkpoint rollback strategies.

    Pseudocode sketch:

    1. def build_infilling_batch(sequence, max_span, mask_ratio):
    2. spans = sample_contiguous_spans(sequence, max_span, mask_ratio)
    3. inputs = replace_spans_with_blank(sequence, spans)
    4. targets = [sequence[start:end] for start, end in spans]
    5. return inputs, targets
    6. def train_step(batch):
    7. inputs, targets = batch
    8. logits = model(inputs, attention_mask=build_infilling_mask(inputs))
    9. loss = autoregressive_loss(logits, targets)
    10. return loss

    The value of this objective is that a single architecture can support text understanding, question answering, summarization, code completion, and dialogue generation. Engineering teams, however, should not chase an elegant objective function alone—use ablation experiments to validate the real-world benefit of span ratios, segment lengths, context windows, and data mixtures.

    2. Model Tiering and Selection Methods

    A mature general-purpose LLM family is rarely a single model; it is a tiered matrix: a lightweight open-source tier for low-latency, low-cost, and edge scenarios; a standard base tier for general QA, summarization, and knowledge processing; a high-performance commercial tier for complex reasoning, long-document analysis, and high-value tasks; a multimodal tier for image-text, document, and screenshot understanding; and an agent-enhanced tier for tool calling, task planning, and execution feedback. Selection should not hinge on parameter count alone—build a multi-dimensional evaluation matrix.

    Dimension Key Question Recommended Approach
    Task complexity Multi-step reasoning, code generation, or long-document analysis needed? Start with a small model to establish a baseline, then upgrade based on measured gains
    Context length Long documents, ticket histories, or multi-turn dialogues involved? Evaluate long-text retrieval and cross-paragraph consistency
    Inference cost Latency, throughput, GPU memory, cost per request Plot a cost-effectiveness curve rather than looking at peak performance alone
    Modality requirements Images, tables, or scanned documents involved? Separate text-only and multimodal pipelines
    Tool calling Retrieval, computation, or business-system operations needed? Evaluate call success rate, parameter validation, and failure rollback
    Compliance and licensing Data classification, deployment form, license constraints Maintain model cards and a compliance review checklist

    The guiding principle is “small models first, evaluation-driven, scale by scenario.” For classification, extraction, and ticket routing, lightweight models combined with rules and retrieval are often sufficient; for complex planning and high-stakes decisions, bring in high-performance models and control cost through caching, batching, and distillation.

    3. Capability Evaluation and Trustworthy Assessment

    Coding, long-text processing, and logical reasoning are core LLM capabilities, but public benchmarks are only a reference point. Enterprises should build internal evaluation sets covering real business distributions, edge cases, and adversarial samples. Evaluation should cover at least: task accuracy, code pass rate, long-text retrieval accuracy, multi-step reasoning consistency, tool-call success rate, refusal rate, hallucination rate, safety and compliance hit rate, plus latency P50/P95, throughput, memory footprint, and cost per token.

    A recommended evaluation workflow:

    • Baseline establishment: Fix prompt templates, decoding parameters, and evaluation scripts to guarantee reproducibility.
    • Contamination detection: Run similarity checks between public benchmarks and training data to avoid inflated scores.
    • Layered reporting: Break down results by task type, text length, domain, and difficulty, so averages don’t hide weaknesses.
    • Human review: Retain manual sampling and error attribution for high-risk scenarios.
    • Regression testing: Re-run the regression suite after every model upgrade, quantization, distillation, or prompt change.

    Public leaderboards are fine for initial screening but cannot replace business-specific evaluation. In production, response latency, failure recovery, permission control, and auditability often matter more than single-shot quality.

    4. Multimodal and Agent Capability Extensions

    Multimodal capability is typically achieved through joint vision-language modeling, covering image-text QA, document understanding, chart parsing, and screenshot understanding. Engineering challenges include balancing image resolution against token compression, preserving layout structure, robustness to OCR noise, and modality alignment. For contracts, reports, and ticket screenshots, a combined pipeline of “multimodal understanding + structured extraction + rule-based validation” is recommended, rather than letting the model output final business conclusions directly.

    Agent capabilities extend the model from an “answerer” to a “task executor.” Typical components include a task planner, tool registry with parameter validation, execution sandbox, memory management, audit logging, and failure rollback. The key is not unlimited autonomy but clear boundaries: which tools may be called, which operations require approval, how failures are retried, and how unauthorized actions are blocked. Agent evaluation should track tool-selection accuracy, parameter validity, task completion rate, and side-effect control.

    5. Open-Source Ecosystem, Hardware Adaptation, and Deployment Engineering

    Open-source distribution lowers the barrier to adoption and builds a developer ecosystem, but enterprises still need to watch licensing, model cards, safety alignment, and version management. For hardware adaptation, establish operator coverage matrices, precision-alignment testing, graph compilation optimization, communication optimization, and performance regression mechanisms; validate quality loss separately for quantization formats, mixed precision, and long-context caching. Common deployment techniques include inference gateways, dynamic batching, KV caching, quantization, distillation, result caching, rate limiting, and elastic autoscaling.

    When deploying in the cloud, general cloud capabilities—container platforms, object storage, logging services, and monitoring/alerting—can host training artifact management, inference service orchestration, and service observability. For example, Baidu AI Cloud offers such general-purpose cloud services for this purpose. A sample configuration:

    1. service:
    2. replicas: auto
    3. batching: dynamic
    4. cache: kv
    5. observability:
    6. metrics: true
    7. logs: true
    8. traces: true
    9. safety:
    10. rate_limit: true
    11. audit_log: true

    Before deployment, make sure the following are settled: model versions bound to prompt template versions, canary release and rollback strategies, cache invalidation rules, long-text truncation policies, and graceful degradation paths. For high-concurrency scenarios, dynamic batching and KV caching significantly improve throughput, but guard against long requests adding tail latency for short ones.

    6. Industry Applications and Governance Practices

    General-purpose LLMs can support public-service governance with policy Q&A and ticket classification, manufacturing with equipment-manual retrieval and log diagnostics, finance with compliance review and research-report summarization, and education with lesson-preparation assistance and Q&A. The common pattern across industries: the model does not replace business systems directly, but is embedded in a “retrieve–generate–validate–audit” closed loop.

    For governance, establish data classification and masking, content safety filtering, permission control, operation auditing, model cards, red-team testing, and version rollback mechanisms. High-risk outputs should require cited sources, confidence indicators, and human review. After launch, continuously monitor hallucination rate, refusal rate, user feedback, and cost metrics, forming an “evaluate–deploy–monitor–iterate” cycle.

    7. Best Practices Checklist

    1. Define tasks and acceptance metrics before choosing model scale and form.
    2. Establish baselines with lightweight models, then upgrade along the cost-effectiveness curve.
    3. Use public benchmarks only for initial screening; business evaluation sets must cover the real long tail.
    4. For long-text scenarios, prioritize validation of retrieval accuracy and cross-paragraph consistency.
    5. In multimodal scenarios, use structured extraction and rule-based validation to reduce hallucinations.
    6. Agents must have tool whitelists, approval flows, sandboxes, and audit logging.
    7. Optimize latency, throughput, cost, and observability together when deploying.
    8. Manage model versions, prompt templates, and evaluation sets under unified versioning.
    9. Run dual regression on precision and performance for quantization, distillation, and hardware adaptation.
    10. Bake safety, compliance, and human review into the production workflow—not as afterthoughts.

    Overall, the competitiveness of a general-purpose LLM family comes not just from parameter scale, but from the systematic coordination of pretraining architecture, evaluation systems, multimodal and agent extensions, open-source ecosystems, hardware adaptation, and industry governance. Developers should center on business goals and build reproducible evaluation and deployment engineering to balance quality, cost, and risk.