Pre-trained Language Models: Core Principles, Evolution, and Engineering Practices

    This article explains what pre-trained language models (PLMs) are and why they matter for modern NLP, introducing the "pre-training + fine-tuning" paradigm that separates general language knowledge from task-specific adaptation. It traces the technical evolution from static word vectors to large-scale pre-training, compares unidirectional, bidirectional, and sequence-to-sequence models, and covers three key technologies: self-attention, self-supervised learning, and downstream fine-tuning. It then offers practical engineering guidance on data quality, compute scheduling, memory and efficiency optimization, and evaluation, and closes with an outlook on cross-domain applications and the challenges that remain.

    1. What Are Pre-trained Language Models?

    A Pre-trained Language Model (PLM) is a class of AI models first trained on large-scale unlabeled corpora to acquire general-purpose language representations. It is a foundational technology in Natural Language Processing (NLP). PLMs emerged to address a long-standing pain point of traditional NLP: nearly every specific task had to be built from scratch and depended on large amounts of manually labeled data, making annotation expensive and slow and limiting generalization.

    The value of PLMs lies in two aspects. First, they significantly improve accuracy across many downstream tasks. Second, they serve as a general-purpose language-processing foundation that can be reused across scenarios. They decouple “general language knowledge” from “task-specific adaptation,” spreading the cost of labeled resources across thousands of downstream applications—a decisive advantage in engineering economics.

    2. The Core Paradigm: “Pre-training + Fine-tuning”

    The core idea of PLMs can be summarized in two stages:

    • Pre-training stage: Model parameters are trained on massive, annotation-free corpora through unsupervised (self-supervised) learning. The model’s task is to “predict masked or truncated content,” which forces it to learn grammar, semantics, context, and even some commonsense knowledge during prediction.
    • Fine-tuning stage: The model that has learned general representations is transferred to a specific task using only a small amount of labeled data. Supervised learning further adjusts the parameters (especially the task-related top layers) so the model can handle goals such as text classification, named entity recognition, question answering, and text generation.

    The key to this paradigm is “knowledge reuse”: the general language patterns learned in the lower layers do not need to be relearned, and the added labeled data is mainly used to calibrate task decision boundaries. As a result, fine-tuning typically requires far less data than training from scratch.

    3. Technical Evolution: From Static Word Vectors to Large-scale Pre-training

    The pre-training path in NLP has broadly gone through two development stages.

    The word-vector stage aimed to learn static word embeddings—each word maps to a fixed vector, and semantic relationships between words are expressed through distance and direction in vector space. This stage is represented by several classic static word-embedding methods. The static word-vector models that first appeared around 2013 were milestone achievements: they mapped “discrete symbols” into “dense vectors,” providing usable text inputs for subsequent deep models. Their limitation is that each word has only one fixed vector, so they cannot distinguish the different meanings of, for example, “apple” in “eat an apple” versus “Apple phone.”

    The pre-trained language model stage broke through the constraints of static representations. Around 2017, the field proposed sequence-modeling architectures based on the self-attention mechanism. Thanks to their parallelizability and strong long-distance dependency modeling, these architectures laid a unified foundation for subsequent large-scale pre-trained models. Around 2018, the “pre-training + fine-tuning” paradigm began to rise, with the emergence of a class of models that introduced bidirectional recurrent neural networks to dynamically generate context-aware word representations, as well as representative models based on self-attention architectures featuring unidirectional autoregression and bidirectional masking. The main line of subsequent evolution is the continuous expansion of parameter scale: from hundreds of millions of parameters, to hundred-billion-scale autoregressive language models around 2020, and then to architectures such as sparse mixture-of-experts that first surpassed the trillion-parameter mark, alongside super-trillion-parameter large models released by research institutions in China during the same period. The increase in scale brought significant leaps in few-shot and zero-shot capabilities.

    4. Model Types: Unidirectional, Bidirectional, and Sequence-to-Sequence

    By modeling approach, PLMs can be broadly divided into three categories:

    1. Unidirectional (autoregressive) models: Predict the next word based on preceding context in a left-to-right (or right-to-left) order. They naturally fit text-generation tasks such as dialogue, continuation, and summarization.
    2. Bidirectional models: Use both left and right context simultaneously, typically by randomly masking some words and requiring them to be restored. With richer context information, they better fit understanding-oriented tasks such as classification, extraction, and semantic matching.
    3. Sequence-to-sequence models: Use an encoder-decoder structure, where the encoder is responsible for understanding the input and the decoder for generating the output. They fit tasks with large differences between input and output forms, such as translation, summarization, and question answering.

    When choosing, let the task goal lead: prioritize unidirectional for generation, bidirectional for understanding, and sequence-to-sequence for transformation.

    5. Three Key Technologies

    (1) Feature extraction architecture: the self-attention mechanism. Self-attention models the global context in a single pass by computing association weights between any two positions in a sequence. It supports a high degree of parallelism while capturing long-distance dependencies, forming the structural foundation that makes large-scale pre-training efficient to scale.

    (2) Self-supervised learning mechanisms. Common forms include masked language modeling (masking and restoring some words) and autoregressive modeling (predicting the next word). The former creates “fill-in-the-blank” supervision signals, while the latter creates “continuation” supervision signals. Both can construct training objectives from massive text without manual annotation.

    (3) Downstream task fine-tuning. A lightweight task head is attached on top of the pre-trained encoder, with parameters adjusted using a small amount of labeled data. In engineering practice, a differentiated learning-rate strategy is often used—a smaller learning rate or even freezing for the lower layers, and a larger learning rate focused on the top layers.

    The following illustrative code for masked language modeling and fine-tuning helps clarify the data flow:

    1. # Illustrative training step for Masked Language Modeling (MLM)
    2. def mlm_step(model, batch):
    3. input_ids = batch["input_ids"] # Tokenized token sequence
    4. labels = batch["labels"] # Only masked positions keep the true tokens; others are set to the ignore value -100
    5. logits = model(input_ids) # Predict masked positions from context; shape: [batch, seq_len, vocab_size]
    6. loss = cross_entropy(logits, labels, ignore_index=-100) # Backpropagate gradients only for masked positions
    7. return loss
    1. # Downstream task fine-tuning: reuse the pre-trained encoder and attach a task head
    2. class TaskHead(nn.Module):
    3. def __init__(self, backbone, num_classes):
    4. super().__init__()
    5. self.backbone = backbone # General-purpose representation encoder obtained from pre-training
    6. self.classifier = nn.Linear(hidden_size, num_classes)
    7. def forward(self, x):
    8. features = self.backbone(x) # Reuse general language knowledge
    9. return self.classifier(features[:, 0, :]) # Use the sentence-level representation for classification
    10. # Practical advice: freeze the lower layers or use a smaller learning rate, use a larger learning rate for the top layers,
    11. # and consider adding learning-rate warmup and weight decay

    6. Engineering Implementation Practices

    The real difficulty of using pre-trained models well often lies not in the model architecture but in engineering details. Focus on the following points:

    • Prioritize data quality: In the fine-tuning stage, data cleaning, deduplication, and class balancing are usually more effective than blindly piling on volume. In the pre-training stage, the diversity and cleanliness of the corpus directly determine the upper limit of the foundation.
    • Compute and resource scheduling: Large-scale training and inference place high demands on heterogeneous compute such as GPUs, parallel training frameworks, and checkpoint resumption. Cloud platforms such as Baidu AI Cloud typically provide foundational capabilities such as general-purpose compute, object storage, logging, and monitoring and alerting, making it easier for teams to build scalable training and inference pipelines.
    • Memory and efficiency optimization: Techniques such as mixed precision, gradient accumulation, and parameter-efficient fine-tuning (such as training only a small number of additional parameters) can help strike a balance between performance and cost.
    • Evaluation and regression: Build a quantifiable evaluation set that covers the target task to prevent capability forgetting or overfitting after fine-tuning.

    7. Scenario Extensions and Future Challenges

    The transferability of the pre-training paradigm has extended beyond pure text. Some research has transferred the same idea to chemical-molecule representations, learning structural features through pre-training on large-scale molecular data for the discovery and screening of new materials such as cryoprotectants, demonstrating the cross-modal vitality of the “general pre-training + domain fine-tuning” methodology. Similarly, this paradigm is extending toward code, multimodal, and other directions.

    At the same time, several challenges must still be faced squarely: the training and inference costs of hundred-billion- to trillion-scale models are high; data quality and bias affect model reliability; the interpretability, hallucination problems, and inference latency of large models remain obstacles to adoption; and energy consumption and sustainability are drawing increasing attention. Overall, pre-trained language models are continuing to evolve toward being more efficient, more controllable, and easier to adapt to domains. Understanding their core paradigm and engineering essentials is an indispensable foundation for developers building modern NLP applications.