Pre-trained Language Models: Principles, Evolution, and Engineering Practices
Pre-trained language models (PLMs) have become the go-to foundation for modern natural language processing. This article explains why traditional, task-specific NLP approaches fall short, then walks through the two-stage "pre-train, then fine-tune" paradigm with pseudocode. It traces the field's evolution from static word embeddings to contextual pre-training and scaling, examines the self-attention architecture and its advantages, and compares the three main model families: unidirectional, bidirectional, and sequence-to-sequence. It also covers the key techniques behind PLMs, such as masked language modeling and parameter-efficient fine-tuning. Finally, it outlines typical applications and offers practical engineering advice on data, compute, model sizing, fine-tuning, and evaluation for production deployments.
Why Traditional NLP Struggles, and Why Pre-trained Language Models Emerged
In traditional NLP, models are typically designed and trained separately for each specific task. Whether it is text classification, named entity recognition, or sentiment analysis, they all rely heavily on large amounts of manually labeled data. However, collecting labeled data is expensive and slow, and it can never cover the full range of language phenomena. As a result, such models often underperform in data-scarce scenarios and generalize poorly.
Pre-trained Language Models (PLMs) were introduced to break this “one model and one annotation set per task” pattern. The core idea is to split “understanding language” into two stages: first, the model learns general linguistic regularities from large-scale unlabeled text; then, it is adapted to a specific task using a small amount of labeled data. This “pre-train, then fine-tune” paradigm provides a reusable, general-purpose solution for language understanding and generation.
Its main value lies in two aspects. First, it significantly improves accuracy and robustness across NLP tasks, because the model has already accumulated rich semantic and syntactic knowledge from large-scale corpora. Second, it serves as a general-purpose foundation for language processing: after fine-tuning, one pre-trained model can serve multiple downstream tasks, greatly reducing redundant development effort.
The Core Idea: A Two-Stage Learning Paradigm
PLMs are usually built on encoder or decoder architectures based on the self-attention mechanism, and their learning process is clearly divided into two stages.
Pre-training stage: An extremely large corpus is used for unsupervised (self-supervised) learning, with the goal of learning general language representations. Since it does not depend on manual annotation, the model can make full use of the massive text resources available on the internet, extracting lexical and syntactic knowledge and, to some degree, semantic and world knowledge.
Fine-tuning stage: The pre-trained model is transferred to a specific task, and the model parameters are further adjusted with supervised learning using a small amount of labeled data. Compared with training from scratch, fine-tuning typically requires only a small fraction of the labeled data yet can quickly achieve or even surpass the performance of task-specific models.
This decoupled design brings a direct benefit: the expensive training of general-purpose capability happens only once, while the cost of task adaptation is spread across many tasks. The basic flow is shown in the pseudocode below:
# Stage 1: Pre-training (large unlabeled corpus, self-supervised objective)model = initialize_model()for batch in unlabeled_corpus:loss = self_supervised_objective(model, batch) # e.g., masked language modelingloss.backward()optimizer.step()# Stage 2: Fine-tuning (small labeled dataset, supervised objective)for batch in labeled_dataset:logits = model(batch.inputs)loss = task_loss(logits, batch.labels) # e.g., cross-entropy for the downstream taskloss.backward()optimizer.step()
Technical Evolution: From Static Word Vectors to Deep Pre-training
The development of pre-training in NLP has mainly gone through two stages: static word vectors and pre-trained language models.
Stage 1: Static word vectors. This stage aimed to learn static word embeddings, mapping words into a low-dimensional dense vector space so that semantically similar words are also close to each other in that space. Around 2013, the field introduced milestone static word embedding methods, and later improvements appeared that combined global co-occurrence statistics and subword information. These approaches significantly alleviated the curse of dimensionality and semantic sparsity of traditional one-hot encoding. However, their fundamental limitation was “one word, one vector”: whether the word “apple” appears in “eat an apple” or in “Apple phone,” the resulting vector is identical, so polysemy cannot be distinguished.
Stage 2: Contextual pre-training. In 2017, a new neural network architecture based on self-attention was proposed. With parallelizable computation and strong ability to model long-range dependencies, it laid a unified architectural foundation for the large models that followed. Around 2018, contextual word vector methods that modeled context with bidirectional recurrent networks, as well as various pre-trained models built on the new architecture, emerged one after another, and the “pre-train + fine-tune” paradigm became established. These models can generate word representations dynamically according to context, genuinely solving the polysemy problem.
Scaling stage. After that, model parameters and training data scale continued to climb: from hundreds of millions of parameters in the early days, to hundreds of billions around 2020, and then to trillions around 2021. This showed an almost exponential growth trend.
The Key Architecture: Self-Attention
The core that supports modern pre-trained models is a neural network architecture based on Self-Attention. Unlike recurrent neural networks, which process words one by one recursively and are hard to parallelize, the self-attention mechanism lets every position in the sequence directly “attend” to all other positions, dynamically weighting and aggregating information by computing correlations among the Query, Key, and Value.
Its core advantages are threefold:
- Strong long-range dependency modeling: Any two positions are directly connected, so the path length is always one, avoiding the decay of information with distance that occurs in recurrent structures.
- Highly parallelizable: Computations for all positions can be carried out simultaneously, making full use of large-scale compute.
- Good scalability: By stacking multiple identical layers and introducing multi-head attention, model capacity grows steadily with scale.
These properties are what allow the architecture to carry parameter counts ranging from hundreds of millions to trillions.
Three Main Model Types: Unidirectional, Bidirectional, and Sequence-to-Sequence
Depending on the visibility of attention and the network structure, PLMs are mainly divided into three types:
- Unidirectional models can only see the current position and the context to its left (modeling from left to right). Essentially autoregressive language models, they excel at open-ended text generation such as dialogue and story continuation.
- Bidirectional models use context from both sides, and are typically trained with Masked Language Modeling (MLM). They excel at understanding tasks such as text classification, question answering, and named entity recognition.
- Sequence-to-sequence models use an encoder–decoder structure: the encoder understands the input, and the decoder generates the output. They suit tasks that need to “convert input into output,” such as machine translation, text summarization, and sentence rewriting.
Choosing the right type essentially depends on whether the downstream task leans toward understanding or generation.
Key Techniques Explained
(1) Feature extractor. The encoder or decoder based on self-attention is responsible for feature extraction. Through multi-layer stacking, it abstracts features layer by layer, gradually moving from surface lexical features to deep semantic features.
(2) Self-supervised learning mechanism. This is what makes pre-training on unlabeled data possible. Taking Masked Language Modeling as an example, the model randomly masks some tokens in the input text and then tries to reconstruct the masked content, so that it is forced to learn contextual semantics while “guessing words”:
# Core logic of masked language modelingtokens, labels = mask_random_tokens(input_ids, mask_prob=0.15)outputs = encoder(tokens) # Bidirectional contextual encodingloss = cross_entropy(predict(outputs), labels) # Loss is computed only for masked positions
Besides masked language modeling, causal language modeling (autoregressively predicting the next token) is another common self-supervised objective.
(3) Downstream task fine-tuning. On top of the pre-trained weights, usually only a task-specific output layer needs to be added, followed by training with a small amount of labeled data. In addition to full-parameter fine-tuning, parameter-efficient fine-tuning updates only a very small number of parameters to adapt to a new task. This significantly saves compute and storage overhead and mitigates overfitting in small-sample settings.
Application Value and Typical Scenarios
PLMs have become a general-purpose foundation for many NLP applications. Typical scenarios include text classification and sentiment analysis, information extraction and named entity recognition, intelligent question answering, machine translation, text summarization and generation, and code generation. Beyond plain text, the ideas have also been transferred to other fields: research teams have applied pre-trained models to the representation learning of chemical molecules, for example in screening stages of cryoprotectant development. This shows the paradigm has universal value across modalities and domains.
Engineering Practices and Common Challenges
Deploying PLMs in production requires attention to the following aspects:
- Data quality and scale: Pre-training results depend heavily on the size, diversity, and cleanliness of the corpus, so deduplication, denoising, and sensitive information filtering should be done well.
- Compute and resource management: Training large models requires enormous compute. Containerized platforms on the cloud can be used to elastically schedule compute, object storage can manage massive training corpora, and logging and monitoring/alerting services can track training status and anomalies in real time.
- Trade-off between scale and cost: Bigger is not always better; choose an appropriate model size based on task complexity, latency requirements, and cost budget.
- Fine-tuning strategy and overfitting prevention: In small-sample scenarios, prefer parameter-efficient fine-tuning, combined with regularization and early stopping.
- Evaluation and iteration: Establish quantifiable evaluation metrics that fit the business, continuously track post-launch drift, and iterate in time.
In summary, PLMs turn general language understanding into reusable infrastructure through the paradigm of “large-scale self-supervised pre-training + small-scale supervised fine-tuning.” Understanding their architecture evolution, type divisions, and training objectives, and doing systematic planning for data, compute, and evaluation in engineering, is the key to building high-quality NLP systems.
