From Static Word Vectors to Pre-trained Language Models: The Evolution of NLP Representation Learning
A practical walkthrough of how natural language processing representation learning moved from static word embeddings to context-dependent pre-trained language models. The article covers why one-hot encoding falls short, the prediction-based and count-based routes to static word vectors, the subword techniques used to handle out-of-vocabulary words, and why "one vector per word" breaks down on polysemy. It then explains context-dependent representations, the pre-train and fine-tune paradigm, the self-attention architecture and its attention formula, and the main self-supervised objectives (autoregressive, masked, and permutation language modeling). It closes with a side-by-side comparison of the two stages and engineering guidance on model selection, fine-tuning strategies, and compute and infrastructure requirements.
1. Introduction: Two Leaps in Representation Learning
One of the core challenges in natural language processing is making machines understand and represent human language. Because computers can only process numbers, mapping discrete symbols such as words, characters, and subwords into a continuous vector space has always been a foundational problem in NLP. Looking back at the field’s evolution, two milestones stand out: the word embedding stage, represented by static word vectors, and the pre-trained language model stage, centered on context-dependent representations. These two stages are not a simple case of one replacing the other. They form a continuous progression in which representation capability moves from coarse-grained and static toward fine-grained and dynamic.
2. Stage One: Static Word Vectors
2.1 Where One-Hot Encoding Breaks Down
Early text representations typically relied on one-hot encoding: assign one dimension to every word in the vocabulary, set that dimension to 1 for the word in question and 0 everywhere else. This approach has two fatal flaws. The first is the curse of dimensionality: when the vocabulary reaches tens of thousands or hundreds of thousands of words, the vectors grow correspondingly long and remain extremely sparse. The second is semantic isolation: any two word vectors are orthogonal to each other, so the approach cannot express that “cat” and “dog” are semantically similar. The distributional hypothesis supplies the theoretical basis for a better approach: a word’s meaning is determined by its context, and words with similar meanings tend to appear in similar contexts.
2.2 Technical Routes to Static Word Vectors
Building on that idea, several representative static word vector methods emerged. Their shared goal is to learn a dense, low-dimensional word vector matrix in which words that are semantically or grammatically similar end up closer together in the vector space. In terms of implementation, these routes fall roughly into two categories:
- Prediction-based methods: These use a local sliding window as the modeling unit and learn vectors either by predicting the context from the center word or by predicting the center word from its context. They belong to the category of shallow neural networks, have a clearly defined training objective, and capture local semantic relationships well.
- Count-based methods: These first build a global co-occurrence matrix and then derive dense vectors through techniques such as matrix factorization. They make full use of global statistics and are relatively inexpensive to train.
In addition, to mitigate the out-of-vocabulary (OOV) problem, some improved approaches introduced subword or character-level modeling. These break words into smaller units such as character n-grams, which gives the model the ability to handle unseen words and, to some degree, to express a word’s internal morphology.
2.3 The Fundamental Limitation of Static Word Vectors
Regardless of the technical route taken, static word vectors share one defining characteristic: one word, one vector. Each word corresponds to a single, fixed vector across the entire corpus, and this leads to two unavoidable problems. First, static vectors cannot handle polysemy. The word “apple” means something entirely different in “she ate an apple” than in “the Apple event,” yet a static representation can only return the same vector for both, discarding the context. Second, being context-independent limits a model’s ability to capture subtle semantic distinctions. It was precisely these limitations that pushed researchers toward representations that can sense context.
3. Stage Two: Pre-trained Language Models
3.1 From Static to Dynamic: Context-Dependent Representations
The most fundamental breakthrough of this stage is replacing static, fixed vectors with context-dependent dynamic representations. The same word is encoded into different vectors in different sentences and contexts, which resolves polysemy disambiguation naturally. Underneath this is the pre-train and fine-tune paradigm: first, self-supervised pre-training on a large unlabeled corpus so the model learns general linguistic regularities; then, transferring the learned parameters to a specific downstream task and fine-tuning on a small amount of labeled data to achieve strong results.
3.2 The Core Architecture: Self-Attention
The architecture underpinning this shift is the encoder-decoder design built around self-attention, proposed around 2017. The core idea of self-attention is that when processing a given position in a sequence, that position interacts with every other position in the sequence through weighted contributions, dynamically capturing long-range dependencies. The basic computation can be written as follows:
Attention(Q, K, V) = softmax(Q · Kᵀ / √d_k) · V
Here, Q (query), K (key), and V (value) are obtained through linear transformations of the input representation, and √d_k is a scaling factor used to stabilize gradients. Unlike a recurrent neural network, which must compute step by step in sequence order, self-attention can process the entire sequence in parallel, and the path length between any two positions is constant. This both improves training efficiency and significantly strengthens the ability to capture long-range dependencies. On top of this, multi-head attention lets the model attend to different kinds of information in parallel across multiple representation subspaces, while positional encoding compensates for the fact that the attention mechanism itself contains no notion of order. Together, these form the key components of the architecture.
3.3 Diverse Pre-training Objectives
Around this architecture, the core of the pre-training stage is designing a sensible self-supervised objective. Mainstream pre-training objectives fall into roughly three categories:
- Autoregressive language modeling: Predict the next word from the preceding context. This is left-to-right generative modeling and naturally suits text generation tasks.
- Masked language modeling (autoencoding): Randomly mask some words in the input and have the model restore them using bidirectional context, which yields richer bidirectional semantics and suits understanding-oriented tasks particularly well.
- Permutation language modeling: Randomly permute the prediction order so that the objective keeps bidirectional context while remaining compatible with autoregressive generation, attempting to combine the strengths of the first two objectives.
These different objective designs gave rise to a family of pre-trained models with distinct characteristics, each with its own strengths in text generation, semantic understanding, and related tasks.
4. Comparing the Two Stages and Their Connections
From an engineering perspective, the two stages differ fundamentally yet are clearly connected. The table below summarizes the comparison across several key dimensions:
| Dimension | Static Word Vectors | Pre-trained Language Models |
|---|---|---|
| Representation granularity | Word-level, static | Word/subword-level, context-dependent |
| Context awareness | None | Yes (bidirectional or unidirectional) |
| Transfer method | Concatenating word vectors | Fine-tuning the whole set of parameters |
| Task adaptation | Features must be redesigned | General backbone plus lightweight fine-tuning |
As this shows, the pre-trained stage did not discard the core idea of the word vector stage. It pushed the notion of “learning representations from large-scale unlabeled corpora” to its limit, with the only change being that the object of learning shifted from static word vectors to the parameters of the entire model.
5. Engineering Practices
5.1 Model Selection
In real projects, the right objective and structure should be chosen according to the nature of the task. Generation tasks such as summarization, dialogue, and continuation are better served by an autoregressive decoder structure. Understanding tasks such as classification, extraction, and semantic matching are better served by a bidirectional encoder structure trained with masked language modeling. Beyond that, teams need to balance quality, inference latency, and resource usage, and where necessary choose lightweight variants that have been compressed, distilled, or quantized.
5.2 Fine-Tuning Strategy
The value of pre-trained models lies in their transferability. Common transfer approaches include fine-tuning only the top few layers or adding a new task head (parameter-efficient fine-tuning), and full-parameter fine-tuning. When labeled data is scarce, parameter-efficient methods, such as training only a small number of inserted parameters, can approach the results of full fine-tuning at lower cost and are worth trying first.
5.3 Compute and Infrastructure
Pre-training and large-scale fine-tuning are compute-intensive workloads with high demands on processing power, memory capacity, and distributed training capability. In practice, teams typically build scalable training and inference pipelines on cloud platforms that provide heterogeneous computing resources such as accelerator card clusters, distributed training frameworks, object storage for large corpora and model weights, and logging, monitoring, and alerting services. Using managed cloud capabilities for compute scheduling, model training, and online inference services lets teams focus on the business problem rather than low-level operations.
6. Conclusion and Outlook
From static word vectors to pre-trained language models, representation learning in natural language processing has undergone a profound transformation: from discrete and sparse to continuous and dense, and then to context-dependent and dynamic. Understanding this progression helps explain where today’s mainstream technologies came from, and it also provides a clear basis for decisions about model selection, fine-tuning optimization, and engineering deployment. Looking ahead, as multimodal pre-training, more efficient variants of self-attention, and alignment techniques continue to advance, the capability boundary of pre-trained language models will keep expanding, which makes this an area worth sustained attention and investment for developers.
