Language Models Explained: From N-Grams to Neural Network Modeling
This article explains how language models assign probabilities to word sequences and why they are foundational to natural language processing. It covers the mathematical formulation using the chain rule, the Markov assumption and smoothing in n-gram models, and how neural language models use word embeddings, recurrent architectures, attention mechanisms, and cross-entropy training to model context. It also discusses perplexity as an evaluation metric and practical considerations for building language models.
1. What Is a Language Model?
A language model is an abstract mathematical model of natural language. Its core question is: how plausible is a sequence of words in a language, and how likely is it to occur? In other words, a language model assigns a probability to any word sequence to describe how well that sequence follows linguistic patterns.
From an engineering perspective, language models are foundational infrastructure for many natural language processing tasks. Whether the task is next-word prediction in input methods, machine translation, post-processing for speech recognition, or modern large-model text generation, the underlying task often involves estimating the probability of the next word. Understanding how language models are formulated is the first step toward understanding modern text intelligence technologies.
2. Mathematical Foundations of Language Modeling
The mathematical goal of language modeling is clear: given a word sequence $w_1, w_2, \dots, w_n$, estimate its joint probability $P(w_1, w_2, \dots, w_n)$.
By the chain rule of probability, the joint probability can be decomposed into a product of conditional probabilities:
P(w_1, ..., w_n) = P(w_1) · P(w_2|w_1) · P(w_3|w_1,w_2) · ... · P(w_n|w_1,...,w_{n-1})
This formula is complete in theory, but it faces two major challenges in practice. First, the parameter space grows exponentially with sequence length, making reliable estimation difficult. Second, long-distance dependencies create large historical conditioning contexts, leading to severe data sparsity. For this reason, different families of language models make approximations and simplifications within this framework. The main differences lie in how much historical information is used and how conditional probabilities are modeled.
3. N-Gram Statistical Models
3.1 The Markov Assumption
N-gram models are among the earlier language models to be systematically used. Their core simplification is the Markov assumption: the current word depends only on a limited number of preceding words and is independent of more distant history.
- When n = 1 (Unigram), each word is assumed to be independent.
- When n = 2 (Bigram), the current word depends only on the previous word.
- When n = 3 (Trigram), the current word depends on the previous two words.
For example, in a bigram model, the conditional probability is approximated as:
P(w_i | w_1...w_{i-1}) ≈ P(w_i | w_{i-1})
This reduces the parameter space from exponential growth to a manageable scale. In practice, it is enough to count the co-occurrence frequencies of adjacent words in a corpus.
3.2 Probability Estimation and Smoothing
The simplest maximum likelihood estimate is obtained by dividing frequencies:
P(w_i | w_{i-1}) = count(w_{i-1}, w_i) / count(w_{i-1})
However, many reasonable word combinations never appear in a real corpus. This can cause their estimated probabilities to become zero, which in turn makes the probability of an entire sentence zero. To address data sparsity, smoothing techniques are widely used, such as Laplace smoothing and Kneser-Ney smoothing. These methods assign a small probability to unseen events to improve model robustness.
3.3 Advantages and Limitations
The advantages of n-gram models are that they are intuitive, interpretable, and fast to train and run. They remain useful for small-scale tasks and baseline comparisons.
Their limitations are also clear: they cannot capture long-distance semantic dependencies; as n increases, parameter sparsity becomes more severe; and because they rely entirely on discrete symbol statistics, they lack the ability to model semantic similarity between words.
4. Neural Network Language Models
To overcome the limitations of n-gram models, neural network language models were developed. They replace discrete counting with continuous vector spaces.
4.1 Word Embeddings and Distributed Representations
A key contribution of neural language models is the introduction of word embeddings: each word is mapped to a low-dimensional dense vector, and semantically similar words are closer together in the vector space. This gives the model a natural ability to generalize semantically. Even if a particular word combination never appears during training, the model can still make a reasonable estimate by leveraging vector similarity.
4.2 From Recurrent Networks to Attention Mechanisms
- Recurrent neural networks (RNNs) pass information between words through hidden states. In theory, they can model dependencies of arbitrary length, but they suffer from vanishing gradients.
- Long short-term memory networks (LSTMs) and gated recurrent units (GRUs) use gating mechanisms to alleviate the difficulty of retaining long-sequence memory.
- The Transformer architecture, based on self-attention, has become the foundation of mainstream language models because of its ability to support parallel computation and model global dependencies.
4.3 A Conceptual Training Loop
The following example code illustrates the basic training idea of a neural language model. It is only conceptual and is not tied to a specific framework:
for epoch in range(num_epochs):for tokens in dataloader: # tokens: a sequence of wordsinputs = tokens[:, :-1] # Input: the first n-1 wordstargets = tokens[:, 1:] # Labels: the shifted-right target wordslogits = model(inputs) # Predicts the next-word distribution at each positionloss = cross_entropy(logits, targets)optimizer.zero_grad()loss.backward()optimizer.step()
The core idea remains the same: given the preceding context, predict the next word, and use cross-entropy loss to continuously adjust parameters so that the probability distribution output by the model approaches the true distribution of language.
5. How to Evaluate Language Models
Perplexity is one of the most commonly used metrics for evaluating language models. It can be understood as the average branching factor of the model over a test corpus. A lower value means the model is more confident in predicting real text and is less “perplexed.”
When evaluating language models, it is important to strictly separate the training set from the test set to avoid data leakage, which can artificially inflate the metric.
6. Practical Considerations and Outlook
When building language models in real engineering projects, several factors matter: data quality and deduplication directly determine the upper limit of model performance; tokenization granularity affects vocabulary size and sparsity; n-gram models can serve as fast baselines for comparison, while neural network models are better suited to scenarios with higher semantic requirements.
After deployment, model performance can be tracked over time with logging services and monitoring alerts. Object storage can also be used to accumulate corpora and support a sustainable iteration cycle.
The evolution of language models—from statistical counting in n-gram models to distributed semantic modeling in neural networks—reflects the broader shift in natural language processing from symbolic processing to representation learning. Understanding this progression provides a solid foundation, whether one later works on retrieval-augmented generation, text classification, or generative applications.
