The Evolution of Large Language Models: From Biomimicry to Scale
Large language models did not appear out of nowhere—their technical lineage stretches back more than a century of research into the human brain. This article traces that journey across four stages: mimicking neurons, mimicking recurrent circuits, lifting recurrence constraints through attention, and aligning models with human feedback. Along the way, it walks through the key milestones—from Cajal's neural circuits and the McCulloch-Pitts neuron, through perceptrons, backpropagation, RNNs, and LSTMs, to the Transformer architecture, the pretraining paradigm, scaling laws, and RLHF. It then examines the engineering focus of the current era—long context, inference efficiency, multimodality, agents, and supporting infrastructure—and closes with four patterns that govern this evolution plus practical advice for developers who need to make sound engineering decisions in a fast-moving field.
Introduction: A Century-Long “Biomimetic” Thread
Large language models did not emerge from nowhere. Their technical origins trace back more than a hundred years to research on the human brain. Viewed across this history, the journey divides roughly into four stages:
- Mimicking the neuron: modeling neural cells with mathematical units;
- Mimicking recurrent circuits: letting information circulate through a network so it can “remember”;
- Removing the recurrence constraint: using attention to let any two positions interact directly;
- Mimicking human learning feedback: using human preferences to make model output match expectations.
Each leap is, at its core, an answer to the same question: how to make a machine learn the patterns of language and the expression of knowledge from data, with ever greater efficiency.
This article unpacks the key technical milestones along that thread and, at the end, summarizes the patterns of this evolution and offers guidance for engineering practice.
1. The Starting Point: What Is a Language Model
Before discussing the history, we need to define the object. From an information-theoretic perspective, a language model is a function that models the joint probability (P(w_1,\dots,w_T)) of a word sequence (w_1, w_2, \dots, w_T). By the chain rule:
P(w₁, w₂, ..., w_T) = P(w₁) × P(w₂ | w₁) × P(w₃ | w₁, w₂) × ... × P(w_T | w₁, ..., w_{T-1})
In practice, training is usually simplified to predicting the next word:
import torchimport torch.nn.functional as Fdef next_token_loss(logits, targets):"""Compute the training loss for "predicting the next token".logits : model output with shape (batch, seq_len, vocab_size)targets: ground-truth token IDs with shape (batch, seq_len), i.e. the input shifted by one"""return F.cross_entropy(logits.reshape(-1, logits.size(-1)),targets.reshape(-1),ignore_index=-100, # padding positions are excluded from the loss)
This objective may look simple, yet it runs through the entire history from n-gram statistical language models to today’s large language models. The difference lies in the span of context and the parameter capacity: early models looked at only the previous one or two words, while today’s models can attend to thousands or even tens of thousands of words at once, with hundreds of billions of parameters.
So the key question for understanding the history of large models has always been the same: how far can the context reach, how much can the model remember, and can the computation run in parallel. The technical evolution below breaks through these three dimensions layer by layer.
2. The Biomimetic Foundation: Starting from 86 Billion Neurons
2.1 What the Brain Taught Us
The human brain has roughly 86 billion neurons, connected into a complex network through about 100 trillion synapses. A neuron receives input signals, fires a spike when the input crosses a threshold, and transmits it through synapses. This structure holds three profound lessons:
- Distributed representation: information is not stored in a single neuron but distributed across the strengths of massive numbers of connections;
- Plasticity: synaptic connections change with experience—the essence of “learning” is adjusting connection weights;
- Circuits and feedback: recurrent connections are pervasive in the nervous system; outputs can be fed back as inputs, sustaining ongoing activity.
Machine learning draws precisely on these three points:
| Brain mechanism | Corresponding concept in machine learning |
|---|---|
| Synaptic connection strength | Model parameters (weights) |
| Experience accumulation and synaptic plasticity | Gradient updates based on data and a loss function |
| Neuron activation | Activation functions (step, Sigmoid, ReLU, etc.) |
| Recurrent connections | Recurrent neural networks, autoregressive structures in attention |
| Multi-region coordination | Multi-branch structures such as multi-head attention and mixture of experts |
It is worth noting that this borrowing is heuristic. The feedforward, self-attention architecture used by today’s large language models already differs substantially from the biophysical mechanisms of the brain. A more meaningful analogy lies in the learning paradigm: humans acquire expressive ability through massive language input, while large models complete pretraining through massive text corpora.
2.2 The Scale Gap and Computational Simulation
In terms of scale, the total parameter count of mainstream large models (typically in the hundreds of billions) is on the same order of magnitude as the number of synapses in the brain, and their performance on many language tasks is increasingly approaching—and on some benchmarks exceeding—the human average. But a similar order of magnitude does not mean the same mechanism:
- The brain runs on about 20 watts; training a large model relies on compute clusters built from large-scale accelerator cards;
- The brain has continual learning and common-sense reasoning; large models can still hallucinate and struggle to update their knowledge autonomously.
A more accurate framing is therefore this: a large model is a large-scale engineering realization of the process of “extracting patterns from data,” not a replica of the human brain. Let us begin with the earliest biological and mathematical observations.
3. Theoretical Seeds (1900–1950): Three Foreshadowings
3.1 The Discovery of Recurrent Structures
Around 1901, the Spanish histologist Santiago Ramón y Cajal, studying the cerebellar cortex, found that Purkinje cells, granule cells, and others connect to one another through axons and dendrites, forming several closed loops. He proposed the concept of an “excitation loop”: a neural impulse can travel repeatedly around a loop, allowing activity to sustain itself over a period of time.
This discovery broke the prevailing belief that the nervous system had only one-way afferent and efferent pathways, and it was the first biological confirmation of the existence of recurrent connections. Decades later, recurrent neural networks became the engineering continuation of exactly this idea.
3.2 Markov Chains: The Mathematical Starting Point of Sequence Modeling
In 1906, the Russian mathematician Andrey Markov, studying the patterns of vowels and consonants in Pushkin’s Eugene Onegin, proposed the Markov chain. Its core assumption is:
The system’s state at the next moment depends only on its current state, not on any earlier history—the “memorylessness” property.
This seemingly simple assumption gave sequence data its first rigorous mathematical framework. Later n-gram statistical language models were built on it: a trigram model predicts the next word from the previous two, estimating the conditional probability (P(wt \mid w{t-1}, w_{t-2})) by counting word frequencies over a large corpus.
The Markov chain contributed two things: first, the idea of “describing language generation with conditional probabilities”; second, the notion that “historical information can be compressed into a finite state.” The latter can still be seen in recurrent neural networks and in the KV cache of the Transformer. Its limitation is equally clear: constrained by the curse of dimensionality, n cannot be too large; the model cannot capture long-range dependencies, nor generalize to word combinations unseen during training.
3.3 The McCulloch-Pitts Neuron Model
In 1943, the neurophysiologist Warren McCulloch and the logician Walter Pitts published A Logical Calculus of the Ideas Immanent in Nervous Activity. They proposed a highly simplified neuron model, later known as the M-P model:
- Receive multiple inputs, multiply each by a weight (w_i), sum them, and add a bias (b);
- Compare the weighted sum against a threshold and use a step function to decide an output of 0 or 1.
x₁ ──w₁──┐x₂ ──w₂──┤ Σ + b ──→ [ threshold check ] ──→ y ∈ {0, 1}... │xₙ ──wₙ──┘
This was the first artificial neural network model, and it proved in mathematical terms the equivalence between a “neuron” and a “logical operation.” Its limitation was that the weights had to be set manually, with no way to learn them automatically. This gap was filled 15 years later by the perceptron.
3.4 The Turing Test: A Standard for Evaluating Intelligence
In 1950, Alan Turing published Computing Machinery and Intelligence, opening with the question: “Can machines think?” Sidestepping the difficulty of defining “think,” he proposed what became known as the Turing test—the imitation game:
- The evaluator converses with a machine and a human purely through text;
- If the evaluator cannot reliably distinguish which is which, the machine is judged to exhibit intelligence.
The influence of this idea far exceeded expectations at the time. It established the fundamental paradigm of “evaluating intelligence through natural-language understanding and generation,” and made language processing one of the most central research directions in artificial intelligence. Today’s evaluations of large models (dialogue quality, writing ability, knowledge Q&A, and so on) are, in spirit, a continuation of this paradigm.
4. The Birth of the Field and Early Neural Networks (1950–1980)
4.1 The Birth of Artificial Intelligence
In 1955, John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon co-drafted a conference proposal. To avoid the disciplinary narrowness implied by names such as “automata studies” or “complex information processing,” McCarthy formally named the field “Artificial Intelligence.” In the summer of 1956, these scholars held a workshop of about two months at Dartmouth College in the United States, discussing “whether machines can simulate intelligence.” This meeting is widely recognized as the birth of AI as an independent discipline.
Notably, the proposal included a dedicated section on “natural language,” envisioning programs that could communicate with people in human language. The problem of language has been at the core since the field’s inception.
4.2 The Perceptron and the “Father of Deep Learning”
In the late 1950s, Frank Rosenblatt proposed the perceptron. It is essentially “an M-P neuron that can learn”: given an input and a desired output, if the prediction is wrong, it adjusts the weights according to the error:
w_i ← w_i + η × (y_true - y_pred) × x_i (η is the learning rate)
This rule was inspired by the Canadian neuropsychologist Donald Hebb. Hebbian theory holds that “neurons that fire together wire together,” and the perceptron’s weight update is the mathematical realization of exactly this idea.
The perceptron’s key contribution was the learning rule: weights no longer had to be set manually but were obtained automatically from data. In the “perceptron convergence theorem,” Rosenblatt proved that as long as the data are linearly separable, the perceptron’s learning algorithm is guaranteed to converge in a finite number of steps. This laid the groundwork for statistical learning theory.
In 1960–1961, he went further and proposed a multilayer perceptron with “closed-loop cross connections,” letting the output of one layer feed back to influence the next. In 1962, his book Principles of Neurodynamics systematically summarized this body of theory. For pioneering learnable neural networks, Rosenblatt is remembered by later generations as the “father of deep learning.”
The first low point: In 1969, Minsky and Seymour Papert rigorously proved in their book Perceptrons that a single-layer perceptron cannot solve problems that are not linearly separable, such as XOR:
XOR truth table:x₁=0, x₂=0 → 0x₁=0, x₂=1 → 1x₁=1, x₂=0 → 1x₁=1, x₂=1 → 0 (positive and negative cases alternate in the plane and cannot be separated by a single straight line)h₁ = 1 if (x₁ + x₂ ≥ 1) else 0 # detects "at least one is 1"h₂ = 1 if (x₁ + x₂ ≥ 2) else 0 # detects "both are 1"y = h₁ - h₂ # output layer sums: the result is 1 exactly when one is 1
This construction shows that introducing a hidden layer lets a perceptron network express XOR. But at the time, multilayer networks lacked an effective training method, and research interest declined accordingly. For the next decade or so, neural network research fell into a trough, and the center of gravity shifted to symbolicism and expert systems.
4.3 The Hopfield Network: Associative Memory Through “Energy”
In 1982, the physicist John Hopfield borrowed the spin-glass model from statistical physics to propose a fully connected, symmetric-weight recurrent neural network—the Hopfield network. It defines an energy function:
E = -1/2 × Σᵢ Σⱼ wᵢⱼ · sᵢ · sⱼ (sᵢ ∈ {-1, +1} is the state of neuron i)
Its dynamics resemble “going downhill”:
- A partial or noisy input is loaded as the initial state;
- Neurons update asynchronously by a rule, and each step never increases the system’s energy;
- It eventually settles at a local minimum—one stored pattern of memory.
So even with incomplete input, the network converges to the nearest stored pattern, achieving content-addressable storage. This is an early engineering prototype of the human “see one, infer ten” style of associative memory, and it made the intuition that “a memory can be seen as an attractor” an important idea in neural networks. But the Hopfield network has limited storage capacity (about 0.15 times the number of neurons) and can easily fall into unintended local minima.
5. Recurrent Neural Networks and Their Improvements: Teaching Models to “Remember”
5.1 Backpropagation and the Formal Introduction of RNNs
To make multilayer networks usable, one had to solve the problem of “how to distribute error across depth.” In 1986, Geoffrey Hinton, David Rumelhart, and colleagues published a paper in Nature that systematically applied the backpropagation algorithm to multilayer neural networks. Its core idea is to propagate error back layer by layer using the chain rule:
# Gradient sketch for a two-layer network (simplified theoretical model)z1 = W1 @ x + b1 # hidden-layer linear transformh = relu(z1) # hidden-layer activationy_hat = W2 @ h + b2 # output layerloss = CE(y_hat, y_true)# Backpropagation: differentiate layer by layerdL_dz1 = (W2.T @ dL_dy_hat) * (z1 > 0)dL_dW1 = dL_dz1 @ x.TdL_dW2 = dL_dy_hat @ h.T
Around the same time, backpropagation was also used to train networks with recurrent connections. At this point, the recurrent neural network (RNN) formally took shape: it replicates the same feedforward network along the time axis, feeding each moment’s output in as the next moment’s input.
Before unrolling Unrolled over timex₁ → [ A ] → h₁ x₁ → [A]→ h₁ → [A]→ h₂ → [A]→ h₃↑ h₁ x₂ ↗ ↑ x₂ ↗ ↑ x₃x₂ → [ A ] → h₂↑ h₂x₃ → [ A ] → h₃(A is the recurrent unit with shared parameters; vertical dashed lines indicate gradient backpropagation along the time direction)
RNNs gave a model, for the first time, a variable memory length: the hidden state is a vector continually updated by new inputs, and in theory it can be as long as you like.
Training method: Around 1989, Paul Werbos and Richard Zipser and others systematically studied the “backpropagation through time” (BPTT) algorithm. It treats a recurrent network as a feedforward network unrolled along time and then computes gradients with standard backpropagation, becoming the standard method for training sequence models and still used today in time-series forecasting, speech processing, and more.
Theoretical limitation: In 1991, Sepp Hochreiter and others systematically analyzed the gradient behavior of recurrent networks and pointed out the fundamental difficulty of “capturing long-term dependencies”:
- Vanishing gradients: multiplying the derivatives of activation functions (Sigmoid has a maximum of just 0.25) decays distant information exponentially, so parameters barely get updated;
- Exploding gradients: when the product exceeds 1, gradients grow exponentially, causing numerical overflow and oscillating divergence.
A simplified summary is this: it is not that recurrent networks cannot remember long sequences, but that they find it very hard to learn to use long-range dependencies.
5.2 LSTM: Protecting the “Memory Channel” with Gates
In 1997, Hochreiter and Jürgen Schmidhuber proposed the Long Short-Term Memory network (LSTM), with a straightforward design rationale:
- If memory is passed along by an identity linear transform, the gradient product stays at 1 and memory does not vanish;
- But memory also needs to be updated and cleared, or the model cannot handle new information;
- The solution is to add “gates”: small neural networks that learn when to write, when to forget, and when to read out.
# Forward pass of a single LSTM time step (pseudocode)i_t = σ(W_i · [h_{t-1}, x_t] + b_i) # input gate: how much new information to writef_t = σ(W_f · [h_{t-1}, x_t] + b_f) # forget gate: how much of the old memory to keepo_t = σ(W_o · [h_{t-1}, x_t] + b_o) # output gate: how much of the hidden state to emitg_t = tanh(W_g · [h_{t-1}, x_t] + b_g) # candidate memory contentc_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t # cell state: the "conveyor belt" of memoryh_t = o_t ⊙ tanh(c_t) # hidden state: the output to the outside# where σ is the Sigmoid and ⊙ is element-wise multiplication
The key is the cell state (c_t): it propagates along time by “additive updates,” so gradients can flow back approximately unchanged, letting information travel across dozens or even hundreds of time steps.
Subsequent developments (1997–2014):
- In 1997, Schmidhuber’s doctoral student Mike Schuster proposed the bidirectional recurrent neural network (BRNN), processing the input sequence in both forward and reverse directions so that each position’s representation fuses left and right context;
- Around 1999, researchers added a forget gate to the LSTM, forming today’s standard structure;
- In 2014, Kyunghyun Cho and others proposed the Gated Recurrent Unit (GRU), simplifying the three gates into two (“update gate” and “reset gate”) and merging the cell state with the hidden state, reducing parameters and computation while achieving comparable performance;
- In the same period, the encoder-decoder framework became mainstream: one recurrent network compresses the source sentence into a fixed-length vector, and another generates the translation from it.
import torch.nn as nnclass Seq2Seq(nn.Module):def __init__(self, vocab_size, emb_size, hidden_size):super().__init__()self.embedding = nn.Embedding(vocab_size, emb_size, padding_idx=0)# GRU (gated recurrent unit); this only illustrates how the interface is usedself.encoder = nn.GRU(emb_size, hidden_size, batch_first=True)self.decoder = nn.GRU(emb_size, hidden_size, batch_first=True)self.proj = nn.Linear(hidden_size, vocab_size)def forward(self, src, tgt):enc_out, hidden = self.encoder(self.embedding(src))# The encoder's final hidden state becomes the decoder's initial state, i.e. the "fixed-length semantic vector" bottleneckdec_out, _ = self.decoder(self.embedding(tgt), hidden)return self.proj(dec_out) # shape: (batch, tgt_len, vocab_size)
5.3 The Bottleneck: Serial, Forgetting, and the Information Bottleneck
Although LSTM and GRU greatly alleviated the gradient problem, the recurrent architecture still faces three bottlenecks:
- No parallelism: step t must wait for step t−1 to finish, so parallel hardware such as GPUs is hard to fully utilize;
- Long-range dependencies remain fragile: after information is passed across dozens or hundreds of steps, it is inevitably attenuated or overwritten;
- The semantic bottleneck: the encoder-decoder framework compresses an entire source sentence into a single fixed-length vector, and as sentences grow longer, information is necessarily lost.
The crux is that the recurrent structure can never escape the “passing messages one step at a time” pattern. To break through, one must switch to a different path.
6. Attention and the Transformer: From “Sequential Passing” to “Global Interaction”
6.1 From Alignment Weights to Attention
In machine translation, statistical methods had long used “alignment” to describe the correspondence between words in the source and target languages. Once deep learning took hold, researchers wanted the model to learn these correspondences automatically.
In 2014, Dzmitry Bahdanau and colleagues proposed an elegant idea: rather than relying only on that fixed vector, the decoder would dynamically compute its “attention weights” over the hidden states at each encoder time step every time it generated a word. The higher the weight, the more important the corresponding source word is to the current translation decision. This is the attention mechanism.
Encoder outputs: h₁ h₂ h₃ ... hₙ (representations of each source position)│ │ │▼ ▼ ▼Decoder state s: ──→ alignment score eₜ = score(s, hᵢ)──→ normalization αₜ = softmax(eₜ)──→ context vector cₜ = Σ αₜᵢ · hᵢIntuitively: when translating "bank," the model pays more attention to "river" or "money" depending on context.
Intuitively, attention taught the model to “look things up on demand”: instead of compressing everything into one vector, it retrieves the needed information instantly from any position in the source sentence at each generation step. Machine translation quality rose sharply. Several subsequent studies showed that attention in dot-product, additive, and multiplicative forms performs roughly on par.
6.2 The Transformer: A Paradigm Shift in Architecture
In 2017, Ashish Vaswani and seven other researchers published Attention Is All You Need, proposing the Transformer, which discards recurrence and convolution entirely and stacks networks built only on attention.
① Self-attention. Q, K, V (query, key, value) are all linear projections of the same sequence’s representations, so any two positions in the sequence can interact directly:
import mathimport torchimport torch.nn.functional as Fdef scaled_dot_product_attention(q, k, v, causal=False):"""q, k, v: shape (batch, heads, seq_len, head_dim)causal: when True, apply a causal mask so autoregressive generation cannot "see the future""""scores = q @ k.transpose(-2, -1) / math.sqrt(q.size(-1))if causal:len_q, len_k = q.size(-2), k.size(-2)# Lower-triangular mask: row i may attend only to columns j ≤ i (aligned by offset during incremental decoding)causal_mask = torch.ones(len_q, len_k, dtype=torch.bool, device=q.device).tril()scores = scores.masked_fill(~causal_mask, float("-inf"))weights = F.softmax(scores, dim=-1)return weights @ v
- Scaling factor: dividing by (\sqrt{d_k}) prevents the dot product from becoming too large at higher dimensions, which would push the Softmax into a saturated region with tiny gradients;
- Multi-head attention: running h groups of attention with different projections in parallel lets the model attend to multiple relationships at once—syntax, coreference, semantics, and more—then concatenate and fuse them.
② Three key design choices:
- Positional encoding: self-attention carries no ordering information, so position must be injected via sinusoidal/cosinusoidal encodings or learnable vectors, supplying “coordinates” for global interaction;
- Residual connections and layer normalization: these stabilize deep training. Later practice showed that pre-normalization (Pre-LN), placing normalization before each sublayer, favors more stable convergence for large models;
- Feed-forward networks: position-wise two-layer multilayer perceptrons, typically paired with GLU-family activation functions, handle feature transformation and knowledge storage.
③ Complexity comparison. Why did the Transformer overtake the field? Compare the per-layer cost of each layer type for d-dimensional representations and sequence length n:
| Layer type | Per-layer complexity | Sequential operations | Maximum path length |
|---|---|---|---|
| Recurrent | O(n·d²) | O(n) | O(n) |
| Convolutional (kernel width k) | O(k·n·d²) | O(1) (1D) | O(log_k n) |
| Self-attention | O(n²·d) | O(1) | O(1) |
| Restricted self-attention (window size r) | O(r·n·d) | O(1) | O(n/r) |
The maximum path length of self-attention is 1: any two positions are separated by a single layer of interaction, so information need not be passed and attenuated layer by layer through recurrence. The price is complexity that grows quadratically with sequence length—which is exactly what later long-context research focused on.
④ From decoding to generation. The Transformer paper’s original two tasks were machine translation and English constituency parsing, using an “encoder-decoder” and an “encoder-only” structure respectively. Researchers soon realized that keeping only the decoder and enabling a causal mask turns the model itself into a powerful autoregressive generator whose training objective is exactly language modeling.
⑤ Influence. In 2017, the paper won the annual NeurIPS Best Paper Award; in 2024, two of its authors, Llion Jones and Aidan Gomez, jointly received the paper’s Test of Time Award. A 2023 survey found that roughly 70% of papers published in the field since 2022 involve the Transformer. It is fair to say that nearly every mainstream large language model today uses it as its core foundation.
6.3 The Bottleneck: Attention Is Not Free
- Quadratic complexity: doubling the sequence length roughly quadruples the attention computation;
- KV cache: during autoregressive generation, previously computed historical K and V can be cached and reused, but memory usage grows linearly with context length;
- Optimization techniques: around these two points, the industry has developed IO-aware chunked attention algorithms, Multi-head Latent Attention (MLA), Grouped-Query Attention (GQA), sliding windows, paged memory management, and more—see Section 9.
7. The Pretraining Paradigm and Scaling Laws: The Path to “General Capability”
7.1 Groundwork for the Pretraining Idea
Around 2013, distributed word-vector methods mapped each word to a low-dimensional dense vector, making it computable for the first time that “similar meaning means similar vectors.” In 2018, context-dependent word representations (epitomized by ELMo, based on a bidirectional recurrent language model) further demonstrated that the same word should have different representations in different contexts.
In parallel, the ImageNet competition in computer vision drove the paradigm of “general-purpose backbone plus task fine-tuning”: first train general features on large-scale data, then transfer them to downstream tasks. The NLP community began to ask: could the language domain replicate this path?
7.2 Establishing the Pretrain-Then-Finetune Paradigm
- Generative pretraining (early 2018): perform autoregressive language modeling on large amounts of unlabeled text, then fine-tune on downstream tasks. It showed that “pretrain first, then fine-tune” can significantly improve performance across many tasks.
- Bidirectional pretraining (late 2018): researchers proposed the “masked language model,” randomly masking some words in a sentence for the model to reconstruct, so every word could fuse left and right context; combined with a next-sentence-prediction task to learn relationships between sentences, it set a new state of the art on many understanding-oriented tasks at the time.
Input: Today [MASK] is nice, let's go to the [MASK] togetherTarget: Today the weather is nice, let's go to the park togetherAdvantage of bidirectional pretraining: when predicting "park," the representation of "go" has already fusedthe right-side context, carrying more information than left-to-right prediction alone.
This established the “pretrain then fine-tune” paradigm:
- The model first learns general language patterns and world knowledge from ultra-large-scale unlabeled corpora, then makes a small adjustment for a specific task;
- Compared with “training from scratch,” it greatly lowers the barrier and cost of task development;
- Around 2019, research further unified a wide range of tasks into a “text-to-text” form, making pretrained models even simpler to use.
Its essence is the success of “unsupervised pretraining.” This is highly symbolic in the “three rises and three falls” of deep learning:
| Period | Dominant paradigm | Milestones and outcomes |
|---|---|---|
| Early stage (1943–1958) | Supervised perceptron | Perceptron convergence theorem; fell into a trough in 1969 over problems like XOR |
| First revival (1986–1995) | Shallow backpropagation | Recurrent and convolutional networks emerged, but limited by data and compute, they briefly faded |
| Second revival (from 2012) | Large-data supervised learning | Deep networks broke through in visual recognition; NLP used “task-specific architectures plus supervised data” |
| Current stage (from 2018) | Large-scale unsupervised pretraining | Massive unlabeled text drives the emergence of general capability; self-supervised learning becomes the main line |
The significance of self-supervised learning is this: it turns “massive unlabeled text” into a training resource. Trillions of tokens exist on the internet—a data scale the old paradigm of manual labeling could never match.
7.3 Scaling Laws: From Experience to Predictability
In 2020, Jared Kaplan, Sam McCandlish, and others at OpenAI published scaling law research: without overfitting, a model’s cross-entropy loss shows a smooth power-law relationship with parameter count, data volume, and compute:
# Idealized empirical relationships (illustrative; dozens of fitting forms exist in practice)L(N) ≈ (N_c / N)^α_N # N: parameters, α_N ≈ 0.076L(D) ≈ (D_c / D)^α_D # D: training tokens, α_D ≈ 0.095L(C) ≈ (C_c / C)^α_C # C: compute, α_C ≈ 0.050 (α_C ≈ α_N × α_D)
This means loss reduction is predictable: as long as you increase investment, you can reliably estimate the improvement. OpenAI’s team “bet” on this, using about $34 million to train GPT-3 with 175 billion parameters, igniting a race toward large-scale pretraining.
In 2022, a DeepMind team further revised the “parameter-to-data ratio” conclusion: under a given compute budget, model parameters and training tokens should scale up at roughly the same ratio (empirically about 20 tokens per parameter), rather than piling on parameters alone; a well-trained smaller model is often more economical at inference. Many later open models follow this principle—for example, training with about 2 trillion tokens on 7 billion parameters. This conclusion matters especially to cost-sensitive enterprises:
- Capacity out of balance with data: too many parameters and too little data leads the model to “memorize by rote”;
- Too few parameters and too much data: the data goes to waste.
In engineering practice, ratios are usually set accordingly and then dynamically adjusted based on the convergence curve.
7.4 In-Context Learning: Task Switching Without Updating Parameters
Another breakthrough GPT-3 brought is in-context learning: by giving a few input-output examples in the prompt, the model can complete a new task without updating any parameters:
[Few-shot prompt example (illustrative)]Decide the sentiment of the comment below; output only "positive" or "negative".Comment: This phone's battery life is way better than I expected!Sentiment: positiveComment: The shipping was too slow; it took a week to arrive.Sentiment: negativeComment: The image quality is average, but the price really is cheap.Sentiment:
This ability is closely tied to scale: in the scaling-law study, smaller models performed near random guessing on zero-shot and few-shot tasks, while models with around 100 billion parameters improved markedly under few-shot conditions. This is often cited as a typical example of “emergent abilities.”
However, some research points out that certain “emergent” phenomena may stem from discontinuities in the evaluation metrics (such as exact match or all-or-nothing accuracy); with continuous metrics, the curves are often far smoother. The lesson for evaluation design is this: the choice of metric directly affects how we judge the trend in a model’s capabilities.
7.5 Sparsification: Making Parameter Scale More “Economical”
For a dense model, every added parameter participates in the computation for every token. The sparse gating idea of Mixture of Experts (MoE)—proposed in 2017 and validated at scale in trillion-parameter models around 2021—changed this:
- Split the feed-forward layer into multiple parallel “expert” networks;
- A gating network selects a few (say, the top 2) experts for each token;
- Each token passes only through the selected experts, then is combined by a weighted sum of the gating weights.
Token t → gating network → select Top-2 experts → Expert₂ and Expert₅ compute → combine by gating weights
Effect: the total parameter count grows substantially while the parameters activated per token stay relatively limited. This decouples model capacity from compute cost, making it one of today’s important architectures.
The costs are memory usage and load-balancing issues: if some experts are over-selected, the rest are undertrained; and if tokens are distributed unevenly across devices, overall training slows down. Engineering practice typically adds an auxiliary load-balancing loss along with strategies such as expert parallelism.
8. The Alignment Era: Making Models “Speak Human”
8.1 The Problem: Capability Is Not the Same as Usability
After pretraining, a base model is essentially a “text continuer”: faced with a question like “How can I quickly brew a good cup of coffee?”, it may continue with more related questions instead of giving a direct answer. This is because the pretraining objective is “predict the next word,” while humans expect “follow instructions and give a helpful, harmless answer.” A mismatch of objectives calls for alignment.
8.2 Reinforcement Learning from Human Feedback (RLHF)
In early 2022, OpenAI released the InstructGPT paper, systematically validating the “reinforcement learning from human feedback” (RLHF) route, which became the technical foundation for conversational large models. The process has three steps:
Step 1: Supervised fine-tuning (SFT)Collect human demonstrations: for many prompts, annotators write ideal answersFine-tune the pretrained model with supervised learning so it first learns to "answer on instruction"Step 2: Reward model (RM)For the same prompt, have the model generate several candidate answers and let annotators rank themTrain a reward model to learn the human preference standard:loss = -log( σ( r(x, y_win) - r(x, y_lose) ) ) # preference ranking lossStep 3: Reinforcement learning optimization (Proximal Policy Optimization, PPO)Use the reward model to score the policy model's (i.e. the current large model's) outputsBalance "maximizing reward" against "not deviating too far from the reference model":max E[ r(x, y) - β × KL(π(y|x) || π_ref(y|x)) ]where β controls the strength of the deviation constraint and the KL term prevents the model from "gaming the reward model"
Design considerations:
- Ranking is more reliable than scoring: asking people for an absolute score tends to be inconsistent, while comparing two answers is easier to keep consistent;
- The reward model generalizes well: it only needs to learn preference patterns from human comparisons, without re-labeling every new task;
- The KL penalty is indispensable: without this constraint, the model may produce lengthy, sycophantic outputs—“rewarded by the reward model but not actually liked by people”—the so-called “reward hacking.”
The conversational large model that arrived in December 2022 achieved a smooth multi-turn dialogue experience through this combination of techniques, and set off a wave of applications in generative AI.
8.3 Follow-up Progress in Alignment
- Preference optimization without reinforcement learning: such as Direct Preference Optimization (DPO), which turns preference learning directly into a classification-style loss, removing the reward model and online RL step for more stable training;
- Reasoning reinforcement: using automatic feedback from verifiable answers (such as math and code), combined with rule-based rewards or process reward models, to strengthen the model’s long-chain reasoning;
- Constitutional AI: using a set of written principles (the “constitution”) to guide the model to critique and revise its own answers, reducing reliance on human labeling.
It bears emphasizing that RLHF improves “usefulness and controllability” but does not eliminate hallucination at its root. So “retrieval-augmented generation plus fact-checking plus evaluation monitoring” remains indispensable.
9. The Current Engineering Focus (From 2023 Onward)
The basic paradigm of pretrain-finetune-align has stabilized, and the focus of competition has shifted to three directions: how to use models well, use them reliably, and afford them.
9.1 Long Context
The mainstream means of extending the context window and improving long-text utilization include:
- Rotary Position Embedding (RoPE): injecting position information into the attention computation in the form of a rotation matrix;
- Position interpolation and methods like YaRN: scaling or adjusting position encodings beyond the training length to reduce performance loss during extrapolation;
- Attention optimization: chunked attention, sparse attention, and hybrid architectures based on linear attention or state space models;
- KV cache optimization: Grouped-Query Attention (GQA) and Multi-head Latent Attention (MLA), which reduce cache size.
But a longer window does not mean effective use. Existing research (such as “needle in a haystack” evaluations) shows that models are quite sensitive to where information sits: key information in the middle of the input is more easily overlooked. In practice, teams often use “context engineering”: through structured formatting, placing or repeating key information at the top, and summarizing/compressing, so that truly important information lands where the model uses it more readily.
9.2 Inference Cost and Efficiency
| Optimization direction | Typical methods | Benefit | Main cost / caveat |
|---|---|---|---|
| Quantization | Low-bit representation of weights and activations (e.g. INT8/INT4), with group scaling and per-channel calibration | Substantially lower memory use and bandwidth demand | Possible accuracy loss; must be validated by evaluation |
| Sparsification | MoE, weight pruning | Lower compute per token | Higher memory and scheduling complexity |
| Distillation | Using a large model as a teacher to guide a small model | The small model gains capabilities close to the large one | Requires output data from the teacher model |
| Speculative decoding | A small draft model proposes candidates in batches; the large model verifies them in parallel | Reduces the cost of generating one by one | Complex to implement; must preserve the output distribution |
| Dynamic batching | Continuous batching, paged memory management (KV cache paged on demand) | Higher throughput, lower memory fragmentation | Scheduling and preemption policies must be carefully designed |
Inference cost can be estimated with a simple rule of thumb (ignoring the attention part):
Inference cost ≈ parameters × bytes per parameter × number of output tokens# Example (for estimation only): a 7-billion-parameter BF16 model at 2 bytes per parametermemory_GB = 7e9 * 2 / 1024**3 # about 13 GB (weights only)# Compute for generating about 512 tokens in a single request (forward pass is about 2N FLOPs/token)flops = 2 * 7e9 * 512 # about 7.2 × 10^12 FLOPs# If the context is 4096, attention compute estimated from Q@Kᵀ and Attention@Vn, d, v = 4096, 7e9, 50257 # d is the total dimension, v is the vocabulary sizeattn_flops = 4 * n * n * d # about 4.7 × 10^14 FLOPs, already dominant
You can see that the attention overhead of long contexts quickly becomes the main cost. Separating and optimizing “batch inference throughput” from “single-request latency” is key to service design. At the same time, model selection can be tiered: routing high-frequency, simple requests to small models and complex requests to large ones.
9.3 Multimodality and Agents
- Multimodality: text, images, audio, and video are increasingly modeled in a unified token space, and end-to-end “input in, output out” architectures are becoming mainstream;
- Agents: connecting to databases, search, and business APIs through function calling, combined with multi-step planning and reflection mechanisms to complete complex tasks;
- Protocols and evaluation: unifying model-invocation protocols and standardizing tool calling lowers integration costs; evaluation systems around planning, tool use, and long-horizon task completion are being steadily improved.
9.4 Infrastructure Support
Bringing large models into production requires a full stack of supporting infrastructure:
- Data layer: object storage holds training corpora, model weights, and version snapshots; message queues decouple data pipelines; lakehouse solutions support data cleaning and version management;
- Compute layer: containerized platforms schedule GPU resources uniformly, supporting elastic scaling and task isolation; distributed training frameworks handle gradient synchronization and memory optimization across multiple machines and cards;
- Observability layer: logging, monitoring and alerting, and tracing cover issues such as stalled training, memory overflow, and inference-latency jitter;
- Security and compliance: content safety filtering, data masking, access auditing, and traceability of model and data versions.
Taking Baidu AI Cloud as an example, its general-purpose capabilities span categories such as object storage, message services, container engines, logging, monitoring and alerting, and model service platforms. Enterprises can build a full “data-training-inference-operations” pipeline on this basis, combining compute, storage, and model capabilities as needed and avoiding reinventing the wheel.
10. Patterns of Evolution and Practical Advice
10.1 Four Patterns of Evolution
- What is represented: from symbols (words, n-grams) to dense vectors (word vectors), then to context-dependent representations (pretrained models)—models gradually shed the rigidity of “one word, one meaning.”
- Modeling structure: from the serial dependence of recurrence to the global, parallel interaction of attention—efficiency and information utilization leap forward together.
- Training signal: from statistical counting and manual labeling to self-supervised pretraining and human feedback—the scale and quality of training data rise together.
- Scale and efficiency: from piling on parameters to piling on data, then to sparsification and inference optimization—the focus of competition shifts from “can it be bigger” to “can it be cheaper.”
10.2 Practical Advice for Developers
- Let evaluation drive iteration: build an evaluation set that covers real business scenarios; keep metrics as continuous and fine-grained as possible to avoid being misled by a single metric;
- Retrieve first, then fine-tune, then align: inject business knowledge primarily through retrieval-augmented generation, and nail down behavioral style and format constraints through fine-tuning and alignment;
- Design for cost up front: work backward from throughput/latency targets to model size and context length, and make good use of tiered routing and inference optimization;
- Close the loop on hallucination: form a loop through citation and provenance, fact-checking, human review for sensitive scenarios, and feeding online feedback back into the system;
- Keep architecture in mind: understanding how attention and positional encoding work lets you make sound trade-offs on long context, KV cache, and memory optimization.
Conclusion
From the neural circuits drawn by Cajal in 1901 to the trillion-parameter large language models of 2026, across more than 120 years the main thread of technology has stayed clear: simulating “learning from experience” in ever more efficient ways.
- McCulloch and Pitts gave the neural network its skeleton;
- Rosenblatt taught it to adjust itself;
- Hochreiter and Schmidhuber gave it a memory of variable length;
- Attention taught it global interaction;
- Pretraining let it absorb vast knowledge;
- Human feedback taught it to work with people.
The future of large models will most likely continue along this “biomimetic journey”: closer to human-like efficient learning (such as continual and sample-efficient learning), more trustworthy reasoning, and more natural multimodal interaction. For developers, rather than chasing every new concept, it is better to build systematic understanding along this main thread: knowing why each technique appeared, what it solved, and what new problems it introduced—only then can you make sound engineering decisions amid a fast-iterating technology wave.
