Seven Years of LLM Architecture Evolution: A 2025 Technical Deep Dive into Transformer Design
This article traces how large language model architecture has evolved from the first decoder-only models around 2019 to today's flagship open-source models. Rather than benchmark scores, it focuses on the architectural modules themselves—positional encoding, normalization, attention, feed-forward networks, prediction targets, and engineering choices—explaining the motivation behind each change and its real impact on inference memory, compute, and throughput. It closes with practical guidance for developers on model selection, memory and throughput estimation, and common misconceptions, making it a useful reference for anyone choosing or deploying an LLM.
Introduction: What Has Changed in LLM Architecture Over Seven Years?
Around 2019, the first batch of decoder-only autoregressive language models appeared. Seven years have passed since then. If you place an early model’s structure diagram next to a flagship open-source model’s diagram from today, the first impression is often a sense of familiarity: both use the Transformer decoder as their backbone, stacking self-attention modules and feed-forward network modules in alternation, with residual connections running throughout.
Yet the structural details have changed significantly. For example:
- Absolute positional encoding has gradually given way to rotary positional encoding (RoPE);
- Multi-head attention has largely given way to grouped-query attention (GQA);
- Gated activation structures like SwiGLU have replaced the classic GELU feed-forward network.
This raises a question worth pondering: with such limited changes, are we seeing an architectural breakthrough, or a continuation of the ongoing refinement of an existing architectural foundation?
Methodologically, answering whether a particular architectural change “works” is genuinely difficult. Different models differ enormously in training data, data mixtures, training strategies, and hyperparameters, and these are often not fully disclosed. Even when a technique shows clear ablation results in one paper, it is hard to guarantee it holds under another team’s data and compute conditions.
Still, “ranking” is not the only goal. Examining structural changes in the architecture itself offers at least two kinds of value:
- Understanding trends: seeing which problems current model designers care about most, such as inference cost, long context, and training stability;
- Guiding implementation: architectural decisions directly affect inference-time memory footprint, computation, and throughput—practical constraints that developers cannot avoid when selecting and deploying a model.
For these reasons, this article sets aside benchmark scores and training algorithms and focuses on the architectural modules themselves, tracing the evolution from early decoder models to today’s flagship open-source models.
Reading tip: This is a long article. Readers short on time can start with the overview comparison table in Chapter 1, then jump to the relevant chapter based on their deployment or training needs.
1. The Architecture at a Glance: The Basic Skeleton and Six Key Changes
A typical decoder Transformer can be broken down into three parts:
- Embedding layer: maps input tokens into vectors;
- Stacked Transformer blocks: each block typically contains a self-attention sub-layer and a feed-forward network (FFN) sub-layer, both wrapped in residual connections;
- Output layer: after a final normalization, a linear layer maps to the vocabulary to produce the probability distribution over the next token.
From early models to today’s flagship open-source models, the real changes are concentrated inside these three parts:
| Architecture Module | Early Models (~2019) | Current Mainstream (2024–2025) | Primary Motivation |
|---|---|---|---|
| Positional encoding | Learnable absolute position vectors added to word embeddings | Rotary positional encoding (RoPE), combined with length-extrapolation methods | Length generalization, relative position modeling |
| Normalization | LayerNorm after each sub-layer | RMSNorm before each sub-layer; some models add Q/K normalization | Training stability, computational efficiency |
| Attention | Standard multi-head attention (MHA) | Grouped-query attention (GQA), sliding-window hybrid attention, low-rank compression | Reducing KV cache, extending context |
| Feed-forward network | Two-layer FFN with GELU activation | SwiGLU gated structure and sparse mixture-of-experts (MoE) | Higher expressiveness at equal compute |
| Prediction target | Single-token next-word prediction | Auxiliary objectives such as multi-token prediction (MTP) | Denser training signal, faster inference |
| Normalization placement | Post-sub-layer normalization | Pre-sub-layer normalization; some models drop output normalization | Cleaner residual path |
These changes roughly unfold along two main lines:
- The efficiency line: minimizing inference overhead under the same compute budget, especially memory and memory-bandwidth overhead;
- The capability line: supporting longer contexts, more stable ultra-large-scale training, and higher data efficiency.
The sections below cover each item in order: positional encoding → normalization → attention → feed-forward networks → prediction targets → engineering.
2. Positional Encoding: From Absolute Position to Rotary Position Encoding
2.1 Why Positional Encoding Is Needed
The self-attention mechanism is inherently insensitive to input order. Without position information, shuffling the input order leaves the resulting set representation unchanged. To model word order and syntactic structure, a language model must inject position into the model in some way.
The early approach was intuitive and simple:
- Learn a position vector for each position and add it to the word vector;
- Assign each word an independent, trainable parameter.
This works well within the training length, but it has two inherent shortcomings:
- Poor extrapolation: beyond the training length, the model has no corresponding position vector available;
- Unstable extrapolation behavior: even if the position table is forcibly extended, position vectors beyond the training range lack meaningful gradient signals, so the model’s generalization to unseen lengths is limited.
Subsequent research therefore turned to two directions: one is position bias whose attention scores decay naturally with distance; the other is positional encoding that does not depend on learnable parameters and has well-defined mathematical properties. Rotary positional encoding belongs to the latter category and has gradually become the de facto mainstream choice.
2.2 The Core Idea of Rotary Positional Encoding
The key idea behind RoPE is that instead of adding a position vector to the input, it applies a rotation matrix that depends only on position to the query vector q and key vector k within each attention head.
Its core property can be described with a concise relationship:
⟨R(m)·q, R(n)·k⟩ = g(q, k, m − n)
Here R(m) denotes the rotation for position m, and the angle brackets denote the dot product of two vectors. The meaning of this equation is that the dot product of the rotated query and key depends only on their relative distance m − n, independent of absolute position.
This design brings three benefits:
- Attention scores inherently carry relative-position semantics, making it easier for the model to learn locality and distance decay;
- No additional trainable position parameters are required;
- The computation is just element-wise multiply-add operations, so the overhead is very small.
Concretely, RoPE pairs up the dimensions of each attention head two by two, treating each pair as a 2D vector. For a token at position m, its i-th dimension pair is rotated by an angle m·θᵢ, where the frequency θᵢ = base^(−2i/d) and d is the head dimension. The base is commonly chosen between 10,000 and 1,000,000.
Different dimensions correspond to different frequencies:
- Low dimensions: high frequency, fast rotation, suitable for capturing local, fine-grained relative positions;
- High dimensions: low frequency, slow rotation, suitable for capturing long-range relationships.
This multi-scale structure is in the same spirit as the multi-resolution ideas in signal processing.
import torchdef apply_rope(x, positions, base=10000.0):"""x: Q or K tensor of shape [batch, num_heads, seq_len, head_dim]positions: position indices of shape [seq_len]"""d = x.shape[-1]assert d % 2 == 0 # dimensions are paired two by two# rotation frequency for each dimension pairinv_freq = base ** (-torch.arange(0, d, 2, device=x.device).float() / d)angles = torch.outer(positions.float(), inv_freq) # [seq_len, d/2]cos, sin = angles.cos(), angles.sin()# pair up dimensions as the x, y components of 2D vectorsx_even, x_odd = x[..., 0::2], x[..., 1::2]x_rot_even = x_even * cos - x_odd * sinx_rot_odd = x_even * sin + x_odd * cosreturn torch.stack([x_rot_even, x_rot_odd], dim=-1).flatten(-2)
RoPE is position-dependent and is usually applied to Q and K only inside the attention module. The feed-forward network is inherently position-agnostic, so there is no need to apply it again there.
2.3 Long-Context Extrapolation: Letting the Model “See Farther”
RoPE performs excellently within the training length, but when extrapolated directly to longer sequences, the attention distribution may flatten out and effective information can be diluted. In practice, three common approaches are used, and they are often combined:
Position-interpolation methods: These scale the rotation angles, effectively “compressing” longer positions into the angle range seen during training. The most naive linear interpolation scales all positions by the ratio of target length to training length, which also affects the low-frequency dimensions—precisely the ones that carry long-range information. As a result, the industry has developed frequency-segmented improvements: high-frequency dimensions are kept as-is where possible, and only the low-frequency dimensions are scaled proportionally.
Extrapolation-type methods: For example, NTK-aware base-frequency adjustment increases the base to slow down the rotation, keeping the model within a relatively gentle range of angle changes even at longer positions. There are also temperature-scaling schemes that further distinguish dimension categories, adjusting the attention temperature to prevent attention from becoming overly diffuse on ultra-long sequences.
Continued long-context training: Beyond architectural-side adjustments, models are simply trained further on long-text corpora. This is currently the most reliable component for supporting ultra-long contexts, and the first two categories of methods usually serve as its “initialization” conditions.
2.4 Other Technical Routes
It is worth noting that position bias is not limited to the rotary approach:
- Linear position bias: adding a penalty term to the attention scores that decreases monotonically with relative distance—simple in structure and predictable in extrapolation behavior;
- Removing explicit positional encoding entirely: relying on the hierarchical stacking of causal masks to implicitly convey position information, which also achieves decent results in some scenarios.
These routes each have trade-offs, but among today’s open-source flagship models, RoPE and its length-extrapolation extensions remain the most widely adopted combination.
3. Normalization and Residual Connections: The Cornerstone of Training Stability
3.1 From LayerNorm to RMSNorm
Early models commonly used layer normalization (LayerNorm), whose computation involves three steps: computing the mean, computing the variance, and scaling. RMSNorm simplifies this by removing the “subtract mean” step and the variance computation, keeping only scaling based on the root mean square:
RMSNorm(x) = x / sqrt(mean(x²) + ε) × γ
def rmsnorm(x, gamma, eps=1e-6):# compute root mean square along the last dimensionrms = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps)return x * rms * gamma
The removed steps are not merely redundant: in large-scale training, the normalization operator involves cross-element reduction operations, and eliminating one full reduction means fewer memory accesses and higher hardware utilization. At the same time, preserving each dimension’s original sign and only unifying the overall scale also makes gradient behavior smoother.
3.2 Pre-Normalization and Post-Normalization
A second important change is the placement of normalization. Early models mostly placed normalization after the sub-layer:
h = LayerNorm(x + Sublayer(x))
This is often called post-normalization (Post-LN). Its problem is that normalization comes after the residual addition, so during backpropagation gradients must pass through the normalization layer, and deep networks are prone to instability early in training, usually requiring techniques such as learning-rate warmup.
The current mainstream approach is pre-normalization (Pre-LN):
h = x + Sublayer(Norm(x))
class PreNormBlock(torch.nn.Module):def __init__(self, norm, attn, ffn, num_heads, d_model):super().__init__()self.norm1, self.norm2 = norm(d_model), norm(d_model)self.attn, self.ffn = attn, ffn # attn returns (output, KV cache)def forward(self, x, past_kv=None):a, new_kv = self.attn(self.norm1(x), past_kv)x = x + a # attention residualx = x + self.ffn(self.norm2(x)) # FFN residualreturn x, new_kv
The advantage of pre-normalization is that there is no normalization operator on the residual main path, so the path from input to output approximates an identity mapping, and gradients can flow back to shallow layers smoothly. This significantly eases the stability problems of large-model training, allowing deeper, wider networks to train under larger batch sizes.
3.3 Other Detail Adjustments
After pre-normalization became the consensus, several adjustments at the detail level still deserve attention:
- Removing the final output normalization: some models drop the normalization before the output layer, letting the final hidden state go directly into the linear projection, preserving the “purity” of the residual stream;
- Sub-layer initialization and scaling: appropriately scaling the output of the residual branch (for example, scaling weights by layer count) can further smooth the early training of ultra-deep networks;
- Q/K normalization: normalizing Q and K before the dot product to control its magnitude. Under ultra-long contexts and mixed-precision training, attention scores are prone to numerical explosion or collapse, and this kind of approach markedly improves numerical stability.
4. Attention: From Multi-Head to Grouped-Query, and Beyond
4.1 Why Attention Structures Are Repeatedly Reworked
During autoregressive generation, producing each new token requires the keys (Key) and values (Value) of all preceding tokens. To avoid recomputation, inference typically caches them in memory, known as the KV cache.
The size of the KV cache can be roughly estimated as:
KV cache bytes ≈ 2 × num_layers × num_kv_heads × head_dim × bytes_per_value × seq_len × batch_size
The factor of 2 comes from the two caches, K and V. Take a 60-layer model with 128 dimensions per KV head at FP16 precision as an example: each token requires roughly 30 KB per KV path. If each query head keeps its own independent KV, a 100K-token context needs nearly 6 GB of KV cache for a single request; but with a grouped scheme where every 8 query heads share one KV group, it drops to about 0.75 GB.
The numbers change with configuration, but the conclusion is clear: in long-context and high-concurrency scenarios, the KV cache is often the main consumer of inference memory, sometimes exceeding the model weights themselves. This is why the evolution of attention structures almost always revolves around “how to compress the KV with almost no loss of expressiveness.”
4.2 From MHA and MQA to GQA
The design space of attention structures can be characterized along two dimensions: the relationship between the number of query heads (Query Head) and key-value heads (KV Head).
- Multi-head attention (MHA): each query head has its own independent K and V. This has the strongest expressiveness but the largest KV cache;
- Multi-query attention (MQA): all query heads share the same K and V. The KV cache shrinks to 1/n of the original (n being the number of query heads), but expressiveness drops and training is prone to instability;
- Grouped-query attention (GQA): something in between. Every g query heads share one K and V group, so the KV cache shrinks to 1/g of the original, while retaining multiple key-value groups maintains sufficient expressiveness.
In implementation, GQA simply replicates K and V along the head dimension before doing standard attention computation:
def repeat_kv(x, n_rep):# x: [batch, num_kv_heads, seq_len, head_dim]if n_rep == 1:return xbatch, kvh, seq, d = x.shapex = x[:, :, None].expand(batch, kvh, n_rep, seq, d)return x.reshape(batch, kvh * n_rep, seq, d)
In practice, the number of groups g commonly falls between 2 and 8, depending on model scale and training strategy. Existing research generally shows that moderate grouping causes very little quality loss, while the savings in inference memory and bandwidth are substantial. As a result, GQA became the standard configuration in open-source models within a relatively short time.
4.3 More Aggressive Directions: Low-Rank Compression, Sliding Windows, and Hybrid Attention
Beyond GQA, three more aggressive lines of thinking have emerged in recent years.
The first is low-rank key-value compression. The basic idea is: rather than caching high-dimensional K and V directly for each token, project them first into a low-rank latent vector much smaller than the original dimension. During inference, only the latent vector is cached, and K and V are reconstructed through an up-projection matrix when computing attention. Since the reconstruction matrix is a fixed weight, the cached representation essentially carries “unfoldable key-value information.” In addition, such methods usually apply the positional rotation to only one of the two paths, decoupling it from the compression structure. This approach is more thorough than GQA, and the compression ratio can far exceed what the number of groups can achieve.
The second is a hybrid of sliding-window and global attention. Not every token needs to interact with the entire history. A common approach is:
- Most layers, or most attention heads, use a local window and attend only to the most recent several tokens;
- A few layers, or a few heads, retain global attention responsible for long-range information aggregation.
This design borrows from the idea of hierarchical receptive fields in vision, letting local details and global dependencies each be handled by an appropriate module, and can significantly reduce computation under long contexts.
The third is linear or sub-linear complexity alternatives. Examples include recurrent models centered on state updates and gating mechanisms, and some hybrid architectures that combine linear attention with a small amount of global attention. In theory, these methods can reduce the complexity of sequence modeling from quadratic to linear, but their performance on “precise retrieval” tasks remains debated. A more pragmatic current approach is to retain standard attention in a few global layers, balancing efficiency and retrieval capability.
4.4 A Frequently Overlooked Division of Labor
If you take a model apart, a clear fact emerges:
- Attention is responsible for “mixing”: moving and aggregating information between tokens, determining the context-modeling capability;
- The feed-forward network is responsible for “processing”: performing a nonlinear transformation on each token independently, determining the per-token expressiveness.
This explains why KV cache compression tends to concentrate on the attention side, while parameter-scale expansion tends to occur on the feed-forward side. Understanding this division of labor helps you judge, when selecting a model, whether a given architectural change affects “how far it can see” or “how deeply it can think.”
5. Feed-Forward Networks: From GELU to SwiGLU and Mixture-of-Experts
5.1 SwiGLU: Expressiveness from Gating
The classic feed-forward network is a two-layer linear transformation with an activation:
FFN(x) = W₂ · GELU(W₁ · x)
Modern models more often use the gated structure SwiGLU:
SwiGLU(x) = (Swish(x·Wgate) ⊙ (x·Wup)) · Wdown
Here ⊙ denotes element-wise multiplication, and Swish is a smooth self-gated activation function. This structure has three linear projections, one more set of weights than the classic FFN. To keep the parameter count comparable, the hidden dimension is usually reduced to about 8/3 times the model dimension, rather than the classic 4×.
Classic FFN params: d × 4d + 4d × d = 8d²SwiGLU params: 3 × d × (8/3)d = 8d²
The value of the gating mechanism is that it lets the network “selectively pass” information: one branch acts as a gating signal that controls the activation strength of another branch. Compared with a plain nonlinear activation, this multiplicative interaction is more expressive and, at equal parameter count, typically delivers a steady improvement.
5.2 Sparse Mixture-of-Experts: Trading Capacity for Efficiency
If SwiGLU works within a single feed-forward network, sparse mixture-of-experts (MoE) works at the level of how feed-forward networks are organized as a whole.
Its basic structure is to replace a single large FFN with multiple parallel “expert” FFNs, where a lightweight router selects a few experts to participate in the computation for each token. This produces two key metrics:
- Total parameters: determine the model’s “knowledge capacity”;
- Activated parameters: the number of parameters actually involved in computation per token, determining the compute cost.
Once these two metrics are decoupled, the model can significantly increase its total capacity by adding more experts while keeping inference compute roughly unchanged. This is one of the important reasons why open-source flagship models have scaled up rapidly over the past two years while the per-token inference cost has not risen proportionally.
class MoEFFN(torch.nn.Module):def __init__(self, d_model, d_ff, num_experts, top_k):super().__init__()self.gate = torch.nn.Linear(d_model, num_experts, bias=False)self.experts = torch.nn.ModuleList([SwiGLUFFN(d_model, d_ff) for _ in range(num_experts)])self.top_k = top_kdef forward(self, x):# x: [num_tokens, d_model]scores = self.gate(x).softmax(dim=-1) # routing probabilitiesweights, idx = scores.topk(self.top_k, dim=-1) # pick top_k experts per tokenout = torch.zeros_like(x)# compute grouped by expert to avoid per-token loopsfor eid in range(len(self.experts)):token_ids, slot_ids = torch.where(idx == eid)if token_ids.numel() == 0:continuey = self.experts[eid](x[token_ids])w = weights[token_ids, slot_ids].unsqueeze(-1)out.index_add_(0, token_ids, y * w)return out
5.3 Key Engineering Points in Routing Design
The success or failure of sparse MoE depends largely on routing design. Common practical considerations include:
- Fine-grained experts: making experts smaller and more numerous, with each token activating more experts. This makes specialization easier to achieve than “a few large experts” and makes routing decisions smoother;
- Shared experts: setting up a few shared experts that always participate in computation, handling general knowledge and common transformations, while the remaining routed experts focus on differentiated capabilities. This helps reduce knowledge redundancy across experts;
- Load balancing: if routing collapses onto a few experts, the remaining experts receive no training, which both wastes parameters and slows computation. The traditional approach adds a balancing term to the loss function, but this competes with the language-modeling objective and may harm the final result. Newer approaches no longer introduce an auxiliary loss; instead, they maintain a dynamically adjustable routing bias for each expert: lowering its bias when an expert is overloaded and raising it when underloaded. This balances the load without interfering with the main task’s gradients.
5.4 The Cost of Sparse Structures
Sparse structures do not come with benefits only. Their costs mainly appear in three areas:
- On the training side: they require additional mechanisms such as a routing network, capacity control, and expert parallelism, noticeably increasing engineering complexity;
- On the inference side: all weights must still reside in memory, while each token activates only a small fraction of them, shifting the bottleneck from compute to memory bandwidth;
- On the deployment side: when experts are spread across different devices, tokens must be routed between devices, introducing additional communication overhead.
Therefore, “a large total parameter count” does not equal “expensive inference,” but it does mean higher memory footprint and deployment complexity, requiring careful trade-offs among capacity, latency, and cost.
6. Prediction Targets: From Single Token to Multi-Token Prediction
A traditional language model predicts only the next token at each position. Multi-token prediction (MTP) adds several prediction modules alongside the main prediction head, predicting the 2nd, 3rd, and up to the k-th future tokens simultaneously.
Its value shows up on two levels:
- On the training side: each position provides a richer supervision signal, forcing the model to plan over longer-range dependencies ahead of time—especially beneficial for reasoning tasks;
- On the inference side: the additionally trained prediction modules can be reused as a source of “drafts” for the main model; combined with speculative decoding, one forward pass can verify multiple candidate tokens, reducing decoding latency.
It is worth emphasizing that MTP is not simply widening the output layer. To ensure the prediction modules share context information, hidden states are usually passed between the modules, forming an auxiliary feature-propagation chain. This chain can be optionally enabled during inference: the full chain is used when pursuing quality, while only the main prediction head is kept when pursuing speed.
7. Cross-Comparison: The Trade-Off Logic of Architectural Choices
Summarizing the changes above, we can distill a few clear selection logics:
| Design Goal | Common Architectural Approaches | Primary Benefit | Primary Cost |
|---|---|---|---|
| Extend context length | RoPE + length extrapolation + continued long-context training | Supports longer inputs | Attention cost grows quadratically with length |
| Reduce inference memory | GQA, low-rank key-value compression | KV cache shrinks significantly | Expressiveness may drop slightly |
| Reduce inference compute | Sliding-window hybrid attention, linear attention | More compute savings on long sequences | Precise retrieval capability needs validation |
| Increase model capacity | Sparse MoE, shared experts | More total parameters at nearly constant compute | Higher memory, communication, and deployment complexity |
| Improve training stability | Pre-normalization, RMSNorm, Q/K normalization | Deeper, wider networks train stably | More implementation details |
| Improve data efficiency | SwiGLU, multi-token prediction | Better results at equal compute | More complex training and inference pipelines |
A notable pattern is that almost all evolution makes “additions” at the structural level, but the goal is “subtraction” in overhead. This reflects the core tension of large-model engineering—the ongoing tug-of-war between capability and cost.
8. The Implementation View: How Developers Should Approach These Changes
For developers, the ultimate purpose of understanding architectural evolution is to make good selection and deployment decisions. The following suggestions are worth considering.
8.1 Focus on Effective Metrics, Not a Single Parameter Scale
When evaluating a model, it is advisable to look at three categories of metrics simultaneously:
- Activated parameters: determine per-token computation and latency;
- KV cache strategy: number of groups, compression method, and whether sliding windows are supported;
- Context and position-extrapolation scheme: the native context length, and whether stability is maintained after extrapolation.
Looking only at total parameter count makes it easy to misjudge inference cost; looking only at context length may overlook memory pressure under long text.
8.2 Estimating Memory and Throughput
Before deployment, you can estimate resource requirements in a simplified way:
Weight memory ≈ num_parameters (including experts) × bytes_per_parameterKV cache ≈ 2 × num_layers × num_kv_heads × head_dim × bytes_per_value × seq_len × concurrency
From this, you can quickly judge: given the available GPU resources, how long a context and how large a concurrent batch can be supported. For MoE models, you also need to additionally consider the bandwidth and communication latency of the devices hosting the experts.
8.3 A General Architecture Pattern for Serving
Drawing on common industry practice, a typical inference service can be abstracted into the following layers:
- Access layer: a unified API gateway responsible for authentication, rate limiting, and request routing;
- Scheduling layer: dynamic batching and request queuing, organizing requests of varying lengths into efficient batches;
- Cache layer: reducing redundant prefill computation through key-value cache reuse and tiered storage;
- Compute layer: containerized deployment of inference instances, elastically scaling based on load;
- Observability layer: continuously tracking time-to-first-token, throughput, memory watermark, and error rate through logging services and monitoring/alerting.
When deploying in the cloud, this architecture can be built on general-purpose cloud capabilities:
- Model files and training data can be stored in object storage;
- Inference services can be orchestrated and elastically scaled via a container platform;
- Decoupling between tasks can be achieved through message queues;
- Runtime status can be continuously observed through logging services and monitoring/alerting.
General-purpose capabilities offered by cloud providers such as Baidu AI Cloud can serve as infrastructure options for such a solution.
8.4 Three Common Misconceptions
- Misconception 1: Bigger total parameters are always better. For MoE models, the per-token cost is determined by activated parameters and memory footprint, not total parameters.
- Misconception 2: Longer context is always better. Ultra-long contexts bring rapid growth in attention computation and linear expansion of the KV cache, requiring trade-offs based on the task’s actual needs.
- Misconception 3: A newer architecture must be better. Architectural effectiveness depends heavily on training data and training strategy. The same structure can show marked differences in different teams’ hands, so evaluation should be based on measured results on your own business data.
Conclusion: Polishing the Foundation Is Also a Form of Evolution
Returning to the opening question: over seven years, have we seen breakthrough changes?
If “breakthrough” means overturning the existing paradigm and starting from scratch, then the answer leans negative. The skeleton of the decoder Transformer—residual connections, normalization, and the combination of attention and feed-forward networks—remains the basic construction of today’s large models.
But if “breakthrough” means keeping this construction running at larger scale, longer context, and lower cost, then the answer is undoubtedly positive. RoPE solved position generalization, RMSNorm and pre-normalization solved training stability, GQA and low-rank compression solved inference memory, SwiGLU and sparse MoE solved the decoupling of capacity and compute, and multi-token prediction improved data efficiency and decoding speed. None of these overturns and starts over, yet together they support the boundaries of today’s large-model capabilities.
For developers, the practical value of understanding these evolutions lies in the ability to:
- When selecting a model, read the architectural descriptions in model cards and judge their true cost structure;
- When deploying, plan memory and concurrency appropriately based on KV cache strategy and context length;
- When optimizing, pinpoint whether the bottleneck lies in attention, the feed-forward network, or the communication layer.
The continual polishing of the foundation is itself a form of evolution. And understanding how the foundation is polished is precisely the fundamental skill every large-model practitioner should have.
