LLM Architecture in 2025: A Complete Breakdown of the Components Behind Modern Large Language Models
This article offers a component-by-component breakdown of mainstream open-source large language model (LLM) architectures in 2025. The decoder-only skeleton is largely unchanged, but nearly every internal part—positional encoding, attention, the feed-forward network, and normalization—has been reworked for efficiency. The guide walks through rotary position embeddings (RoPE) and long-context frequency scaling, grouped-query attention (GQA) and KV-cache compression, SwiGLU and RMSNorm, mixture-of-experts (MoE) sparsity, sliding-window attention and hybrid backbones, tokenizers, and the parallel-training and inference engineering behind them. It also covers common misconceptions, a practical checklist for reading model technical reports, and methods for estimating self-hosted inference cost. Readers will gain the vocabulary and judgment to decide which architectural choices matter for their own deployment scenarios.
1. Seven Years Later: What Stayed and What Changed
From the first autoregressive decoder architectures, to the stable technical paradigm that emerged around 2019, to the new generation of open-source flagship models released in 2024–2025, roughly seven years have passed. Placing the configurations of early representative models side by side with those of popular open-source models from the past two years creates a sense that is “both familiar and unfamiliar.”
Familiar: the overall skeleton. The vast majority of mainstream open-source models still use a decoder-only structure. The backbone remains a stack of identical Transformer blocks that starts from an embedding layer and ends with an output layer producing the next-token prediction.
Unfamiliar: the internal parts. From how positional information is encoded, to how attention heads share keys and values, to the activation function, normalization scheme, and overall shape of the feed-forward layer, nearly every component has been replaced or rebuilt.
1.1 Key Components That Have Already Been Swapped Out
| Component | Typical early approach | Current mainstream approach |
|---|---|---|
| Positional information | Learnable or sinusoidal absolute positional encodings, fixed sequence length | Rotary position embeddings (RoPE), with frequency settings tuned for long context |
| Attention | Multi-head attention (MHA), each head with its own K and V | Grouped-query attention (GQA), multiple query heads sharing a set of K, V |
| Feed-forward network | Two fully connected layers with GELU activation | SwiGLU gated structure, with hidden width adjusted accordingly |
| Normalization | Layer normalization that fully standardizes features | Root-mean-square normalization (RMSNorm), dropping the mean calculation |
| Feed-forward form | Dense parameters, all active in every computation | Sparse mixture-of-experts (MoE), each token activating only some parameters |
| Context length | Usually 1K–2K tokens | 32K as a starting point; 128K and beyond is now fairly common |
In addition, several designs that previously appeared mainly in research work are steadily moving into the engineering mainstream: sliding-window attention, gating and routing structures in the feed-forward layer, backbones that mix attention with recurrent or convolutional modules, and caching and quantization optimizations aimed at inference. These designs all point toward the same goal: achieving stronger capabilities at a controllable cost.
1.2 “Same Skeleton, Rebuilt Parts”
A key observation is that most model changes happen at the “parts” level, not the “skeleton” level.
For a cheaper, more scalable new structure to replace a mature paradigm, three conditions must be met at the same time:
- It performs at least on par under an equal training budget;
- It has a mature enough toolchain for parallel training and inference;
- It runs fast and cheaply on hardware.
If any one condition is missing, the new structure stays in the paper. By contrast, swapping out pluggable components such as RoPE, GQA, and SwiGLU carries manageable risk and depends less on training frameworks and hardware. This lets them spread across the industry within just two or three years, whereas skeleton-level innovation needs a much longer validation cycle.
1.3 Underneath It All Is a Reallocation
Looked at together, these parts serve a single theme: allocating limited parameters, memory, and compute more effectively to where they are actually needed.
- RoPE turns positional information from “a fixed index looked up in a table” into “an extrapolatable relative relationship”;
- GQA concentrates KV-cache capacity onto a few shared key-value sets;
- SwiGLU and RMSNorm trade less cost for more stable expressive power;
- MoE decouples total capacity from per-computation cost;
- Sliding windows and hybrid backbones compress the overall attention compute.
For this reason, this article prefers to describe the 2025 LLM architecture as a “continuously refined efficiency revolution” rather than a “rebuild-from-scratch architectural revolution.” Still, it is worth noting the handful of structural explorations that are trying to challenge the standard paradigm of computing attention over the full sequence.
1.4 Why Architecture Comparison Is Hard, and Why It’s Still Worth Doing
Comparing the architectures of different large models to judge which designs truly work has always been difficult, because:
- Training data, data mixtures, training duration, and hyperparameters differ enormously and are often disclosed only partially;
- Even models with similar structures can differ markedly in performance.
This shows that architecture is only one dimension of a model’s capability. Even so, reading architectures is worthwhile, because it can at least answer three questions:
- Where do the developers think the biggest bottleneck currently lies—quality, memory, compute, or length?
- Which deployment scenarios does the design lean toward, such as high concurrency, low latency, very long documents, or multi-GPU clusters?
- Which improvements are general and transferable to models you build or fine-tune yourself?
Based on this judgment, this article does not discuss benchmark rankings or training-algorithm details. Instead, it focuses on structural changes, and limits its scope to text models; multimodal aspects are not covered here.
2. Analysis Framework: Decomposing a Model into Five Parts
To compare architectures systematically, we first need a unified way to decompose them. A typical decoder-only model can be broken from input to output into five parts:
text→ (1) tokenizer + embedding layer→ (2) positional encoding (combined with each attention head)→ (3) stack of N Transformer blocks├─ normalization (before each sub-layer)├─ attention sub-layer: MHA / MQA / GQA / sliding window / compressed cache├─ residual connection├─ normalization└─ feed-forward sub-layer: dense GELU → gated SwiGLU → sparse MoE→ (4) output normalization→ (5) output projection (optionally weight-tied with the embedding) → distribution over the vocabulary
Alongside this, cost must be accounted for from two angles.
Training side:
- Memory is determined jointly by parameters, gradients, and optimizer states;
- Compute grows mainly with parameter count and the number of training tokens;
- Parallel communication determines whether the model can scale to more devices.
Inference side:
- Weight memory is determined by parameter scale;
- KV cache grows linearly with context length and number of concurrent requests;
- The attention part’s compute grows quadratically with sequence length;
- The feed-forward part’s compute grows linearly with sequence length;
- The decoding stage generates one token at a time, bounded by memory bandwidth, so time-to-first-token and throughput are both key metrics.
The following sections analyze the main changes in each of the five parts.
3. Positional Encoding: From Absolute Indices to Relative Rotation
3.1 Why Positional Encoding Is Needed
Self-attention is inherently insensitive to token order: shuffling the input order does not change the set of each token’s representations (attention results are merely reordered along with the input). Positional information must therefore be injected explicitly. Positional encoding has gone through roughly three stages.
Stage 1: absolute positional encoding. Two representative approaches exist: using fixed sine and cosine functions, or learning an embedding vector for each position directly during training. Both let the model perceive position, but with clear limitations:
- Learned positional embeddings cannot naturally extrapolate to longer positions not seen during training;
- Fixed encodings can be computed for any position, but the model tends to memorize the position index itself rather than the relative distance between tokens.
Stage 2: relative position schemes. One family adds a distance-dependent linear-decay bias to the attention scores, so the model naturally favors nearby tokens; another explicitly models relative distance between keys and queries. These approaches better match the intuition that “relative position matters more than absolute position” in language, but they increase implementation complexity and can slow down some efficient inference kernels.
Stage 3: rotary position embeddings (RoPE). RoPE has now become the de facto standard in open-source large models.
3.2 The Core Idea of RoPE
RoPE adds no positional vector to the input. Instead, it applies a position-dependent rotation matrix to the query and key vectors. For a d-dimensional query vector q at position m, the dimensions are paired up, and each pair (q₂ᵢ, q₂ᵢ₊₁) is rotated as follows:
theta_i = base^(-2i/d)q'₂ᵢ = q₂ᵢ · cos(m·theta_i) - q₂ᵢ₊₁ · sin(m·theta_i)q'₂ᵢ₊₁ = q₂ᵢ · sin(m·theta_i) + q₂ᵢ₊₁ · cos(m·theta_i)
Key vectors are handled the same way. This yields two important properties.
First, attention scores depend only on relative position. After rotation, the inner product of query and key satisfies:
⟨f(q, m), f(k, n)⟩ = g(q, k, m - n)
That is, the scoring function depends only on the position difference (m − n). The relationship between positions 10 and 12 is mathematically identical in form to that between positions 1000 and 1002. This is precisely the source of RoPE’s potential for length extrapolation.
Second, different dimensions correspond to different wavelengths. As i increases, theta_i shrinks rapidly and the rotation gets slower:
- Lower-numbered dimensions are “high frequency,” changing noticeably between adjacent positions and suited to short-range relationships;
- Higher-numbered dimensions are “low frequency,” completing a full rotation only over longer distances and suited to long-range relationships.
Superimposing multiple frequencies forms an encoding structure reminiscent of multi-scale representations.
It is worth emphasizing that RoPE is defined only inside the attention mechanism; it affects the relative relationship between queries and keys and does not change the token embeddings themselves. As a result, RoPE can be combined flexibly with various positional schemes at different layers.
3.3 An Illustrative Implementation
The code below uses only basic numerical operations to illustrate the computation logic of RoPE; it is not a production-grade implementation:
from math import cos, sindef rope_pairs(x, pos, head_dim, base=10000.0):"""Apply RoPE to a head_dim-dimensional vector at a single position.x: query or key vector of length head_dimpos: absolute position of this token in the sequencebase: frequency base; often increased for long-context scenarios"""out = []for i in range(head_dim // 2):theta = base ** (-2.0 * i / head_dim) # rotation frequency for the i-th dimension pairangle = pos * thetaeven, odd = x[2 * i], x[2 * i + 1]out.append(even * cos(angle) - odd * sin(angle))out.append(even * sin(angle) + odd * cos(angle))return out
In real frameworks, precomputed cos/sin caches are used, and a fused kernel applies the rotation to all positions and all heads at once.
3.4 The Key to Long Context: Frequency Configuration
RoPE’s default base is typically 10000, a setting matched to early context lengths of around 1K–2K. Stretching the sequence to hundreds of thousands of tokens directly causes the high-frequency dimensions to rotate through many periods, producing an “aliasing” effect similar to that in signal processing, and positional distinguishability drops accordingly. Common improvement strategies fall into three categories:
- Position interpolation: compress actual positions by a factor before feeding them into RoPE, effectively “squeezing” the entire long text into the range the model is familiar with;
- Frequency scaling: increase the base, or apply non-uniform scaling to theta_i by frequency band, preserving the extrapolation ability of high-frequency dimensions while slowing the variation of low-frequency ones;
- Continued long-text training: continue pretraining or do long-context adaptation on the extended length, so the model genuinely learns to use distant information.
These strategies are often combined. In addition, some models now assign different frequency parameters to different parameter partitions or data categories, for finer control over long- and short-range information.
For practitioners, the single most important point is: the “context length” in an architecture report is not the same as the length a model can actually use effectively. How to test for “effective context” is discussed later.
4. Attention: From “Each to Its Own” to “Shared by Group”
4.1 Why Attention Needs to Slim Down
Attention is the most flexible and most expensive component in a Transformer. Autoregressive decoding splits inference into two phases:
- Prefill phase: process all input tokens in parallel at once, fully exploiting the parallelism of matrix operations;
- Decode phase: generate tokens one by one, where each step must read the key-value cache (KV cache) of all previous tokens.
In the decode phase, as the context grows, the memory bandwidth needed to load the KV cache often becomes the main bottleneck. As a typical example: a model with 32 layers, 32 attention heads per layer, a head dimension of 128, and FP16 precision requires 2 × 32 × 128 × 2 = 16384 bytes for K and V combined per token per layer; across 32 layers that is about 0.5 MB. When the context length reaches 128K with 32 concurrent requests, the KV cache alone requires about 2 TB of memory—far beyond any single GPU or single machine. Compressing the KV cache is therefore a core driver of attention design evolution.
4.2 MHA → MQA → GQA
In standard multi-head attention (MHA), each query head has its own keys and values. The extreme approach is multi-query attention (MQA): all query heads share a single set of keys and values. This shrinks the KV cache directly to 1/n of its original size (n being the number of heads), but the model’s expressive power drops noticeably and training can become less stable.
Grouped-query attention (GQA) is a compromise between the two: split the query heads into several groups, with each group sharing a set of keys and values.
- When each group has only one key-value set, GQA degenerates into MQA;
- When each query head has its own keys and values, GQA degenerates into MHA.
| Scheme | Number of key-value groups | KV cache relative to MHA |
|---|---|---|
| MHA | Same as the number of query heads | 1 |
| GQA | Number of query heads ÷ group size | 1 ÷ group size |
| MQA | 1 | 1 ÷ number of query heads |
An equivalent “broadcast” view helps explain GQA; the illustrative code below shows the core logic:
def expand_kv(kv, num_q_heads, num_kv_heads):"""Expand n_kv_heads key-value groups to num_q_heads heads by replication.kv has shape [batch, n_kv_heads, seq_len, head_dim]returns shape [batch, num_q_heads, seq_len, head_dim]"""assert num_q_heads % num_kv_heads == 0group_size = num_q_heads // num_kv_headsbatch, _, seq_len, head_dim = kv.shape# Replicate each key-value head group_size times; query heads are grouped accordinglykv = kv[:, :, None, :, :].expand(batch, -1, group_size, seq_len, head_dim)return kv.reshape(batch, num_q_heads, seq_len, head_dim)
This is a change that dramatically reduces inference memory with almost no loss in quality. The rapid adoption of GQA illustrates one main thread of LLM architecture evolution: “reserve expensive resources for where they are truly needed.”
4.3 Going Further: Key-Value Compression and Local Attention
Beyond GQA, there are two complementary routes.
The first is key-value compression. One approach first projects each position’s keys and values into a low-dimensional “latent vector,” caches this compressed representation, and reconstructs the keys and values when computing attention. Cache capacity then scales with the compression dimension rather than with “number of heads × head dimension.” Such methods achieve higher compression ratios but change the mathematical form of attention, so they require corresponding modifications to efficient attention kernels.
The second is sliding-window attention (SWA). Standard attention compute grows quadratically with sequence length—attention for 300K tokens costs roughly 100× that for 30K tokens. Sliding-window attention lets each token attend only to the most recent w positions, reducing complexity from O(L²) to O(L·w), where L is sequence length and w is window size. It is usually not applied to every layer; instead it is stacked alternately with a small number of “global layers,” balancing local detail against global information. This layer-alternation approach borrows from vision Transformers: local layers handle fine-grained patterns, while sparse global layers handle long-range integration.
The two routes can be stacked—for example, some layers use key-value compression and others use sliding windows, dividing the work by layer.
4.4 Rediscovered Phenomena
As long context becomes standard, several “old problems” in attention research have been rediscovered.
Attention sink. Studies have found that models assign a large share of attention to a few tokens at the very start of the sequence, even when those tokens are not semantically important. This can be understood as a “low-value attention drain” that absorbs excess attention weight. Building on this, one inference optimization is: always keep the KV cache of the first few tokens, and evict the rest in a sliding-window fashion, supporting near-infinite streaming generation under a fixed memory budget.
Lost-in-the-middle information. Over long contexts, models often use information located in the middle of the input markedly less efficiently. This shows a gap between “supporting long context” and “using long context effectively,” and explains why positional encoding, training-data mixtures, and attention sparsification receive so much attention in long-context scenarios.
5. Feed-Forward Networks and Normalization: Better Expression at Lower Cost
5.1 What the Feed-Forward Network Does
Within a Transformer block, attention handles information exchange between tokens, while the feed-forward network (FFN) applies a nonlinear transformation to each token independently. It is the largest part of the model by parameter share (usually about two-thirds of total parameters) and is generally regarded as the main carrier of a model’s “factual knowledge.”
A traditional FFN is a two-layer “expand → nonlinear activation → contract” structure:
FFN(x) = activation(x · W1) · W2
The hidden width is typically 4d (d being the model hidden dimension). Here the FFN contains 8d² parameters.
5.2 SwiGLU: Gating Makes Information Flow More Controllable
Gated linear unit variants (such as SwiGLU) add an extra “gate branch” after expansion, controlling how much information passes through via element-wise multiplication:
SwiGLU(x) = (Swish(x · W_gate) ⊙ (x · W_up)) · W_down
Here Swish(x) = x · σ(x), where σ is the Sigmoid function, and ⊙ denotes element-wise multiplication. An illustrative implementation:
def swish(x):return x / (1.0 + exp(-x)) # x * sigmoid(x)def swiglu_ffn(x, w_gate, w_up, w_down):gate = swish(x @ w_gate) # gate branchup = x @ w_up # information branchreturn (gate * up) @ w_down # contract after element-wise gating
Because SwiGLU contains three weight matrices, directly keeping a hidden width of 4d would inflate the parameter count to 12d². To hold the parameter budget constant, the mainstream approach shrinks the hidden width to about 8d/3 ≈ 2.67d, bringing the parameter count to about 8d², on par with a traditional FFN. At roughly equal parameter counts, SwiGLU typically delivers a stable quality improvement, which is the main reason it rapidly replaced GELU within just a few years.
5.3 RMSNorm: Lighter Normalization
Layer normalization must first compute the mean and variance of the features, then standardize them. Root-mean-square normalization (RMSNorm) only scales by the root mean square, dropping mean centering:
RMSNorm(x) = x / sqrt(mean(x²) + ε) ⊙ g
Its benefits include:
- One fewer cross-dimension reduction, so lower compute cost;
- One fewer set of mean-related statistics to maintain, so a simpler implementation;
- In most large-model settings, quality on par with layer normalization.
Today’s mainstream models commonly adopt a “pre-sub-layer normalization” (Pre-Norm) structure with residual connections. To improve training stability, some models also:
- Add normalization at the entry of both attention and FFN; some implementations deliberately omit the pre-FFN normalization to save compute;
- Apply normalization to queries and keys (QK-Norm) to prevent attention scores from exploding numerically, with clear benefits in large-scale training;
- Normalize the embedding output and the output layer to reduce early-training numerical fluctuations.
5.4 Details Make the Difference
Activation functions, normalization placement, and QK-Norm may each look like “small changes” in isolation. But large-model training consumes enormous compute, so any design that affects training stability and convergence speed can be amplified in the final quality. Many “micro-innovations” in architecture evolution are, in essence, trading very low cost for higher training success rates and better scaling efficiency.
6. Mixture of Experts: Decoupling Total Capacity from Per-Call Compute
6.1 Core Idea
In a dense model, every parameter participates in every forward pass. As the model grows, training and inference costs grow together. Sparse mixture-of-experts (MoE) breaks this coupling: it replaces the feed-forward sub-layer with several parallel “experts” (each essentially a smaller FFN), and a router (a gating network) decides which experts handle each token.
As a result, MoE models have two easily confused scale metrics:
- Total parameters: the sum of all experts and other parameters, which determines memory usage;
- Activated parameters: the number of parameters a single token actually passes through, which determines compute.
For example, an MoE model with tens of billions of total parameters but only about one-tenth of that activated has a per-call compute cost close to a much smaller dense model, while its knowledge capacity is closer to a large model. This “large capacity, low compute” characteristic makes MoE an important direction in LLM scaling over the past two years.
6.2 Key Design: Routing and Load Balancing
A typical top-k routing process looks like this:
def moe_forward(x, router_w, experts, top_k=2):"""Illustrative logic of an MoE feed-forward sub-layer."""scores = x @ router_w # score each token against each experttop_scores, top_idx = topk(scores, top_k) # select the k highest-scoring expertsweights = softmax(top_scores) # normalize only among the selected expertsout = 0for slot in range(top_k):idx = top_idx[:, slot] # index of the expert selected in this slotw = weights[:, slot] # that expert's weightexpert_out = dispatch_and_compute(x, idx, experts[idx]) # group by expert, then batch-computeout = out + w * combine(expert_out, idx) # merge back into original orderreturn out
Real systems use grouping and communication primitives so that all experts can compute in parallel. Several key issues arise in routing design:
- Load balancing: if a few experts are over-selected, the rest are under-trained and can be “starved.” Common measures include a load-balancing auxiliary loss and expert-capacity limits; some implementations add random perturbations to encourage exploration.
- Fine-grained and shared experts: slicing experts finer and having each token select more experts increases combinatorial flexibility; adding a small number of always-active shared experts carries the foundational capabilities every token needs, letting the routed experts focus on differentiated knowledge.
- Training stability: routers can fluctuate sharply on small batches, so they usually need router normalization, numerical-stability optimizations, and suitable initialization.
6.3 Engineering Costs
MoE is not free; it shifts pressure from “compute” to “memory” and “communication”:
- Memory pressure: all expert parameters must stay resident, so memory demand is set by the total parameter count.
- Communication overhead: experts are usually spread across different devices, so each token must be sent to the device hosting its target expert and the results sent back, substantially increasing collective-communication volume.
- Inference scheduling complexity: in the same batch, different tokens may go to different experts, making batching efficiency hard to guarantee and requiring expert parallelism and grouped scheduling.
MoE is therefore better suited to teams able to deploy multiple GPUs and nodes. When resources are constrained, a dense model or a small MoE model is often more pragmatic.
7. Long Context and Hybrid Backbones: The New Front Line of the Efficiency Race
7.1 What Context Length Means
Over the past two years, open-source model context windows have expanded from a few thousand tokens to tens of thousands or even hundreds of thousands. The main drivers are a few scenarios:
- Repository-level code understanding: fitting multiple files and their dependencies into a single request;
- Long-document processing: one-shot understanding and summarization of contracts, research reports, and technical manuals;
- Agent applications: tool-call histories, multi-turn conversations, and intermediate results accumulate continuously, placing high demands on window capacity.
But longer context is not always better; it brings at least three costs:
- Quadratic attention compute;
- Linearly growing KV-cache memory;
- Decay in the model’s ability to use information effectively.
7.2 Effective Context vs. Advertised Context
“Supports an X-token context” and “can accurately use information within an X-token range” are two different things. Common industry test methods include:
- Needle-in-a-haystack retrieval: plant several unrelated facts in a long text, then locate them when questioned;
- Variable tracking: change the value of the same variable multiple times in the text and check whether the model uses the final value;
- Multi-hop reasoning: the answer requires first finding several fragments scattered across the text and then combining them.
In practice, many models use middle information markedly less efficiently than information at the two ends—the “lost-in-the-middle” phenomenon mentioned earlier.
7.3 Engineering Techniques
To make long context truly usable, several engineering techniques have emerged:
- Continued long-text training: gradually increase sequence length during staged pretraining;
- KV-cache quantization and compression: store the key-value cache at lower precision or in compressed form;
- Paged cache management: borrowing from the idea of operating-system virtual memory, split the cache into fixed-size blocks, allocate by block, reduce fragmentation, and improve memory utilization;
- Chunked prefill: split long inputs into smaller chunks computed in batches, avoiding overly high single-pass compute peaks;
- Prefix reuse: reuse the cache of shared prefixes such as system prompts and documents, avoiding redundant computation.
Teams are advised not to look only at the advertised window size when choosing a model, but to test actual performance at the target length using a long-document set close to their own workloads.
8. Tokenizers and Other Components: The Easily Overlooked Link
8.1 Tokenization Strategy
Mainstream models mostly use byte-level BPE tokenization. Its advantages include:
- It can cover any UTF-8 text, with almost no characters that cannot be encoded;
- The tradeoff between vocabulary size and compression rate is fairly flexible.
Vocabulary size directly affects the model’s parameter structure and resource use. The embedding matrix and output projection together account for twice “vocabulary size × hidden dimension” in parameters (if embeddings and output are not weight-tied). For a model with a vocabulary of 128,000 and a hidden dimension of 5120, these two parts total about 1.3 billion parameters—not negligible. Some models therefore choose a larger vocabulary to improve compression for multilingual and code text, then save parameters by sharing input and output embeddings; others keep a smaller vocabulary and share weights, leaving the parameter budget to the backbone.
For Chinese and multilingual scenarios, tokenization efficiency is especially important. The fewer tokens a given semantic content occupies, the lower the inference cost and the longer the usable context.
8.2 Token Embeddings and the Output Layer
The embedding layer maps tokens to high-dimensional vectors. Some models additionally use techniques such as factorized embeddings or larger embedding dimensions to strengthen representational power.
The output layer has two approaches:
- Share weights with the embedding layer (weight tying), which saves parameters;
- Use an independent output projection, which usually gives a small quality improvement.
Output-layer normalization performs one more normalization before the output projection to stabilize the numerical distribution.
9. Training and Inference: The Parallel Engineering Behind Architecture
Architecture design is never isolated; it must match parallel strategies and hardware characteristics.
9.1 Training Side
- Data parallelism: each device holds a full copy of the model, processes a different batch of data, then synchronizes gradients. It is easy to implement, but memory usage does not decrease with device count.
- Tensor parallelism: split a single operator (such as matrix multiplication) across devices to compute collaboratively; communication is frequent, so it is typically used only on high-speed interconnects within a single machine.
- Pipeline parallelism: assign different layers to different devices and process batches like a pipeline, which requires sensible splitting to reduce “bubble” idle time.
- Sequence/context parallelism: split along the sequence dimension; a common technique for long-context training.
- Expert parallelism: specific to MoE models, placing different experts on different devices.
- Memory-optimization techniques: sharding of parameters and optimizer states, activation recomputation, and so on, further reducing memory usage.
Modern large-model training generally combines multiple forms of parallelism, and architecture design must consider in advance whether these splitting schemes are compatible.
9.2 Inference Side
Inference performance depends on three factors: arithmetic intensity, memory bandwidth, and cache capacity. In the decode phase, each token’s compute is small, so the system is mainly memory-bandwidth-bound; the prefill phase can fully exploit compute. Common inference optimizations include:
- Efficient attention kernels: reduce intermediate memory reads and writes;
- Continuous batching: dynamically concatenate different requests to improve GPU utilization;
- Weight and activation quantization: lower memory usage and improve throughput;
- Speculative decoding: use a small draft model or the model’s own shallow outputs to “guess” upcoming tokens, then have the main model verify them in a batch, reducing the number of decode steps.
Architecture design affects how well these optimizations work. For example:
- GQA makes the KV cache smaller, easing cache quantization;
- MoE introduces dynamic routing, making batch scheduling more complex;
- Sliding-window attention lowers the KV-cache capacity requirement but may limit some cache-reuse strategies.
When reading an architecture, it is worth asking: “Will this design make things easier or harder in the inference system?”
10. Common Misconceptions
Misconception 1: More parameters always means a stronger model. Scale remains an important factor, but architecture, data quality, and training methods matter just as much. An MoE model may have a very large total parameter count yet not much actual compute; conversely, some models with a smaller parameter scale can perform excellently when well trained.
Misconception 2: Newer architectures are always better. RoPE, GQA, and SwiGLU are popular precisely because they have been repeatedly validated in extensive practice. A novel structure is not necessarily more stable, and it still depends on whether the toolchain is mature.
Misconception 3: Longer context always means stronger ability. Context length is only a capacity metric; what truly matters is effective context. Content beyond a model’s effective range not only fails to help but may introduce interference.
Misconception 4: MoE is always cheaper. MoE saves compute but increases memory usage and communication cost. When hardware conditions are insufficient, a dense model may be more cost-effective.
Misconception 5: The tokenizer doesn’t matter. Tokenization efficiency directly affects token cost and context utilization. For the same content, a higher compression rate means lower cost.
Misconception 6: Only looking at the numbers in the technical report. An architecture description is only a starting point; real quality must be validated with your own business data.
11. How to Use the Architecture Lens: From Selection to Cost Estimation
11.1 An Architecture Selection Checklist
When reading a model technical report, you can go through the following checklist item by item:
| Dimension | What to look at | Why it matters |
|---|---|---|
| Overall structure | Dense or MoE, activated-parameter ratio | Determines the balance of compute and memory |
| Positional encoding | RoPE base, long-context adjustment strategy | Determines length-extrapolation ability |
| Attention design | Number of GQA groups, sliding window or compressed cache | Determines KV-cache size and long-sequence efficiency |
| Feed-forward structure | Activation function, number of experts and shared experts | Affects knowledge capacity and training stability |
| Tokenizer | Vocabulary size, whether weights are shared | Affects multilingual support and token cost |
| Context capability | Advertised length vs. effective length | Affects usability for long-document tasks |
11.2 Matching Tasks and Scenarios
Different tasks place different demands on architectural features:
- High-concurrency short dialogues: care more about time-to-first-token and throughput, where GQA, a small vocabulary, and quantized deployment are advantageous;
- Long-document analysis: care more about effective context and KV-cache management;
- Code generation: high demands on both long context and tokenization efficiency;
- Multilingual scenarios: attention to the tokenizer’s coverage of non-English text.
11.3 A Method for Estimating Inference Cost
Self-hosted inference cost can be roughly split into three parts: memory sets the hardware threshold, throughput sets the service capacity per GPU, and latency sets the user experience. Use the following formulas for a quick estimate:
memory ≈ weight memory + KV cache memory + runtime overheadKV cache memory ≈ 2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × bytes_per_element
Here the factor 2 accounts for keys and values respectively. For example, a 7-billion-parameter FP16 model has weights of about 14 GB; with a configuration of 32 layers, 8 key-value heads, and a head dimension of 128, at 8K context and 32 concurrent requests, the KV cache is about 4 GB. This estimate helps judge whether a single GPU can handle the target scenario and whether quantization or tensor parallelism is needed.
If you use model services provided by a cloud platform, billing is usually centered on the number of input and output tokens; consult the provider’s documentation for specific rules. In any case, it is best to run a “cost–quality” analysis with real business data during the selection phase, rather than looking at unit price alone.
11.4 Engineering Implementation Advice
- Define a business evaluation set first: extract typical samples from real workloads and build a small evaluation set covering metrics such as accuracy, format compliance, and latency;
- Compare on candidate models: test multiple candidate models on the same evaluation set rather than only looking at public leaderboards;
- Focus on inference-side optimization headroom: techniques such as quantization, batching, and prefix reuse often reduce cost more than switching models;
- Monitor continuously: after launch, track token consumption, latency distribution, and cache hit rates, and optimize promptly.
If you lack the conditions to build your own GPU cluster, you can also rely on general capabilities offered by cloud platforms to reduce the engineering burden, such as:
- Model hosting and inference services, with elastic compute scheduling;
- Object storage for model weights and training data;
- Container platforms for deploying and scaling inference services;
- Logging and monitoring with alerting, to track latency, throughput, and error rates.
For example, the Baidu AI Cloud Qianfan platform provides capabilities such as model invocation, development, and application building, and can serve as one option for enterprises building large-model applications.
12. Conclusion: The Paradigm Endures, Refinement Continues
Looking back from early decoder architectures to the 2025 open-source flagship models, what has truly changed is not the “skeleton” but the efficiency and division of labor of every part on that skeleton.
- RoPE takes positional information from “absolute indices” to “relative relationships”;
- GQA takes attention from “each to its own” to “shared by group”;
- SwiGLU and RMSNorm trade a lighter cost for better expression and training stability;
- MoE decouples model capacity from per-call compute;
- Sliding windows, key-value compression, and hybrid backbones are attempting to rewrite the default paradigm of attention.
Behind these changes is the same ongoing “efficiency revolution”: under limited parameters, memory, and compute, allocating resources to where they are needed most.
For engineering teams, the value of understanding these architectural details lies in:
- Being able to read technical reports and judge what scenario a model fits;
- Being able to estimate resource requirements and make sound deployment decisions;
- Being able to build your own evaluation system rather than blindly following leaderboards.
LLM architecture is still evolving rapidly. New directions such as multimodality, agents, and inference-time compute are driving the next wave of architectural innovation. But it is foreseeable that “continuously refining a mature skeleton” will remain the most pragmatic and reliable engineering path.
