LLM Architectures Compared: Normalization, Attention, and MoE Trade-offs in 2025

    This article compares the text architecture of today's flagship open-source LLMs against the decoder-only designs of 2019. The macro skeleton has barely moved, but nearly every module has been reworked: RMSNorm and QK-Norm for training stability, RoPE and its extrapolation methods for long context, GQA, sliding-window attention and low-rank KV compression for inference cost, and SwiGLU and mixture-of-experts (MoE) feed-forward networks for decoupling capacity from per-token compute. It walks through each module, the problem it solves, and the price it charges, then covers engineering practice: memory and compute estimation, parallelism strategies, and a checklist for choosing an architecture.

    1. The Skeleton Held Still, the Parts Kept Moving

    Look at the first generation of mainstream decoder-only models released in 2019, then compare them with the flagship open-source models that appeared in 2024–2025, and the first reaction of many readers is: six years on, the structure looks almost unchanged. Is this genuine incremental polishing, or is something else going on?

    My read is that the macro structure is indeed stable, but change at the module level is dense. The big frames survived: the decoder-only Transformer, pre-normalization with residual connections, autoregressive next-token prediction. Open the box, though, and almost every component has been revisited — how normalization is done, what form positional encoding takes, whether attention heads share key/value tensors, whether the feed-forward network uses sparse experts.

    Individually, none of these changes is “disruptive.” Taken together, they determine the three things that matter most in engineering: whether the model can be trained stably, whether inference cost is controllable, and whether long-context workloads can actually run. So even though a modern architecture diagram doesn’t look far from a textbook diagram of a few years ago, comparing these modules one by one is still worthwhile. It shows two things: where leading teams are currently spending compute, and what they are willing to give up for it.

    This is not a performance leaderboard, and it does not go into training-algorithm detail. It focuses on the text architecture of current flagship open-source language models. Rather than listing exact parameter configurations model by model, it emphasizes the reusable design logic behind each choice. Multimodal capability deserves its own discussion and is left for later.

    1.1 Setting a Reference Baseline

    For easier comparison later, here is a schematic modern decoder block. Variable naming follows current mainstream implementations for readability:

    1. class TransformerBlock(nn.Module):
    2. def __init__(self, cfg):
    3. super().__init__()
    4. # Pre-normalization: RMSNorm is the common choice in mainstream implementations
    5. self.norm1 = RMSNorm(cfg.d_model)
    6. self.norm2 = RMSNorm(cfg.d_model)
    7. # Attention: grouped-query attention with rotary position embeddings
    8. self.attn = CausalSelfAttention(
    9. d_model=cfg.d_model,
    10. num_heads=cfg.num_heads,
    11. num_kv_heads=cfg.num_kv_heads, # often num_heads / G, where G is the group count
    12. rope_base=cfg.rope_base,
    13. qk_norm=cfg.qk_norm,
    14. )
    15. # Feed-forward network: SwiGLU (dense or MoE variant)
    16. self.ffn = build_ffn(cfg) # returns SwiGLU or MoEFeedForward
    17. self.res_scale = 1.0 / math.sqrt(2 * cfg.n_layers) # residual scaling factor
    18. def forward(self, x, pos_start=0, kv_cache=None):
    19. h = self.norm1(x)
    20. x = x + self.res_scale * self.attn(h, pos_start, kv_cache)
    21. x = x + self.res_scale * self.ffn(self.norm2(x))
    22. return x

    The code is deliberately compressed into a single block because most of the design variation in modern decoder models has a landing spot somewhere inside it:

    Module Key parameters / components Problem it addresses
    Normalization RMSNorm, QK-Norm Training stability, operator efficiency
    Positional encoding RoPE and its extrapolation methods Relative position modeling, long context
    Attention GQA group count, sliding window, low-rank compression Inference memory and long-sequence compute
    Feed-forward network Activation function, expert count, active experts Decoupling model capacity from compute
    Residual connection Scaling factor, whether biases are kept Gradient stability in deep networks

    1.2 The Module-Level Picture

    Placing a modern decoder architecture next to the typical configuration of mainstream decoder models from around 2019 concentrates the differences in the following places:

    Dimension Typical practice circa 2019 Common practice in current flagship open-source models
    Normalization LayerNorm, with scale and shift parameters RMSNorm, scale only
    Positional information Learned absolute position embeddings RoPE, plus extrapolation extensions
    Bias terms Biases commonly kept in projection layers Most implementations remove projection biases
    Key/value heads Every attention head owns its own keys and values Multiple query heads share one set of keys/values (GQA)
    FFN activation GELU Gated activations such as SwiGLU
    FFN structure Dense MLP Dense SwiGLU or sparse MoE
    Context length Typically a few thousand tokens Commonly hundreds of thousands of tokens

    None of these changes stands alone. The rest of this article takes them apart one by one, in the order module → mechanism → cost.

    2. Normalization: Fewer Parameters for More Stability

    2.1 LayerNorm vs. RMSNorm

    LayerNorm standardizes the feature vector of each token: subtract the mean, divide by the standard deviation, then multiply by a learned scale vector and add a learned shift vector.

    RMSNorm simplifies this. It computes only the root mean square as a scaling factor, does no centering, and has no shift. The formulas:

    1. LayerNorm(x) = γ ⊙ (x − μ) / √(σ² + ε) + β
    2. RMSNorm(x) = ω ⊙ x / √(mean(x²) + ε)

    A schematic implementation:

    1. class RMSNorm(nn.Module):
    2. def __init__(self, d_model, eps=1e-6):
    3. super().__init__()
    4. self.weight = nn.Parameter(torch.ones(d_model))
    5. self.eps = eps
    6. def forward(self, x):
    7. # Compute the root mean square along the last dimension,
    8. # then scale per channel with a learnable weight
    9. rms = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)
    10. return x * rms * self.weight

    There are three benefits:

    1. Fewer parameters: each normalization layer drops a shift vector the length of the hidden dimension. A Transformer block typically has two normalization layers, so a model of depth L saves roughly 2L × d_model parameters.
    2. Less compute: no mean reduction, one subtraction and one addition fewer. Normalization operators are classic memory-bound operations, so reduced memory traffic translates directly into lower latency.
    3. Simpler implementation: kernels are easier to fuse and to optimize for specific hardware.

    2.2 Pre-Normalization Became the Default

    Early work observed that post-normalization — placing normalization after the residual addition — requires smaller learning rates and longer warmup in deep networks, or it tends to diverge. Pre-normalization keeps the scale of the input entering each sublayer stable and smooths the gradient path, which is why it became the default for large models.

    The cost is that the magnitude of the residual stream grows with depth during the forward pass. Poorly parameterized, activations can keep increasing and eventually trigger numerical overflow. Two common mitigations:

    • Residual scaling: multiply each sublayer output by a depth-dependent factor, such as 1/√(2L).
    • Small-variance initialization: use a smaller initialization variance for the output projection layers.

    2.3 QK-Norm: Putting a Speed Limit on Attention Scores

    Self-attention computes the dot product between query and key vectors, divided by √d_head. As training progresses, the logits of some attention heads keep growing in magnitude, which easily overflows under low-precision training such as FP16 — a common trigger of loss spikes.

    QK-Norm normalizes queries and keys along the feature dimension before the dot product (usually with RMSNorm or L2 normalization). This is equivalent to fixing the length of queries and keys first, so that attention weights are decided by direction alone. The dynamic range of the attention scores is thereby bounded. The concrete form:

    1. def scaled_dot_product(self, q, k, v, mask):
    2. # Optional: normalize q and k along the feature dimension
    3. # to keep logits from blowing up numerically
    4. if self.qk_norm is not None:
    5. q = self.qk_norm(q)
    6. k = self.qk_norm(k)
    7. scores = (q @ k.transpose(-2, -1)) / math.sqrt(self.head_dim)
    8. scores = scores.masked_fill(~mask, float("-inf"))
    9. return scores.softmax(dim=-1) @ v

    3. Positional Encoding: From “Absolute Coordinates” to “Rotation Angles”

    3.1 The Limits of Absolute Position Embeddings

    Self-attention is inherently insensitive to order and has to rely on positional information. Early decoder models commonly used learned absolute position embeddings: one vector per position, added to the token embedding. Simple to implement, but with two clear weaknesses:

    • The learned position table has a fixed length, so positions beyond the training length have no corresponding vector.
    • The model learns absolute coordinates rather than relative distances, so extrapolation to longer sequences degrades noticeably.

    3.2 How RoPE Works

    RoPE (rotary position embedding) encodes positional information into the query and key vectors by rotation. Feature dimensions are partitioned into two-dimensional groups; for a token at position m, group i is rotated as follows:

    1. [q_2i, q_2i+1] at position m →
    2. [q_2i·cos(mθ_i) − q_2i+1·sin(mθ_i),
    3. q_2i·sin(mθ_i) + q_2i+1·cos(mθ_i)]
    4. where θ_i = base^(−2i/d), and base is usually 10000 or larger.

    Because the rotation matrix is orthogonal, the inner product of a query and a key depends only on their relative displacement m − n, not on absolute position. This is more natural than absolute position embeddings: whether the sentence starts here or there, the information “three tokens apart” stays the same.

    A schematic implementation (production code usually uses interleaved pairs; this example splits the first and second halves for clarity):

    1. def build_rope(seq_len, head_dim, base=10000.0, device="cpu"):
    2. # Rotation frequency per dimension pair: base^(-2i/head_dim)
    3. inv_freq = 1.0 / (base ** (torch.arange(0, head_dim, 2, device=device).float() / head_dim))
    4. angles = torch.outer(torch.arange(seq_len, device=device).float(), inv_freq) # [seq_len, head_dim/2]
    5. return angles.cos(), angles.sin()
    6. def apply_rope(x, cos, sin):
    7. # x: [..., seq_len, head_dim]; applied to queries and keys only
    8. x1, x2 = x.chunk(2, dim=-1)
    9. cos, sin = cos[None, None, :, :], sin[None, None, :, :]
    10. return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)

    3.3 Long-Context Extrapolation

    RoPE has no extrapolation ability of its own: once positions exceed the training range, the rotation angles enter intervals the model has never seen and quality falls off. Three families of solutions are common in the field:

    1. Position interpolation: map out-of-range positions back into the training interval. This is equivalent to having the model “see” longer text as compressed short text, at the cost of distance resolution.
    2. Frequency scaling: approaches such as NTK-aware scaling and YaRN interpolate only some frequencies — high-frequency dimensions barely move while low-frequency dimensions are stretched substantially, balancing local resolution against global coverage.
    3. Progressive training: increase the context length in stages (for example, a few thousand tokens first, then tens of thousands), adding enough long samples in the final stage so the model gradually adapts to long-range dependencies.

    There is also research exploring the possibility of “no explicit positional encoding at all.” As long as a causal mask is present throughout training, the model may still learn order information implicitly, but conclusions of this kind still await validation at larger scale.

    3.4 Nominal Length Is Not Effective Length

    One point deserves emphasis: a model’s advertised context length is an upper bound on what it can process, not a range it can reliably remember. If a large share of training samples are actually short, the model’s ability to exploit long-range information drops noticeably. Judging long-context capability should combine targeted evaluation tasks with the numbers, rather than reading the advertised figure alone.

    4. Attention: GQA, Sliding Windows, and Low-Rank Compression

    4.1 The KV Cache: Inference Cost’s Hidden Heavyweight

    During decoding, generating each new token requires the keys and values of all previous tokens. Recomputing them at every step is quadratic in cost. In practice the historical keys and values are cached (the KV cache), trading memory for speed. Its footprint can be estimated roughly as:

    1. KV memory ≈ 2 (keys and values) × layers × KV heads × head dim × sequence length × batch × bytes per value

    Taking 32 layers, head dimension 128, sequence length 128K tokens, batch size 1, and BF16 precision as illustrative assumptions:

    Attention scheme KV heads (illustrative) KV cache (illustrative) Relative multiple
    Multi-query attention (MQA) 1 ~1.1 GB 1×
    Grouped-query attention (GQA) 8 ~8.6 GB 8×
    Multi-head attention (MHA) 32 ~34.4 GB 32×

    Under the same assumptions, the KV cache alone can take up a substantial share of the memory on mainstream accelerators. GQA is the most direct compromise among these.

    4.2 MHA, MQA, and GQA

    • MHA (multi-head attention): every query head has its own key/value heads. Strongest expressive power, largest KV cache.
    • MQA (multi-query attention): all query heads share one set of keys and values. The KV cache shrinks to 1/H (H = number of heads), but the quality loss is usually noticeable.
    • GQA (grouped-query attention): query heads are split into G groups, each sharing one set of keys and values. G = H degenerates to MHA, G = 1 to MQA. Real configurations often use around 8 groups, balancing memory against quality.

    A schematic implementation:

    1. def repeat_kv(x_kv, n_rep):
    2. # n_rep = number of query heads // number of KV heads
    3. # the expansion is only a compute view and allocates no extra memory
    4. if n_rep == 1:
    5. return x_kv
    6. bs, n_kv_heads, seq, d = x_kv.shape
    7. return (
    8. x_kv[:, :, None]
    9. .expand(bs, n_kv_heads, n_rep, seq, d)
    10. .reshape(bs, n_kv_heads * n_rep, seq, d)
    11. )

    GQA’s appeal is that it is a purely inference-side optimization: during training each query head still behaves independently and only shares one set of keys and values, so the quality loss is limited while the KV cache drops by a multiple. It has therefore become something close to a new default.

    4.3 Sliding Windows: Pulling Quadratic Complexity Back to Linear

    Even with GQA, attention compute still grows quadratically with sequence length. Sliding-window attention restricts each token to attending to a fixed number of preceding tokens (a few thousand, say), which changes two things:

    • Compute complexity drops from O(S²) to O(S×W), where S is the sequence length and W is the window size.
    • For long texts, the KV cache only needs to retain what falls inside the window.

    Purely local attention, however, cuts off long-range dependencies. The common approach is a local + global mix: most layers use sliding windows while a few layers keep full attention, or the two modes are interleaved across layers.

    1. def sliding_window_mask(seq_len, window, device="cpu"):
    2. idx = torch.arange(seq_len, device=device)
    3. delta = idx[:, None] - idx[None, :] # backward distance each token can see
    4. return (delta >= 0) & (delta < window) # causal constraint + window constraint

    This hybrid structure shows up especially often in long-document summarization, repository-level code understanding, and multi-turn conversation.

    4.4 Low-Rank Compression: Squeezing Keys and Values into a Latent Space

    Another line of thinking comes from multi-head latent attention (MLA). Instead of caching complete keys and values, it first applies a down-projection that compresses each token’s key/value representation into a low-dimensional latent vector, which is what gets cached; an up-projection approximately reconstructs it when attention is computed. KV cache size is then determined by the latent dimension rather than by heads × head dim.

    The gains are significant:

    • At comparable quality, the KV cache can be compressed to a fraction of the GQA footprint.
    • At inference time it can be “absorbed” into the query and output projections, reducing matrix work during decoding.

    The cost is greater structural complexity: specialized, well-optimized attention operators are needed to realize the full potential.

    4.5 Hybrid and Alternative Routes

    Long context has also spawned other ideas:

    • Mixing full attention and sliding windows within the same bit budget;
    • Combining local modeling operators such as convolutions;
    • Interleaving with state-space models or linear attention.

    These directions are still evolving quickly. The thread worth following is that the core tension in attention remains the trade-off between expressiveness and inference memory/compute — every new structure redraws that boundary.

    5. Feed-Forward Networks: From GELU to SwiGLU, Then to MoE

    5.1 Gated Activations: Filtering Information with Multiplication

    The classic Transformer feed-forward network is a two-layer structure: expand, activate, project down, with ReLU or GELU as the activation. SwiGLU adds a gating branch: the intermediate dimension is split in two, one half is passed through SiLU and multiplied element-wise with the other, and the result is projected down.

    1. class SwiGLU(nn.Module):
    2. def __init__(self, d_model, hidden_dim):
    3. super().__init__()
    4. self.w_gate = nn.Linear(d_model, hidden_dim, bias=False)
    5. self.w_up = nn.Linear(d_model, hidden_dim, bias=False)
    6. self.w_down = nn.Linear(hidden_dim, d_model, bias=False)
    7. def forward(self, x):
    8. # The SiLU gating branch filters the information branch element-wise
    9. return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))

    Compared with a GELU-based structure of the same parameter count, gated activations usually give better language-modeling results. One detail: at equal parameter count, SwiGLU has three weight matrices, so the intermediate dimension is commonly taken as 2/3 × 4 × d_model, rounded up to a convenient number.

    5.2 MoE: Separating “Total Capacity” from “Per-Step Compute”

    In a dense feed-forward network, parameter count and compute always scale together: every token passes through the entire network. MoE (mixture of experts) replaces one large feed-forward layer with N small expert networks and a gating router, and each token activates only its top_k experts:

    1. class MoEFeedForward(nn.Module):
    2. def __init__(self, d_model, n_experts, top_k, hidden_dim):
    3. super().__init__()
    4. self.router = nn.Linear(d_model, n_experts, bias=False)
    5. self.experts = nn.ModuleList([SwiGLU(d_model, hidden_dim) for _ in range(n_experts)])
    6. self.top_k = top_k
    7. def forward(self, x):
    8. B, S, D = x.shape
    9. flat = x.view(-1, D) # [T, D]
    10. logits = self.router(flat) # [T, N]
    11. weights, idx = logits.softmax(-1).topk(self.top_k, dim=-1)
    12. weights = weights / weights.sum(-1, keepdim=True) # some implementations normalize top-k weights
    13. out = torch.zeros_like(flat)
    14. for e, expert in enumerate(self.experts):
    15. # (token, slot) indexing makes capacity limits and token dropping easy to add later
    16. mask = (idx == e)
    17. rows, slots = mask.nonzero(as_tuple=True)
    18. if rows.numel() == 0:
    19. continue
    20. y = expert(flat[rows])
    21. out.index_add_(0, rows, y * weights[rows, slots].unsqueeze(-1))
    22. return out.view(B, S, D)

    The consequence: total parameter count sets the model’s capacity ceiling, while activated parameter count sets the compute per token. Common configurations reach several hundred billion total parameters while activating only a small fraction per token. Most of the total parameters sit in memory as a “knowledge reserve” that a single step neither reads in full nor participates in.

    5.3 Fine-Grained Experts and Shared Experts

    Routed MoE is only the basic form. Two refinements are common in real designs:

    • Fine-grained experts: cut experts smaller and make them more numerous, so each token can compose a more diverse “expert recipe” and specialization improves. The cost is finer routing, hence more complex communication and scheduling.
    • Shared experts: a small number of experts activated for all tokens carry cross-domain general knowledge, leaving routed experts to focus on differentiated capability. This helps reduce the parameter waste that comes from different experts repeatedly learning the same general patterns.

    5.4 Load Balancing: MoE’s First Hurdle

    Gated routing is naturally winner-take-all: a few experts get selected over and over and train well, the rest barely receive updates, and the model degenerates into a dense model or worse. Common mitigations:

    • Auxiliary balancing loss: constrain the distribution of expert utilization with
    1. L_aux = α × N × Σ_i (f_i × P_i)
    2. where f_i is the share of tokens assigned to expert i, and P_i is the router's average probability for expert i.

    f_i is not differentiable, so gradients flow back through the differentiable P_i, pushing the router toward a more even distribution.

    • Capacity factor and token dropping: set an upper bound on capacity per expert; overflow is dropped or routed straight through the residual connection. A larger capacity factor means less dropping, but higher memory and communication overhead.
    • Router numerical stability: some implementations add a router z-loss — the log of the mean of the squared logits — to keep routing logits from running away and to improve large-scale training stability.

    5.5 The Engineering Bill: Communication and Memory

    MoE is not a free lunch. Two main costs:

    • Communication overhead: training commonly uses expert parallelism, with different experts on different devices. Every token has to be dispatched to the device holding its target expert via All-to-All collective communication, then gathered back after computation. This is very sensitive to network bandwidth and latency, and becomes a bottleneck easily without high-speed interconnect.
    • Inference memory: even though a single step activates only a few experts, the weights of all experts generally have to stay resident in memory. Take an illustrative model with 600B total and 40B activated parameters: in BF16 the weights alone need roughly 1.2 TB, which no single accelerator can hold, so deployment has to combine tensor parallelism, expert parallelism, and other forms of sharding.

    5.6 Choosing Between Dense and Sparse

    Dimension Dense model MoE model
    Compute per token Proportional to total parameters Depends only on activated parameters
    Memory footprint Roughly equal to weight size Total parameters stay resident; usually larger
    Fine-tuning difficulty Relatively straightforward Requires attention to routing balance and expert updates
    Deployment complexity Lower Requires expert parallelism and high-speed interconnect
    Fits Cost-sensitive, latency-sensitive, resource-constrained Quality-first, with ample hardware resources

    6. Other Details Worth Watching

    6.1 Bias Terms: Less Is More

    Modern decoder models generally remove bias terms from attention projections and feed-forward projections, for two reasons:

    • Normalization layers already provide scaling and shifting capability, so the marginal value of an explicit bias is small.
    • Fewer parameters and simpler kernels allow more thorough operator fusion and lower inference latency.

    6.2 Embedding Sharing and Vocabulary Design

    Small models often tie the input token embedding to the output layer, projecting with the transpose of the embedding matrix directly. Both approaches exist in large models, and the trade-off needs to be measured against how much of the parameter budget each takes.

    Vocabulary size is likewise an architectural decision:

    • A larger vocabulary yields fewer tokens for the same text, lowering sequence overhead in training and inference.
    • But embedding and output layer parameters grow accordingly, and representations for low-frequency tokens may be less well learned.

    Vocabulary size therefore has to be settled together with the language distribution of the training corpus.

    6.3 Multi-Token Prediction and Speculative Decoding

    Standard training predicts only the next token. Some models add extra prediction heads so the model predicts the 2nd and 3rd future tokens at the same position. This serves two purposes:

    • During training it acts as an auxiliary task, forcing the model to learn longer-range dependencies and improving representation quality.
    • At inference those heads can serve as a “draft,” working with speculative decoding to reduce the number of autoregressive steps.

    7. Engineering Practice for Architecture Selection

    7.1 Estimating Memory and Compute

    Before actual deployment, resources can be estimated roughly with rule-of-thumb formulas:

    1. Weight memory ≈ total parameters × bytes per parameter (BF16 = 2, FP8 = 1)
    2. KV memory ≈ 2 × layers × KV heads × head dim × sequence length × batch × bytes
    3. Inference FLOPs ≈ 2 × activated parameters × generated tokens
    4. Training FLOPs ≈ 6 × parameters × training tokens (for MoE, driven mainly by activated parameters)

    The decoding phase of inference is memory-bandwidth-bound: each generated token requires reading the weights involved in the activated parameters out of memory once. So the metric that decides decoding speed is not total compute but the activated parameter count and memory bandwidth.

    7.2 Parallelism Strategies and Hardware Fit

    Parallelism What is sharded Communication characteristics Typical deployment
    Tensor parallelism Individual layer weight matrices Synchronization per layer, frequent communication Within a single node over high-speed interconnect
    Pipeline parallelism Layers Pipeline bubbles Across nodes
    Expert parallelism MoE experts All-to-All communication Combined with high-speed networks
    Context parallelism Sequence dimension Ring attention exchanging keys/values Very-long-context training

    Architecture choices directly affect which parallelism strategies are viable:

    • GQA, sliding windows, and low-rank compression shrink the KV cache substantially, making long-context inference easier to fit into limited memory.
    • MoE makes expert parallelism mandatory and raises the demand on high-speed interconnect.
    • Gated activations and normalization operators are individually “small” but called frequently, so fusion quality has a visible effect on end-to-end throughput.

    7.3 Notes on Running on Cloud Infrastructure

    When deploying a model on general-purpose cloud platforms, standard cloud capabilities can reduce engineering cost:

    • Elastic inference serving: run inference services on container services and scale in and out automatically with request volume.
    • Model and data hosting: keep model weights and fine-tuning datasets in object storage, with an image registry for version management.
    • Training job scheduling: run pretraining and fine-tuning on elastic compute resource pools, allocating capacity per job.
    • Observability: use logging services and monitoring/alerting to track time-to-first-token, per-token generation latency, KV cache hit rate, and similar metrics continuously.
    • Quality and safety: connect to model services to integrate LLM capability, and pair them with content-safety features to meet compliance requirements.

    It is worth noting that architecture selection is independent of the cloud provider, but post-deployment cost and stability depend heavily on how well these engineering factors work together.

    7.4 A Selection Checklist

    When facing a new model, work through these questions in order:

    1. Is the inference scenario memory-sensitive? If so, prefer options that lower the KV cache, such as GQA, sliding windows, or low-rank compression.
    2. Is the main workload long context? Confirm the model’s effective context length, not its advertised length.
    3. Is the compute budget limited? MoE has a small activated parameter count and decodes fast, but the memory and communication cost of the total parameter count has to be accounted for.
    4. Will you fine-tune often? Dense models have a more mature fine-tuning path, whereas MoE requires extra attention to routing balance.
    5. Is on-premises deployment needed? Assess weight memory, quantization support, the feasibility of parallelism strategies, and whether the existing network can meet expert-parallelism requirements.

    8. Common Misconceptions and Summary

    8.1 Four Common Misconceptions

    1. “The structure hasn’t changed, so there’s no progress.” A stable skeleton is precisely a sign that engineering focus has shifted to system-level optimization, while the room for evolution inside each module remains large.
    2. “A longer advertised context is always better.” Advertised length is not effective length; it has to be validated against the task.
    3. “MoE uses less compute, so it’s cheaper.” MoE saves per-step compute, not memory or communication. Total cost of ownership has to be assessed as a whole.
    4. “Copying a configuration is enough to reproduce results.” Normalization choices, initialization strategy, residual scaling, routing regularization, and similar details all affect large-scale training stability.

    8.2 Summary

    From 2019 to 2025, the decoder architecture’s skeleton stayed stable while its interior went through a systematic round of “trimming” and “expansion”:

    • Trimming: RMSNorm, removing biases, GQA, sliding windows, and low-rank compression each cut parameters, memory traffic, and inference overhead.
    • Expansion: RoPE extrapolation and progressive training push out the context boundary, while MoE enlarges model capacity without increasing per-step compute.

    For developers, the value of understanding these trade-offs is that when a new model technical report lands, you can quickly tell what it is optimizing, what it is sacrificing, and whether it matches your scenario. What really decides whether a model can be deployed is never the parameter count alone, but how well the architecture choices match the hardware conditions and the business workload.