Anatomy of Modern LLM Architectures: Structure, Parameters, and Hardware Constraints
A walkthrough of how modern large language model designs respond to three hardware constraints: compute, memory bandwidth, and interconnect. This article explains why total parameters and active parameters have diverged under Mixture-of-Experts, how KV Cache grows with long contexts and how it can be compressed, and the main routes to sparse attention, with worked calculations and code examples.
1. Understanding Model Structure Through Hardware Constraints
The architectural changes across recent models can look scattered: some adjust layer ordering, some rewrite attention computation, and some redesign expert routing. From the perspective of hardware accelerators, however, these changes all respond to three classes of constraints:
- Compute: how many multiply-accumulate operations each token requires.
- Memory bandwidth: how many bytes of weights and KV Cache must be read from GPU memory.
- Communication: the cost of exchanging data across multiple devices.
Only by identifying which resource a given change primarily saves can you judge its real impact on hardware.
In this article, B denotes billions of parameters and T denotes trillions of parameters. All quantities are counted in tokens, without plural suffixes.
2. Two Diverging Scaling Curves
Over the past two years, total model parameters have grown from several hundred B to several T, while the active parameters actually participating in computation per token have remained at a few dozen B. The main reason these two curves have diverged is the Mixture-of-Experts (MoE) structure: most parameters sit in an expert pool, and each token is routed to only a handful of experts.
The active ratio can be computed directly:
def active_ratio(total_b: float, active_b: float) -> float:"""Return the ratio of active parameters to total parameters."""return active_b / total_bprint(f"{active_ratio(671, 37):.1%}") # ~5.5%print(f"{active_ratio(1600, 49):.1%}") # ~3.1%print(f"{active_ratio(284, 13):.1%}") # ~4.6%
Total parameters determine the storage footprint of the weights, while active parameters determine the compute per token—and also the amount of weights that must be read at each decoding step. Once these two are decoupled, capacity planning and compute planning must be estimated separately. Taking FP8 storage as an example, each parameter occupies roughly 1 byte, so 1.6T parameters require about 1.6 TB of GPU memory for weights alone—excluding the KV Cache, activations, and communication buffers.
3. Longer Contexts and the KV Cache
As context lengths expanded from 128K to 1M, the direct consequence is that the KV Cache grows linearly with sequence length. In standard multi-head attention, the cache size per token is:
KV cache bytes = 2 × number of layers × number of KV heads × head dimension × bytes per element
Multiplying by sequence length and concurrency gives the total footprint. The following example uses a hypothetical configuration; the numbers are for illustration only:
def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, dtype_bytes=2, batch=1):per_token = 2 * layers * kv_heads * head_dim * dtype_bytesreturn per_token * seq_len * batch# Hypothetical config: 60 layers, 8 KV heads, head dim 128, BF16,# 1M context, single sequencesize = kv_cache_bytes(60, 8, 128, 1_048_576)print(f"{size / 2**30:.0f} GiB") # ~240 GiB
This result shows that simply adding more GPU memory cannot support routine long-context deployments. The industry commonly relies on four compression strategies:
- Fewer KV heads: multiple query heads share one set of key/value heads, shrinking the cache proportionally.
- Low-rank compression: project keys and values into a lower-dimensional latent space, cache only the latent vectors, and reconstruct them—or use them directly—at inference time.
- Selective retention: keep only tokens within a recent window, or only selected historical tokens.
- Cache quantization: store keys and values at lower precision, at the cost of increased numerical error.
Each approach makes different trade-offs among memory footprint, precision, and implementation complexity.
4. Routes to Sparse Attention
4.1 Sliding Window Attention
Each token attends only to the most recent w tokens, so both compute and cache grow linearly with sequence length. The drawback is that long-range information can only propagate across layers step by step, so models often keep global attention in some layers, alternating with windowed layers.
4.2 Sparse Selection
Historical tokens are scored, and each query selects only the top-k key/value pairs for computation. Compute scales with k, but scoring and indexing themselves require additional memory access. Below is a simplified single-head sketch, omitting the causal mask and multi-head handling:
import torchdef topk_kv_attention(q, k_cache, v_cache, topk):# q: [d], k_cache: [n, d], v_cache: [n, d]scores = k_cache @ q # [n]idx = torch.topk(scores, min(topk, scores.numel())).indicesattn = torch.softmax(scores[idx], dim=0)return attn @ v_cache[idx] # [d]
5. Closing Perspective
Across MoE routing, KV Cache compression, and sparse attention, the recurring pattern is the same: architectural changes are best understood as trade-offs against the three hardware constraints—compute, memory bandwidth, and communication. Evaluating any design choice starts by asking which of these resources it actually saves, and what it gives up in return.
