Decoding Modern Large Language Model Architecture and Parameter Scaling
This article analyzes the architectural evolution of modern Large Language Models (LLMs), focusing on the underlying hardware constraints of compute, memory access, and communication. It explores the decoupling of total and activated parameters through Mixture of Experts (MoE), layer layouts, attention sparsification for long context handling, and expert routing. Additionally, it examines numerical formats, hardware accounting methods, and future trends in LLM architecture design.
I. Common Hardware Constraints Behind Structural Differences
Discussing the structure of modern LLMs goes beyond configuration tables like the number of layers, heads, or experts. Variations in attention, Mixture of Experts (MoE), residual connections, and numerical formats across different public approaches may seem scattered, but on accelerators, they are ultimately governed by three constraints: compute, memory access, and communication.
- Compute determines the forward computation per token, roughly proportional to the activated parameter count;
- Memory access dictates the bandwidth pressure for weight loading, KV Cache read/write, and activation caching;
- Communication determines the scaling efficiency under expert parallelism, tensor parallelism, and pipeline parallelism.
Thus, while a model’s total parameters can grow from hundreds of billions (B) to trillions (T), the activated parameters per token have long remained at the tens of B level. The reason is that total parameters mainly expand model capacity, while activated parameters determine inference compute; the two are decoupled through the MoE architecture. Similarly, expanding context from 128K to 1M does not mean linear KV Cache growth; instead, it drives attention sparsification schemes like low-rank compression, sliding window, block sparsity, index sparsity, and hierarchical compression. As the number of experts evolves from hundreds to nearly a thousand, the routing top-k remains in single digits because finer routing granularity makes load balancing and all-to-all communication more difficult.
In this article, B denotes billion parameters, T denotes trillion parameters, and token is the unified unit. The following sections abstract a reusable architectural analysis framework rather than targeting specific products.
II. Parameter Scaling: Total Parameters, Activations, and MoE
1. Decoupling Total and Activated Parameters
In dense models, processing a token requires passing through almost all parameters, so the activated parameters per token are close to the total parameters. MoE models split the feed-forward network into multiple experts, routing each token to only a few experts. Therefore, the total parameters can be very large while activated parameters remain small.
This leads to two direct consequences:
- Weight volume is determined by total parameters. Even if only partial experts are activated during inference, all expert weights must still reside in memory or be swapped in on demand, making memory capacity and weight loading bandwidth the bottlenecks.
- Compute is determined by activated parameters and routing distribution. Concentrated routing turns hot experts into compute bottlenecks; dispersed routing increases all-to-all communication.
A simplified formula estimates weight volume:
def weight_bytes(total_params, bytes_per_param):# total_params: Total number of parameters# bytes_per_param: Bytes per parameter, e.g., 2 for BF16, 1 for FP8return total_params * bytes_per_param
2. Number of Experts, Routing Granularity, and Shared Experts
Common MoE designs include several shared experts plus several routed experts, selecting top-k experts during routing. Increasing experts can enhance model capacity and combinatorial capabilities, but also brings the following issues:
- Increased routing communication cost: Tokens must be dispatched to experts on different devices and then combined;
- Increased load balancing difficulty: Overloaded experts slow down the entire pipeline;
- More sensitive intermediate dimension design: Large expert intermediate dimensions increase weight volume and communication; small ones lack capacity.
Finer routing granularity can assign a token to more small experts, improving combinatorial flexibility, but all-to-all communication grows roughly linearly with top-k. Engineering practices typically use capacity factors, load balancing auxiliary losses, expert parallelism grouping, and communication overlap to mitigate these issues.
III. Layer Layout: Dense Initiation and Mixed Attention Alternation
Layer layout is not simply “stacking layers,” but a joint arrangement for training stability, memory peak, and inference efficiency. Common patterns include:
- Dense feed-forward networks in the first few layers to stabilize early representations and reduce routing jitter;
- Alternating MoE layers in the middle and later stages to boost capacity;
- Attention layers alternating between sliding window, compression, global, or index sparse to control KV Cache under long contexts;
- Sparser attention or finer expert routing in deeper layers, as deep semantics are more abstract and less dependent on local details.
Layer ordering directly impacts pipeline parallelism partitioning. If memory peaks are exceptionally high in certain layers, pipeline load becomes unbalanced; if attention and expert layers are interleaved poorly, the overlap of communication and computation degrades. Therefore, modern architectures often treat “layer layout” as a design variable equally important as hyperparameters.
IV. Attention Sparsification: Context Extension and KV Cache Compression
The time and memory of full multi-head attention grow quadratically or linearly with sequence length. When context expands from 128K to 1M, the KV Cache becomes the primary memory access bottleneck. Common compression paths include:
- Reducing KV head count: Multi-Query Attention and Grouped-Query Attention reduce caching by sharing K/V heads;
- Low-rank KV compression: Projecting K/V into a low-rank space to reduce per-token cache;
- Sliding window attention: Retaining only recent tokens within a window, lowering local modeling costs;
- Block sparsity and index sparsity: Selecting relevant blocks via index structures to reduce full computation;
- Hierarchical compression attention: Compressing distant tokens to retain global information;
- Mixed attention: Preserving full attention in some layers while using sparse or compressed attention in others.
KV Cache can be estimated as:
def kv_cache_bytes(layers, batch, seq, n_kv_heads, head_dim, bytes_per_elem):# Cache K and V for each layer per tokenreturn layers * batch * seq * 2 * n_kv_heads * head_dim * bytes_per_elem
This formula shows that doubling context roughly doubles KV Cache; reducing KV heads or head_dim directly reduces cache. However, over-compression degrades recall, so “mixed sparsity” is often used rather than full compression: critical layers retain high-fidelity attention, while most layers use window or compressed attention.
V. Expert Routing and Residual Mixing
The core of expert routing is “trading conditional compute for capacity.” The routing formula can be abstracted as: compute expert scores for each token, select top-k experts, and compute a weighted sum. Shared experts always participate, carrying general knowledge and reducing redundancy among routed experts.
Engineering focuses include:
- Routing stability: Early training routing jitter causes drastic changes in expert load;
- Capacity factor: Limits tokens processed per expert; overflow tokens go through residuals or are dropped;
- All-to-all communication: Dispatch and combine are the main overheads of MoE scaling;
- Residual mixing: Multi-path or gated residuals improve deep gradient propagation but increase activation caching and operator fusion difficulty.
MoE communication can be roughly estimated:
def moe_comm_bytes(tokens, topk, hidden, bytes_per_elem):# Approximate lower bound: one dispatch plus one combinereturn 2 * tokens * topk * hidden * bytes_per_elem
VI. Numerical Formats and Hardware Accounting
Low precision is a direct way to reduce weight volume and bandwidth pressure. The use of BF16, FP8, and lower bit-width formats in weights, activations, and KV Cache requires layered handling: normalization, softmax, and accumulation typically retain higher precision, while matrix multiplications can use low precision. KV Cache quantization significantly reduces long-context memory, but errors accumulate with layer count and sequence length.
Hardware-oriented accounting should include at least: weight volume, KV Cache, activation cache, MoE communication, and per-token compute. Operator-level decomposition can list tensor dimensions segment by segment along the attention and feed-forward paths, such as QKV projection, softmax, AV multiplication in attention, gating, dimension lifting, dimension reduction in feed-forward, and dispatch, expert compute, and combine in MoE. Only by placing operators, formulas, tensor dimensions, and bytes in the same accounting table can one determine whether the bottleneck lies in compute, memory access, or communication.
VII. Trend Analysis
Overall, the structural trends of next-generation LLMs can be summarized as: Total parameters continue to expand, activated parameters grow moderately; attention moves from full to mixed sparse; expert routing becomes finer-grained while retaining shared experts; residual connections multi-path to stabilize deep training; numerical formats continue to explore lower bit-widths; context extension relies more on the synergy of compression, retrieval, and sparse indexing. For accelerator planning, one cannot simply leave margin based on peak compute; sufficient bandwidth and interconnect margins must be reserved for weight loading, KV Cache read/write, and all-to-all communication. The essence of structural design is making accountable trade-offs among capacity, compute, memory access, and communication.
