Understanding Modern Large Language Model Architecture and Parameter Scale
This article explains how modern large language model architectures are shaped by hardware constraints in compute, memory access, and communication. It covers the distinction between total and activated parameters, layer layout strategies, attention sparsification for long context, mixture-of-experts routing, residual mixing, numerical precision formats, and operator-level accounting methods, along with engineering recommendations for deployment and capacity planning.
Understanding Modern Large Language Model Architecture and Parameter Scale
At first glance, modern large language model (LLM) design looks like an assembly of different architectural components. In essence, it is a systematic trade-off made under three categories of hardware constraints: compute, memory access, and communication. Total parameters have grown from hundreds of billions to several trillion, while activated parameters per token have long stayed in the tens-of-billions range. Context windows have expanded from 128K to 1M tokens, yet KV caches are being compressed repeatedly. Expert counts have increased from 256 to 896, while routing granularity keeps getting finer. Understanding the relationships between these trade-offs matters more than memorizing which operator a particular layer uses.
1. Why Architecture Should Be Reasoned Backward from Hardware Constraints
The inference cost of an LLM consists mainly of three parts: memory access overhead from loading weights, compute overhead from matrix multiplication and attention, and communication overhead from tensor parallelism, expert parallelism, and pipeline parallelism. An architecture that only pursues expressive power will easily hit the memory wall, bandwidth wall, or interconnect wall during deployment. Therefore, when analyzing a model architecture, ask three questions first: Can the weight volume fit into accelerator memory? Does the per-token activated compute match the target throughput? Will inter-layer and inter-expert communication be limited by interconnect bandwidth? The separation of total and activated parameters exists precisely to expand model capacity while controlling per-token compute. Attention sparsification and KV cache compression control memory access under long context. Fine-grained expert routing improves capacity utilization under a limited activated parameter budget.
2. Parameter Scale: Total Parameters, Activated Parameters, and Sparsity
When discussing parameter scale, at least three concepts should be distinguished: total parameters, activated parameters, and sparsity. Total parameters determine weight volume and model capacity. Activated parameters determine the dominant scale of per-token forward computation. Sparsity can be roughly understood as the ratio of total parameters to activated parameters. A rough estimate of weight volume is: total parameters multiplied by bytes per parameter. With low-precision weights, bytes per parameter drop and memory usage decreases accordingly, but precision loss must be controlled through calibration and evaluation. Per-token forward FLOPs can be approximated at roughly twice the activated parameter count, though the actual value is also affected by attention length, expert routing, and operator fusion, so simple extrapolation is unreliable. A parameter comparison table should include at least: total parameters, activated parameters, number of layers, layer layout, attention type, expert routing method, expert intermediate dimension, residual scheme, context length, and numerical format. Unpublished fields should be marked as unknown rather than inferred.
3. Layer Layout and Layer Ordering
Modern models rarely stack a single type of dense layer. Instead, they mix dense layers with sparse expert layers. The first few layers use dense structures, which helps stabilize early representations. Later layers alternate or segment into mixture-of-experts (MoE) layers to expand capacity at lower activated cost. Layer ordering also affects cache reuse and pipeline partitioning: if sliding-window attention layers, compressed attention layers, and full attention layers alternate, KV cache block management and scheduling policies must be differentiated by layer type. Layer count and hidden dimension jointly determine the granularity of per-layer operators and communication. More layers increase pipeline parallelism depth but also increase inter-layer synchronization. Therefore, a more complex layer layout is not necessarily better; it must be evaluated together with the parallelism strategy, batch size, and context length.
4. Attention Sparsification and Context Extension
Full attention’s compute and cache costs grow quickly with sequence length, so long-context scenarios must introduce sparsification. Common approaches include sliding-window attention, low-rank key-value compression, sparse index selection, block-wise compression, and hybrid combinations of multiple attention types. Sliding-window attention restricts each token’s visible range to a local window, significantly reducing cache size. Low-rank compression reduces per-layer cache volume by compressing key-value representations. Sparse index selection first computes a lightweight index and then selects a few historical blocks. Block-wise compression condenses long sequences into block-level representations. The value of hybrid attention lies in its division of labor: local layers handle fine-grained neighborhood information, compressed layers handle long-distance summarization, and a small number of full-attention layers preserve global interaction. In engineering, layer-type selection can be described with pseudocode:
for layer in model.layers:if layer.kind == "full":mask = causal_mask(seq_len)elif layer.kind == "window":mask = causal_mask(seq_len) & band_mask(window_size)elif layer.kind == "compressed":kv = block_compress(kv_cache, block_size, rank)
A rough estimate of full-attention KV cache size is: 2 × number of layers × number of KV heads × head dimension × sequence length × bytes per element. With low-rank or block-wise compression, the formula must be multiplied by the corresponding compression ratio. When context extends from 128K to 1M, cache pressure does not disappear linearly; it shifts from “cannot fit” to a trade-off between compression precision and indexing overhead.
5. Mixture of Experts and Routing Granularity
The core idea of mixture of experts is to trade sparse activation for capacity. As expert counts grow from 256 to 896, routing granularity becomes increasingly fine. A common practice is for each token to select its top-k experts while retaining shared experts to stabilize representations. Routing can be abstracted as:
def moe_forward(x, experts, shared_expert, k):logits = x @ gate_weight # [B,S,E]idx = topk(logits, k) # [B,S,k]prob = softmax(gather(logits, idx))y = shared_expert(x)for i in range(k):y += prob[..., i] * experts[idx[..., i]](x)return y
The expert intermediate dimension determines per-expert compute, while the expert count determines routing and communication complexity. Fine-grained experts reduce per-expert load but increase all-to-all communication and load-balancing difficulty. Shared experts do not participate in routing; they absorb general knowledge and reduce expert redundancy. Evaluation should focus on expert parallelism partitioning, capacity factor, token dropping strategy, and load-balancing loss—not merely the total expert count.
6. Residual Mixing and Numerical Formats
Residual connections are the foundation of stable training in deep models. Multi-path residual mixing can be understood as weighting or selecting among several residual branches to enhance information flow and training stability, at the cost of increased operator and memory management complexity. Regarding numerical formats, weights, activations, and the KV cache can each use different precisions: low-precision weights reduce volume, low-precision activations improve compute utilization, and low-precision KV caches reduce long-context memory access. Precision choices must be validated through perplexity, long-document evaluation, and task metrics—not inferred from theoretical benefits alone.
7. Operator-Level Decomposition and Accounting Methods
The attention path typically includes QKV projection, positional encoding, attention scores, softmax, weighted summation, and output projection. The feed-forward path includes gate projection, activation, element-wise multiplication, and down projection. MoE layers additionally include routing scores, top-k selection, expert computation, and result aggregation. An operator-level decomposition diagram should annotate tensor dimensions segment by segment, with cache, compute, and weight-volume accounting in the figure notes. A specification comparison table is used for horizontal comparison: when a series has multiple variants, the selection criteria should be stated explicitly, and unpublished items should be marked unknown rather than inferred. Deployment evaluation should simultaneously consider time to first token, per-token latency, throughput, memory usage, and interconnect communication volume.
8. Trends and Engineering Recommendations
Based on current public trends, total parameters will likely keep growing, but activated parameters will remain relatively restrained. Context will continue to expand, with KV cache compression and hybrid attention becoming standard. Expert counts will increase and routing granularity will become finer, while shared experts and load-balancing mechanisms grow more important. Residual mixing and low-precision numerical formats will continue to evolve. For engineering implementation, first build a specification comparison and operator accounting table, then validate throughput and long-context performance through stress testing. When deploying on a general-purpose cloud platform such as Baidu AI Cloud, you can build an evaluation loop using common service categories such as container platforms, object storage, logging services, and monitoring/alerting—but architecture selection should still return to the three fundamental constraints: compute, memory access, and communication. Only by weighing parameter scale, layer layout, attention sparsification, expert routing, and residual mixing in a single constraint table can you leave reasonable headroom for next-generation model architecture decisions and accelerator capacity planning.
