Modern LLM Architectures and Parameter Scaling: A Technical Breakdown

    This article analyzes the dominant architectural patterns of modern large language models through the lens of three underlying hardware constraints: compute, memory access, and communication. It covers the shift from dense Transformers to Mixture-of-Experts, the divergence between total and activated parameters, low-rank KV compression and sparse attention for long contexts, expert routing design, numerical formats, and the common trends shaping next-generation model architectures—along with practical guidance for deployment cost estimation.

    1. Why Architecture Deserves Its Own Discussion

    Recent LLM development may look fragmented: some models add experts, some extend context windows, some compress attention, and some change numerical formats. But when these changes are plotted on the same coordinate system, nearly all of them point to the same set of underlying constraints: compute, memory access, and communication.

    • Compute constraints determine how many floating-point operations a single forward pass can perform, pushing models from dense activation toward sparse activation.
    • Memory access constraints determine the cost of moving weights and intermediate states (especially the KV Cache), driving continuous compression of key-value representations in attention.
    • Communication constraints determine the cost of data exchange across GPUs and nodes during training and inference, driving refined designs for expert parallelism, tensor parallelism, and layer layout.

    With these three threads in mind, the seemingly arbitrary design choices across model families all have clear motivations. This article uses them as a framework to dissect the architectural elements and parameter scaling patterns of modern LLMs.

    2. Overall Architecture: From Dense Transformers to Mixture-of-Experts

    2.1 The Baseline of Dense Architectures

    The dominant backbone of modern LLMs remains the decoder-only Transformer: a stack of layers, each containing a multi-head attention module and a feed-forward network (FFN) module, with residual connections and normalization layers. A typical layer configuration can be expressed as pseudo-config:

    1. model:
    2. layers: 60~80 # depth grows with scale
    3. hidden_size: 4096~8192
    4. num_attention_heads: 64~128
    5. ffn_type: SwiGLU # gated activation is now mainstream
    6. normalization: RMSNorm
    7. position_encoding: RoPE

    The combination of SwiGLU gated FFNs, RMSNorm, and rotary position embeddings has become the de facto standard configuration for most open-source and proprietary models today.

    2.2 Changes in Layer Ordering

    Beyond the standard serial ordering of “attention first, FFN second,” some models have begun experimenting with hybrid layer arrangements: keeping several dense layers at the beginning of the sequence to stabilize early representations, or inserting special layers in the middle and later stages (such as linear attention layers or sliding-window layers) to reduce long-context overhead. This “dense start, sparse majority” layout is essentially a staged trade-off between model quality and inference cost.

    2.3 Parameter Scale: The Growing Gap Between Total and Activated Parameters

    Across the industry, model parameters show a clear pattern:

    Dimension Trend Driver
    Total parameters Growing from hundreds of B to several T Capacity demands—raising the ceiling of knowledge and reasoning
    Activated parameters Staying in the tens of B long-term Keeping inference cost manageable; limited per-token compute
    Context length Extending from 128K to 1M Long-document and long-horizon task requirements
    Number of experts Growing from hundreds to nearly a thousand Fine-grained routing improves expert utilization
    KV Cache Repeatedly compressed Memory and bandwidth pressure under long contexts

    This widening gap is the direct reason for the rise of Mixture-of-Experts architectures: trading sparse activation for large capacity. Each token activates only a small fraction of experts (for example, 8 out of 256). Total parameters determine how much the model “knows,” while activated parameters determine how much it “computes per step.” For inference serving, the latter is what actually drives cost.

    3. Attention: Two Routes of Sparsification and Low-Rank Compression

    Attention is the main source of memory-access overhead in long-context scenarios, making it the most heavily modified module. There are currently two main technical routes:

    3.1 Low-Rank Compression of Keys and Values

    In traditional multi-head attention, each head maintains its own keys and values, so KV Cache size grows linearly with the number of layers and heads. The low-rank approach projects keys and values into a latent space far smaller than the query dimension (for example, a query rank in the thousands with key-value rank compressed to the hundreds), caches only the compressed latent vectors at inference time, and reconstructs on demand. This can shrink KV Cache size by roughly an order of magnitude with almost no accuracy loss. Additionally, decoupling rotary position embeddings onto separate dimensions balances compression rate against position sensitivity.

    3.2 Sparse Attention and Hierarchical Retrieval

    For ultra-long contexts, even a compressed KV Cache grows linearly with sequence length. The idea behind sparse attention is that not every query token needs to attend to all historical tokens. Actual key-value pairs involved in computation are reduced through:

    • Sliding windows: nearby tokens receive full attention, while long-range dependencies are handled by other mechanisms;
    • Coarse-grained selection: importance scores for historical tokens are first computed in a low-precision, coarse manner, and only top-k key-value blocks participate in exact computation;
    • Hierarchical compression: different compression ratios are applied to history at different distances—the farther away, the more aggressive the compression.

    At million-token context scales, the common combination is a hybrid of “local window + sparse selection,” reducing actual attention computation from quadratic in sequence length to approximately linear.

    4. Mixture-of-Experts: Routing Granularity and Shared Experts

    The core design parameters of an MoE layer fall into three groups:

    1. Total experts and activated experts: an evolution from “8 out of 256” toward “6–8 out of nearly a thousand.” As the total number of experts grows while each expert’s intermediate dimension shrinks—so-called fine-grained experts—combinatorial expressiveness improves while activated parameter count stays stable.
    2. Shared experts: one (or a few) shared experts that every token must pass through, handling cross-token general computation and reducing redundancy among routed experts.
    3. Routing strategy: top-k selection based on similarity between tokens and expert centroid vectors, combined with load-balancing losses to prevent expert collapse.

    From a hardware perspective, more and finer experts mean larger weight volume but more fragmented per-step communication, which raises the bar for expert-parallel scheduling and communication pipelines, and has driven engineering optimizations that overlap computation with communication in both training and inference.

    5. Numerical Formats and Residual Mixing

    • Numerical formats: low-precision training and inference (such as the FP8 family of formats) has become a standard means of reducing compute and memory overhead, typically applied in tiers across weights, activations, and caches, with scaling strategies to maintain numerical stability.
    • Residual mixing: as depth increases, managing the scale of the residual stream becomes increasingly important. Some models introduce multi-path residual connections (such as four-way branches), injecting the outputs of different functional sublayers (attention, expert networks, shared networks, etc.) into the residual stream separately, mitigating gradient and activation anomalies in very deep networks.

    6. Common Patterns in Architectural Evolution and Outlook

    Synthesizing the analysis above, four architectural “greatest common denominators” emerge:

    1. Decoupling capacity from cost: total parameters keep expanding while activated parameters remain restrained—this is the precondition for MoE architectures to persist long-term.
    2. KV Cache is the number-one enemy of long context: every attention modification—low-rank compression, sliding windows, sparse retrieval—ultimately serves the same goal: keeping cache size and actual computation from spiraling out of control with context length.
    3. Routing granularity keeps refining: more, smaller experts combined with shared experts are replacing the early design of “a few large experts.”
    4. All changes ultimately trace back to hardware constraints: the three constraint lines of compute, memory access, and communication are a reliable basis for predicting where next-generation models are headed. Architectural innovation is essentially about reshaping the workload to better match the execution characteristics of the underlying hardware.

    For teams doing capacity planning or inference deployment, evaluating a new model should not stop at total parameter count. The key quantities to compute are: activated parameters per token, the KV Cache size formula (number of layers × compressed key-value rank × context length × bytes of precision), and the memory footprint and communication pattern of expert weights. Working out these three numbers enables reliable estimates of deployment cost and hardware selection. Major cloud platforms offering model services (such as Baidu AI Cloud’s foundation model services) commonly tier their service specifications by activation scale and context length, and their pricing and quota logic likewise stems from the structural constraints described above. Understand the architecture, and you understand the cost; understand the trade-offs, and you can anticipate the shape of the next generation of models.