Modern LLM Architecture: Parameters, Model Design, and Hardware Constraints

    Modern large language models cannot be discussed in terms of architecture and parameter scale in isolation. This article explains how total parameters, active parameters, context length, expert counts, and routing granularity interact, and how these design choices map onto three fundamental hardware constraints: compute, memory bandwidth, and communication. It covers KV cache sizing, expert routing overhead, layer layout trade-offs, numerical formats, and a practical headroom model for accelerator capacity planning.

    Modern LLM Architecture: Parameters, Model Design, and Hardware Constraints

    The “architecture” and “parameter scale” of modern large language models cannot be discussed separately. On the surface, changes to attention, expert layers, residual connections, and numerical formats across different architectures seem scattered. But when they land on hardware, they almost all point to the same set of constraints: compute, memory bandwidth, and communication. Therefore, the key to understanding model architecture is not memorizing specific configurations, but seeing clearly the trade-offs among total parameters, active parameters, context length, expert count, and routing granularity—and using that understanding to estimate the capacity and bandwidth headroom that accelerator planning must reserve.

    1. Two Layers of Meaning in Parameter Scale: Total vs. Active Parameters

    When discussing parameter scale, two concepts must first be distinguished. In this article, B denotes billions of parameters and T denotes trillions. Total parameters determine weight size, memory capacity, weight-loading bandwidth, and checkpoint and communication overhead; active parameters determine how many parameters actually participate in computation per token, which is closer to compute demand. A common pattern in industry architecture evolution is that total parameters have grown from hundreds of B to several T, while active parameters have long stayed at tens of B. This shows that model capacity and per-token computation are being decoupled, with Mixture-of-Experts (MoE) architectures as the primary mechanism.

    Hardware planning cannot estimate compute from total parameters alone. Weight size can be approximated as:

    1. weight_bytes ≈ total_params × bytes_per_param

    And the feed-forward compute per token can be roughly expressed as:

    1. fflops_per_token ≈ 2 × active_params

    Attention, routing, aggregation, and communication must be accounted for separately. Substituting total parameters directly into compute formulas overestimates compute demand; looking only at active parameters underestimates memory capacity and weight bandwidth requirements.

    2. Architectural Trade-Offs Map to Three Types of Hardware Constraints

    Differences across models usually concentrate in four areas: layer ordering, attention sparsification, expert routing, and residual mixing. Each corresponds to a different hardware bottleneck:

    • Attention sparsification: reduces the quadratic complexity of attention scoring over long contexts, but may introduce irregular memory access and indexing metadata overhead.
    • KV cache compression: reduces the capacity and read/write bandwidth of the KV cache during decoding, but compression, sharing, or quantization may cause accuracy loss.
    • Expert routing: uses sparse activation to control per-token compute while expanding total capacity, but adds routing overhead, load imbalance, and expert-parallel communication.
    • Layer layout and residual mixing: affects pipeline partitioning, activation storage, and training stability, which in turn affect interconnect and bandwidth planning.

    Therefore, accelerator selection cannot focus solely on peak compute; memory capacity, memory bandwidth, inter-chip interconnects, and communication topology must be considered simultaneously.

    3. The Cost of Attention Sparsification and Context Extension

    When context length grows from 128K to 1M, the KV cache grows linearly with sequence length. Its size can be approximated as:

    1. kv_cache_bytes ≈ 2 × layers × kv_heads × head_dim × seq_len × batch × bytes_per_elem

    where the factor of 2 accounts for keys and values. In long-context inference, the decoding phase is often limited by KV read/write bandwidth rather than by matrix compute alone. Common techniques include low-rank compression, sparse indexing, sliding-window attention, paged management, quantized caches, and cross-layer sharing. The trade-off: the stronger the compression, the smaller the cache, but the more complex the indexing and gather/scatter operations; the shorter the sliding window, the less memory traffic, but the weaker the long-range dependency. Planning should reserve headroom for sequence-length peaks, cache fragmentation, and index metadata.

    4. Expert Routing and Fine-Grained Activation

    The growth of expert counts from 256 to 896, along with increasingly fine routing granularity, is a typical path to expanding total capacity while controlling active compute. By having each token select only a small number of experts, supplemented by shared experts, active parameters can be kept at the tens-of-B scale. However, as the number of experts grows, weight memory must hold all experts, expert parallelism introduces all-to-all communication, and routing imbalance causes tail latency.

    As an approximation, expert communication volume depends on token count, hidden dimension, top-k, and bytes per element:

    1. expert_comm ≈ 2 × tokens × top_k × hidden × bytes_per_elem

    Fine-grained routing improves specialization, but routing metadata, aggregation overhead, and hot-expert problems become more pronounced. Hardware planning needs to reserve headroom for expert-parallel interconnects, load imbalance, and expert weights resident in memory.

    5. Layer Layout, Residual Mixing, and Training Stability

    Layer ordering determines the ratio and sequence of dense and expert layers. Leading dense layers help stabilize representations; trailing expert layers expand capacity; mixed attention layers interleave sliding-window, compressed, and global attention. Layer layout changes affect pipeline-parallel partition balance and the bubble ratio. Multi-path residual mixing improves gradient propagation in deep networks, but increases activation storage and residual read/write bandwidth, and at low precision, numerical stability also requires attention.

    6. Numerical Formats and Operator-Level Accounting

    Low-precision formats such as FP16, BF16, FP8, and low-bit quantization can reduce weight size and bandwidth, but critical layers often require mixed precision. Operator-level accounting should follow two paths—attention and feed-forward:

    Path Key Operators Tensor Dimensions (Illustrative) Primary Accounting
    Attention QKV projection, positional encoding, QK^T, softmax, AV, output projection batch×seq×hidden, heads×seq×head_dim QK^T/AV compute, KV cache, indexed memory access
    Feed-forward Routing logits, top-k, expert linear layers, aggregation, residuals tokens×experts, tokens×top_k×hidden Active parameters, expert weights, all-to-all communication

    The illustrative accounting code below is only meant to show the relationships between variables; in practice, operator implementations and performance profiling must also be considered:

    1. def estimate_kv_bytes(layers, kv_heads, head_dim, seq_len, batch, bytes_per_elem=2):
    2. return 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem
    3. def estimate_weight_bytes(total_params, bytes_per_param):
    4. return total_params * bytes_per_param
    5. def estimate_fflops_per_token(active_params):
    6. return 2 * active_params

    7. A Headroom Model for Accelerator Planning

    Planning should account along four dimensions. Capacity: weights, KV cache, activations, temporary buffers, and fragmentation must all be counted. Bandwidth: weight streaming, KV reads/writes, and communication are the key concerns. Interconnect: expert parallelism, pipeline parallelism, and data parallelism each have distinct communication patterns. Compute: divide effective FLOPs by achievable utilization. In practice, a certain proportion of headroom—typically around 20%–30%—is usually reserved to absorb context peaks, routing imbalance, cache fragmentation, and precision conversion overhead.

    The overall trend is: total parameters continue to grow while active parameters rise slowly; contexts expand while caches are increasingly compressed; expert counts increase and routing becomes finer; layer layouts become more hybrid; and low precision and mixed precision become more widespread. Architecture evolution is not about chasing single-point compute, but about finding a balance among compute, memory bandwidth, and communication. Understanding this balance is what makes it possible to reserve reasonable headroom when planning infrastructure for the next generation of models.