LLM Architecture Explained: Parameter Scale, Sparsity, and Hardware Constraints

    This article explains how large language model architecture choices—parameter scale, dense and sparse layers, attention design, mixture-of-experts routing, numerical formats, and KV cache behavior—affect training cost, inference efficiency, and deployment under hardware constraints. It outlines the key metrics to evaluate, including total and activated parameters, layer layout, attention strategy, expert routing, weight and activation memory, cache size, and communication overhead.

    1. Why Analyze LLM Architecture Separately

    The capabilities of large language models are usually associated with training data, optimization methods, and parameter scale, but training cost and inference efficiency are often determined by network architecture. Even when two models have similar total parameter counts, differences in layer count, attention form, expert routing, or numerical format can lead to significant differences in memory usage, compute density, communication overhead, and latency.

    From a hardware perspective, current model evolution is mainly constrained by three types of resources:

    1. Compute resources: Matrix multiplication, attention computation, and expert feed-forward networks require substantial compute.
    2. Storage and memory access resources: Model weights, activations, and key-value caches continuously consume memory bandwidth and capacity.
    3. Inter-device communication resources: Sparse expert models must dispatch different tokens to different experts, and communication can become a bottleneck.

    Therefore, analyzing a model cannot rely only on “total parameter count.” More useful metrics include total parameters, activated parameters per token, layer count, hidden dimension, number of attention heads, number of experts, number of activated experts per token, context length, and numerical formats for weights and caches.

    2. Parameter Scale: Total Parameters Are Not the Same as Per-Token Compute

    The parameter scale of large language models is generally expressed in B or T, representing billions and trillions of parameters, respectively. In traditional dense models, every token passes through most parameters, so total parameter count is highly correlated with per-token compute.

    Mixture-of-experts (MoE) models are different. They split the feed-forward network into multiple experts and use a router to select a small number of experts for each token. This creates:

    • A larger total parameter count: The model has more experts and greater capacity.
    • Relatively limited activated parameters: A single token invokes only some experts.
    • A more complex communication process: Tokens must be redistributed across devices.
    • Higher deployment requirements: All expert weights usually still need to be stored in the cluster.

    For example, a class of large sparse models may have hundreds of B-level total parameters, while only tens of B parameters are activated per token. Larger designs may reach T-level total parameters while keeping the activated scale in the tens of B. Thus, total parameters mainly reflect model capacity, while activated parameters are closer to the per-token compute cost.

    However, activated parameters do not fully represent inference latency. Expert count, routing balance, device topology, communication bandwidth, and batch size all affect final performance. A model with fewer activated parameters may still be slower than a model with more parameters but more regular communication if its expert distribution is unreasonable.

    3. Layer Count and Layer Layout: Depth, Width, and Structural Mixing

    The main body of a model usually consists of repeated Transformer-like modules. Each layer includes an attention sublayer, a feed-forward sublayer, normalization modules, and residual connections. Common designs adjust the following directions:

    1. Increasing layer count

    Increasing layer count can improve network depth and give the model stronger compositional reasoning ability. However, more layers directly lengthen the serial execution path, which can make per-request latency worse, and they increase overhead for activation storage, synchronization, and scheduling.

    2. Increasing hidden dimension

    Increasing the hidden dimension can improve the representation capacity of a single layer, but the compute of matrix multiplication usually grows approximately with the square of the hidden dimension, and weight and activation usage also increase.

    3. Mixing dense and sparse layers

    Some models keep dense feed-forward layers at the bottom or top and introduce mixture-of-experts layers in the middle. Dense layers can provide stable base representations, while expert layers expand capacity. This layout helps strike a balance among quality, compute, and communication cost.

    4. Using different attention strategies in different layers

    Long-context models may use sliding-window attention, compressed attention, or global attention at different depths. Lower layers focus more on local patterns, middle layers handle information interaction, and a small number of higher layers retain global aggregation capability. This heterogeneous layer design can reduce overall attention cost.

    4. Attention Mechanisms: From Full Connectivity to Compression and Sparsity

    Standard self-attention generates three tensors: queries, keys, and values. For a sequence of length L, full attention computation and storage usually grow quadratically with sequence length. When context expands from tens of K to hundreds of K or even longer, using standard attention directly creates high compute and memory pressure.

    1. Multi-head attention

    Multi-head attention provides independent representation capacity for different subspaces. Its advantages are strong expressiveness and mature implementation, but the key and value caches are large and consume significant memory during long-text generation.

    2. Grouped or shared key-value heads

    By letting multiple query heads share some key-value heads, it is possible to reduce key-value projection parameters and cache capacity. Such methods are especially important for the generation phase, because each generated token requires reading historical keys and values.

    3. Latent compression

    Another class of methods does not directly store full keys and values but compresses them into a low-dimensional latent representation and restores the required information when used. Let the hidden state be (X) and the compression matrix be (W_c). This can be expressed as:

    [
    C = XW_c
    ]

    Here, (C) has a lower dimension than the original key-value tensor. Caching (C) instead of full K and V during generation can help reduce cache capacity and memory bandwidth demand, but it adds decompression or reconstruction compute.

    4. Sparse attention

    Sparse attention accesses only some historical tokens, such as local windows, block regions, or candidate positions filtered by an index. Its core goal is to reduce attention complexity from full-sequence access to limited-range access.

    It is important to note that sparse attention does not mean the model completely ignores long-distance information. Practical systems usually retain long-range dependencies through a small number of global tokens, compressed summaries, or cross-block connections.

    5. Mixture of Experts: Routing Granularity Determines Efficiency

    A mixture-of-experts layer usually consists of multiple feed-forward networks. The router computes expert scores based on token representations and then selects several experts to process the current token. The abstract process can be written as:

    [
    p=\text{Softmax}(XW_r)
    ]

    [
    Y=\sum_{i\in S(x)}p_iE_i(X)
    ]

    Here, (W_r) is the router parameter, (E_i) represents the (i)-th expert, and (S(x)) is the selected expert set.

    1. Number of experts

    More experts provide greater theoretical capacity, but parameter storage and scheduling complexity also increase. Large models may use hundreds or even close to a thousand experts, while each token activates only a few.

    2. Number of activated experts

    The number of experts activated per token directly affects compute. Too few activated experts may limit expressiveness; too many can weaken the benefits of sparsity.

    3. Shared experts

    Some architectures add a shared expert that all tokens pass through, while also selecting several routed experts. The shared expert handles general knowledge, and routed experts handle domain-specific or pattern-specific information, which can help reduce routing instability.

    4. Load balancing

    If a large number of tokens concentrate on a few experts, capacity overflow, queuing, and idle devices can occur. In engineering, load-balancing losses, capacity limits, or dynamic scheduling mechanisms are usually needed. Routing strategy affects not only model quality but also communication efficiency.

    6. Residuals and Numerical Formats: Stability and Deployment Cost

    Residual connections can alleviate optimization difficulties in deep networks. A common approach is to add the sublayer output to the input:

    [
    X_{l+1}=X_l+F(X_l)
    ]

    The placement of normalization, residual scaling, and the order in which different sublayers are combined all affect training stability and gradient propagation.

    In terms of numerical formats, mixed precision is often used during training to reduce memory usage and improve matrix compute throughput. During inference, lower-bit weight or activation representations may be used further. The benefits of low precision include:

    • Smaller weight size;
    • Lower memory bandwidth pressure;
    • Higher matrix operation throughput;
    • Easier deployment on devices with limited memory.

    However, low precision can also cause accuracy loss, amplified outliers, and accumulated errors. Therefore, lower bit width is not automatically better; differentiated treatment is usually needed based on the importance of attention, routers, normalization, and output layers.

    7. The Core Bottleneck of Long Context Is the Cache

    During autoregressive generation, the model stores keys and values corresponding to historical tokens. This data is called the KV Cache. If the sequence length is (L), the layer count is (N), the number of cache heads is (H), each head dimension is (D), and the number of bytes per value is (b), the cache size can be approximated as:

    [
    M_{\text{KV}}\approx 2\times L\times N\times H\times D\times b
    ]

    The 2 corresponds to keys and values.

    As context length expands, the KV Cache grows approximately linearly. Even if model weights fit in memory, the cache can become the main limitation for long-text inference. Therefore, long-context optimization usually focuses on the following directions:

    • Reducing the number of key-value heads;
    • Compressing key-value representations;
    • Using local or block attention;
    • Summarizing historical content;
    • Quantizing the cache;
    • Placing part of the cache in different storage media by layer.

    This is also why KV Cache compression continues to receive attention as context lengths grow.

    8. Hardware Evaluation Should Track Four Types of Metrics

    When planning model deployment, at least the following should be calculated:

    1. Weight capacity

    [
    M_{\text{weight}}\approx P\times b
    ]

    Here, (P) is the number of parameters, and (b) is the number of bytes per parameter. Actual deployment also needs to account for quantization metadata, shard replicas, and runtime buffers.

    2. Activation memory

    Activation memory depends on batch size, sequence length, layer count, and hidden dimension. Training must also store intermediate results required for backpropagation, so it is much higher than inference.

    3. KV Cache

    Long contexts and concurrent requests jointly amplify cache requirements. System design cannot estimate only single-request capacity; it must also consider multiple users generating at the same time.

    4. Communication volume

    Sparse expert models need to perform token dispatch and result collection. If inter-device communication cannot overlap with compute, accelerator utilization may drop. Therefore, expert placement, topology-aware routing, and communication operator optimization are critical.

    Overall, future large language models may show several directions:

    1. Total parameters continue to grow, while activated parameters remain controllable;
    2. Attention becomes more sparse and compressed;
    3. Expert routing shifts from coarse-grained selection to finer-grained combination;
    4. Context length expansion and cache compression advance together;
    5. Different layers use different structures instead of fully homogeneous stacking;
    6. Weights, activations, and caches use layered numerical formats;
    7. Model architecture design pays more attention to hardware topology and communication cost.

    Therefore, evaluating a model cannot rely only on total parameter count or context length. A more complete approach is to first confirm the actual compute path per token, then analyze the memory requirements of weights, activations, and caches, and finally evaluate the communication cost introduced by expert routing and attention sparsity. Only by placing algorithmic structure and hardware constraints in the same framework can we accurately assess training difficulty, inference efficiency, and deployment feasibility.