Modern Large Language Model Architecture, Parameter Scale, and Hardware Constraints

    This article explains how modern large language models are structured and how their parameter scale relates to hardware limits. It covers the Decoder-only Transformer backbone, the difference between total and active parameters, attention and KV Cache design, MoE expert routing, residual and layer layout, and numerical formats. It also provides a framework for estimating weight, cache, and communication costs during deployment planning.

    The structure of modern large language models (LLMs) is not a single question of network topology. It results from the combined effects of parameter organization, attention computation, expert routing, cache management, and numerical formats. Public technical materials show that total model parameters have grown from hundreds of billions to several trillion, while the number of parameters activated per token has stayed at tens of billions for a long time. Context length has grown from 128K to 1M tokens, and KV Cache has been optimized repeatedly through compression, sliding windows, and sparse indexing. The number of experts has grown from hundreds to nearly a thousand, and routing granularity keeps getting finer. These changes look scattered, but they all point to the same hardware constraints: compute, memory access, and communication. This article uses a general technical perspective to break down the structure and parameter scale of modern LLMs and provides a reusable analysis framework.

    1. The Decoder-only Transformer Remains the Common Foundation

    Almost all mainstream LLMs use the Decoder-only Transformer as their backbone. The basic components include a token embedding layer, several decoder blocks, an output projection layer, and normalization. Each decoder block usually contains an attention sublayer and a feed-forward network sublayer, connected through residual connections and normalization. Common engineering implementations use general-purpose techniques such as pre-normalization, RMSNorm, rotary position embedding (RoPE), and gated feed-forward networks.

    From a structural analysis perspective, it is necessary to distinguish between the number of layers and the layer layout. The number of layers determines network depth and the number of cached layers. The layer layout determines how dense layers, mixture-of-experts (MoE) layers, and different attention layers are arranged. For example, some architectures place several dense layers first and several MoE layers after them. This balances the representation stability of shallow layers with capacity expansion in deeper layers. Other architectures use hybrid attention layers, applying full attention, sliding-window attention, or compressed attention in different layers. The layer layout directly affects weight sharding, expert-parallel communication, and KV Cache size, so total parameter count alone is not enough for analysis.

    A simplified decoder block can be abstracted as follows:

    1. # Pseudocode: abstraction of a Decoder-only block
    2. x = x + Attention(Norm(x))
    3. x = x + FeedForwardOrMoE(Norm(x))

    FeedForwardOrMoE can be a dense feed-forward network or a mixture-of-experts layer. In an MoE layer, each token activates only a small number of experts. This increases total parameters while keeping the compute per token under control.

    2. Parameter Scale: Analyze Total and Active Parameters Separately

    The most important distinction when analyzing the parameter scale of modern LLMs is between total parameters and active parameters. Total parameters determine the weight storage size and storage requirements. Active parameters determine the forward compute per token and part of the memory access for activations. In a dense model, total parameters roughly equal the parameters that take part in computation for each token. In an MoE model, total parameters include the weights of all experts, but each token passes through only the few experts selected by the router. As a result, active parameters are much smaller than total parameters.

    The general formulas are:

    1. Total parameters ≈ dense layer parameters + all routed expert parameters + shared expert parameters + embedding and output parameters
    2. Active parameters ≈ dense layer parameters + top-k routed expert parameters + shared expert parameters + embedding and output parameters

    These formulas explain why total parameters have grown from hundreds of billions to several trillion while active parameters have stayed at tens of billions. Growth in total parameters comes mainly from more experts, larger expert intermediate dimensions, and wider layers. Active parameters are controlled by top-k, the shared experts, and the size of the dense layers. In engineering practice, total parameters affect weight loading, sharding, and memory capacity. Active parameters affect compute per token and activation cache size. If capacity planning considers only total parameters, it underestimates communication and scheduling overhead. If it considers only active parameters, it underestimates weight storage and loading pressure.

    3. Attention Design: Full, Sliding-Window, Sparse, and Compressed Attention Evolve in Parallel

    Attention layers are the main source of cost in long-context scenarios. Common designs include multi-head attention, grouped-query attention, and multi-query attention. To reduce KV Cache, grouped-query attention lets several query heads share a small number of key-value heads. More aggressive approaches further compress key-value representations or use low-rank compression. Long-context models also often combine attention types, such as sliding-window attention, sparse-indexed attention, and compressed attention. Local layers use sliding windows to control cache size. Global layers keep key information, and compressed layers downsample historical information.

    KV Cache size can be estimated with the following general formula:

    1. KV Cache bytes ≈ 2 × layers × KV heads × head dimension × sequence length × batch size × bytes per element × compression ratio

    For example, assume 80 layers, 8 KV heads, a head dimension of 128, a sequence length of 128K, a batch size of 1, and 2 bytes per element. The uncompressed KV Cache is then about:

    1. 2 × 80 × 8 × 128 × 131072 × 1 × 2 ≈ 42.9 GB

    With 4x compression, this drops to about 10.7 GB. This example shows that extending context is not just a matter of increasing sequence length. Cache compression must be designed alongside it. Otherwise, the extra memory access caused by long contexts can quickly cancel the gains from higher compute.

    4. Expert Mixture and Routing: Balancing Capacity Expansion and Communication Overhead

    MoE architectures increase total parameters by using many experts and activating only a few of them for each token. A common configuration includes routed experts and shared experts. Shared experts apply to every token and provide baseline capability. Routed experts are selected per token and provide additional capacity. The number of experts has grown from hundreds to nearly a thousand, and top-k is usually a small number. Finer routing granularity increases expert specialization, but it also raises the cost of scheduling, load balancing, and cross-device communication.

    In engineering, the key constraint of MoE layers is expert-parallel communication. Each token must be sent to the device that hosts its selected expert, and the results must be sent back. When there are many experts and routing is scattered, all-to-all communication can become a bottleneck. Modern architectures therefore make trade-offs among expert intermediate dimension, top-k, the ratio of shared experts, and the routing algorithm. Capacity planning must estimate weight size, activation communication volume, and tail latency caused by uneven load.

    5. Residual Mixing and Layer Ordering: Effects on Stability and Hardware Mapping

    Residual connections are the foundation for stable training in deep networks. Some modern architectures add multi-path mixing or learnable scaling to the residual path to improve gradient flow in deep layers and the fusion of representations. Layer ordering also affects hardware mapping. Dense layers suit regular matrix multiplication. MoE layers need expert sharding and routing communication. Hybrid attention layers need different cache strategies. Placing dense layers in shallow layers and MoE layers in deep layers, or alternating different attention layers, are common engineering trade-offs.

    6. Numerical Formats and Hardware Constraints: The Compute, Memory, and Communication Triangle

    Numerical formats directly affect weight size, activation memory access, and matrix multiplication throughput. In general practice, training often uses BF16/FP16 and mixed precision. Inference can further use low-precision formats such as FP8 and INT8, with separate quantization for weights, activations, and KV Cache. Low precision does not simply save memory. It also affects numerical stability and accuracy loss, so it must be combined with calibration and fallback strategies.

    Hardware constraints can be summarized as a triangle. Compute sets the upper limit for matrix multiplication. Memory access determines how fast weights and caches load. Communication determines how well the system scales across multiple cards, multiple experts, and multiple layers. A practical capacity estimation function is shown below:

    1. def estimate_capacity(total_params, active_params, layers, kv_heads,
    2. head_dim, seq_len, batch, weight_bytes,
    3. kv_bytes, kv_compress=1.0):
    4. weight_gb = total_params * weight_bytes / 1e9
    5. kv_gb = (2 * layers * kv_heads * head_dim * seq_len *
    6. batch * kv_bytes * kv_compress) / 1e9
    7. # active_params is used to estimate per-token compute and activation memory access
    8. return {
    9. "weight_gb": weight_gb,
    10. "kv_cache_gb": kv_gb,
    11. "active_params": active_params
    12. }

    This function provides only an order-of-magnitude estimate. Actual planning must also account for activation cache, temporary buffers, communication buffers, and parallel sharding overhead. Teams should leave headroom to absorb long-context growth and uneven expert load, which can cause tail latency.

    7. Trend Analysis and Engineering Recommendations

    Taken together, the structures discussed above point to several trends. Total parameters will keep growing, but active parameters will stay relatively contained. Context will keep expanding, and KV Cache compression and hybrid attention are becoming required design elements. The number of experts is increasing and routing is becoming finer, but communication and scheduling costs rise with it. Low-precision numerical formats are being introduced earlier in the process rather than only as post-training quantization. For architecture selection, teams should first identify the business scenario, such as high-concurrency short-context serving, low-concurrency long-context serving, or expansion of capacity through many experts. Then they can choose a parameter structure that matches that scenario.

    For deployment and capacity planning, we recommend the following:

    1. Estimate weight size, KV Cache, and activation cache separately. Do not estimate using total parameters alone.
    2. For long-context workloads, evaluate cache compression ratios and the attention layer layout first.
    3. For MoE models, focus on expert-parallel communication and load balancing.
    4. Keep a fallback path for accuracy when using low-precision formats.
    5. When deploying on cloud platforms, use general capabilities such as container platforms, object storage, logging services, and monitoring and alerting. Use them to build systems for weight distribution, cache management, and performance observability.

    Only by analyzing structure, parameter scale, and hardware constraints together can teams leave reasonable headroom for next-generation model selection and accelerator planning.