Modern LLM Architecture Explained: Parameters, Context Windows, and Hardware Constraints

    This article explains how modern large language model architectures work within three hardware limits: compute, memory capacity and bandwidth, and inter-device communication. It covers total versus active parameters in Mixture-of-Experts models, KV cache growth in long-context inference, trade-offs among attention compression methods, routing and residual design choices, and a step-by-step method for estimating deployment memory.

    Gains in LLM capability rarely come from adding parameters alone anymore. Model developers now rely more on structural design and resource scheduling so that the same hardware can support larger models and longer contexts. These changes look scattered, but they all come down to three hardware constraints: compute, memory capacity and bandwidth, and inter-device communication. Understanding these constraints is how you judge whether an architecture will run on your hardware, and how fast.

    1. Total vs. Active Parameters: Memory and Compute Are Billed Separately

    Mainstream LLMs have grown from tens of billions to hundreds of billions of total parameters, and some exceed 600B. During inference, however, each token uses only a small fraction of those parameters. The gap comes mainly from Mixture-of-Experts (MoE) networks.

    MoE replaces the standard feed-forward layer with multiple parallel expert networks. A router selects a few experts for each token. According to its public technical report, DeepSeek-V3 has about 671B total parameters and activates about 37B per token. It contains 256 routed experts, activates 8 of them per token, and includes one shared expert that is always active. Mixtral 8x7B uses 8 experts and selects 2 per token. It has about 47B total parameters and about 13B active parameters.

    This has two deployment implications. First, memory is determined by total parameters, because every expert must stay resident. The system cannot predict which experts a given token will select. At FP8, 671B weights need about 671GB of memory. The weights alone exceed the memory of a typical GPU server, so the model must be split across multiple GPUs or machines. Second, compute per token scales with active parameters, so MoE inference is usually faster than a dense model with the same total parameter count. The gap between these two numbers is the trade-off to weigh when deploying.

    2. Context Window: KV Cache Is the Main Memory Cost for Long Text

    Context lengths have grown from 128K to 1M tokens and beyond. Long context helps with code repository analysis, long document summarization, and multi-turn agent tasks. Its main cost, however, is the KV cache.

    The KV cache stores the Key and Value vectors of generated tokens so that later decoding steps can reuse them. Its size grows roughly linearly with sequence length, while the compute cost of standard attention grows quadratically. Keep these apart. The memory capacity pressure comes from the KV cache, and the compute latency pressure comes from attention.

    You can estimate the KV cache size per token with this formula:

    Bytes per token = 2 × layers × KV heads × head dimension × bytes per element

    The factor of 2 accounts for storing both K and V. Take a model with the same structural parameters as Llama 2 70B: 80 layers, 8 KV heads, a head dimension of 128, and BF16 storage (2 bytes per element). Each token takes about 320 KiB. At a 128K context, one sequence’s KV cache is about 40 GiB. With a batch size of 8, the KV cache alone needs about 320 GiB, before counting the weights.

    Queries do not need to be cached. At each decoding step, only the current token’s query is needed, and it can be discarded once attention is computed. KV cache optimization therefore targets Keys and Values.

    3. Trade-offs in Attention Compression Methods

    Attention compression aims to reduce cache size or compute while keeping model quality. The main options differ as follows.

    Method Relative KV cache size Main cost Best suited for
    Multi-head attention (MHA) 1 (baseline, KV heads equal query heads) Largest cache Short contexts, ample memory
    Grouped-query attention (GQA) About 1/(number of groups) Possible slight quality loss General-purpose LLMs, currently the most common
    Multi-query attention (MQA) About 1/(number of query heads) Higher quality risk Scenarios very sensitive to memory and latency
    Multi-head latent attention (MLA) Reported cache reduction of about 93% (DeepSeek-V2 technical report) Extra reconstruction compute, complex implementation Very long contexts where cache is the bottleneck
    Sliding window attention (SWA) Proportional to window size W, independent of total length Long-range dependencies rely on global layers Tasks with strong local dependencies, often mixed with global layers

    Using Llama 2 70B as an example, it has 64 query heads and 8 KV heads. Compared with multi-head attention, where each query head has its own KV set, GQA reduces the cache to one-eighth. Such gains need to be validated on the target task. Theoretical compression ratios alone are not enough.

    Some models interleave local attention layers with global attention layers, with the global layers preserving long-range information. The ratio and placement of local and global layers vary by model, so no fixed arrangement should be assumed to generalize.

    For positional encoding, Rotary Position Embedding (RoPE) and its extrapolation variants, such as NTK interpolation and YaRN, are widely used to extend context beyond the training length. These methods change how long-distance positions are represented. Their performance beyond the training length must be measured on long-context benchmarks.

    4. MoE Routing and Residual Design

    The current trend is to increase the total number of experts while keeping the number of active experts per token the same or lower. In other words, routing granularity becomes finer. Fine-grained experts are more likely to specialize, which reduces conflicts that occur when one expert learns many kinds of knowledge at once.

    Fine-grained routing also creates two problems. The first is load imbalance: some experts are selected frequently while others sit idle. Common fixes include an auxiliary load-balancing loss and a bias term that can be adjusted dynamically during expert selection. The second is higher communication volume. Each token must be sent to the devices hosting its k experts, and the results must be gathered back. Each MoE layer requires all-to-all communication for dispatch and combine, and a larger k means more traffic.

    For residual connections, some research goes beyond the standard residual path. Methods such as Hyper-Connections mix features across multiple paths between layers. These designs can help training stability, but they add memory read/write and communication overhead to each layer. Whether to adopt them depends on whether the training gains outweigh that extra cost.

    For feed-forward networks, SwiGLU is a gated linear unit used by models such as LLaMA. It uses three weight matrices. If the hidden dimension stays the same, it has about 50% more parameters than a standard two-matrix feed-forward layer. Implementations therefore usually shrink the hidden dimension to about two-thirds of the original to keep the total parameter count unchanged. SwiGLU is not the only option. GeGLU and plain GELU feed-forward layers are also used in some models.

    5. Hardware Constraints and Deployment Estimation

    Architecture design shifts pressure across three hardware resources.

    The first is memory capacity. Memory must hold weights, KV cache, activations, and framework overhead at the same time. The second is memory bandwidth. Each decoded token requires reading the weights activated at that step and the KV cache for the sequence, so decoding speed is usually bandwidth-bound. The third is interconnect bandwidth. MoE all-to-all communication is very sensitive to the gap between intra-node and inter-node bandwidth. Intra-node NVLink is much faster than inter-node RDMA networks such as InfiniBand or RoCE, so expert parallelism should stay within high-bandwidth domains where possible.

    Cloud platforms offer heterogeneous accelerators, high-speed RDMA networking, and object storage, which lower the barrier to running self-managed clusters. When choosing a platform, do not look only at single-card compute specs. Compare per-instance memory capacity, intra-node and inter-node interconnect bandwidth, billing models, and scaling response time. Validate all of these numbers against your own workload.

    Estimate memory before deployment with these steps:

    1. Set the target precision. BF16 uses 2 bytes per parameter, FP8 uses 1 byte, and INT4 uses about 0.5 bytes.
    2. Calculate weight memory by multiplying total parameters by bytes per parameter. For MoE models, use total parameters.
    3. Calculate KV cache memory by multiplying bytes per token by the maximum context length, then by the number of concurrent sequences.
    4. Reserve memory for activations, communication buffers, and framework overhead. A margin of 10% to 20% is typical, but the final value should be determined by measurement.
    5. If memory needs exceed one GPU’s capacity, decide how to split the model across tensor, pipeline, and expert parallelism. Then check whether cross-GPU communication after the split stays within high-bandwidth domains.
    6. Run load tests with your target context lengths and real request distribution. Focus on P99 latency and throughput rather than peak figures.

    Using the 671B model above as an example, FP8 weights take about 671GB. Eight GPUs with 141GB each provide about 1,128GB in total. After loading the weights, about 457GB remains. After subtracting the margin, the space left for KV cache shrin