Modern LLM Architecture Evolution and Parameter-Scale Trade-offs

    This article presents a neutral technical framework for comparing modern large language model architectures. It examines the tension among total and activated parameters, context length and KV-cache compression, and expert count and routing granularity. The discussion covers dense-sparse layer placement, attention sparsification and long-context caching, expert routing with shared experts, residual mixing and numeric formats, operator-level accounting for capacity planning, and planning headroom for next-generation models. Readers gain a structured way to translate architectural trade-offs into memory, bandwidth, interconnect, and compute requirements rather than relying on peak specifications alone.

    1. Three Curves That Frame Architecture Evolution

    Modern large language models can be abstracted into three curves that constrain one another. First, total parameters have grown from hundreds of billions to several trillion, while activated parameters per inference have long remained in the tens of billions. Second, context windows have expanded from 128K to 1M, while KV cache has instead been repeatedly compressed. Third, expert counts have increased from 256 to 896, with increasingly fine-grained routing. These curves do not change in isolation. Together they point to a central question: how should model capacity, compute efficiency, memory bandwidth, and interconnect overhead be traded off?

    Understanding these trade-offs is more important than memorizing the specifications of any single model generation. The next generation will most likely continue along the direction of larger total parameters, more controlled activation, longer context, finer experts, and lower precision, but each step will place different pressure on accelerator planning. This article rebuilds a comparable architecture analysis framework from a neutral technical perspective.

    2. Parameter Scale: Decoupling Total and Activated Parameters

    Total parameters determine how much knowledge the model can hold, while activated parameters determine the per-token forward compute. In traditional dense models, the two are essentially the same. Mixture-of-Experts structures decouple them: total parameters can reach hundreds of billions to several trillion, but each token passes through only a small number of experts, keeping activated parameters in the tens of billions. The benefit is that capacity can scale without linearly driving up compute. The cost is larger weight volume, expert-parallel communication, and more complex routing load balancing.

    At least three quantities matter during planning: weight volume is approximately total parameters multiplied by bytes per parameter; per-token compute is approximately proportional to activated parameters; and resident memory must also include KV cache, activation intermediates, and communication buffers. Low-precision numeric formats can significantly reduce weight volume, but they cannot eliminate all-to-all communication in expert parallelism. Therefore, “large total parameters and small activation” does not mean low hardware requirements. It shifts pressure from pure compute to memory capacity, memory bandwidth, and inter-card interconnect.

    3. Layer Order: Mixing Dense and Sparse Layers

    Layer order is an often-overlooked difference. A common approach uses dense structures in the first several layers to stabilize training and general representations, then inserts Mixture-of-Experts layers in later layers to expand capacity. Other models interleave pure sliding-window layers, compressed-attention layers, and sparse layers. Different layer orders directly affect pipeline-parallel split points, load balance across stages, and the frequency of expert-parallel communication.

    When analyzing layer layout, at least record the total number of layers, the ratio of dense to sparse layers, the distribution of attention types, the insertion positions of expert layers, and the hidden dimension and head configuration. If dense layers are concentrated in the front, the front stages of the pipeline carry more compute. If expert layers are dense in the later stages, expert-parallel communication is more concentrated. For accelerator planning, layer order is not a paper specification; it is structural information that determines which segment becomes the bottleneck first.

    4. Attention Sparsification and Long-Context Caching

    Expanding context from 128K to 1M has a direct cost: KV cache growth. Under standard multi-head attention, KV cache can be approximated as:

    1. kv_bytes = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem

    The factor 2 represents the K and V tensors. When sequence length doubles, the cache roughly doubles. Long-context models therefore commonly use low-rank compressed attention, sliding windows, sparse indexing, and hybrid attention with different compression rates. The goal is to reduce cache reads and writes while preserving long-range information.

    Compression is not free. The higher the compression rate, the more complex the indexing and selection logic becomes. The larger the sparse top-k, the harder it is for attention operators to maintain high utilization. When sliding-window layers are mixed with global layers, different cache policies across layers add implementation complexity. When evaluating long-context capability, do not look only at “supports 1M.” Also examine effective cache size, per-token attention read/write volume, number of index heads, compression rate, and cache fragmentation under batching.

    5. Expert Routing: Count, Granularity, and Shared Experts

    Expert counts have increased from 256 to 896, and routing has moved from coarse to fine granularity. The essence is to find a finer compromise between total-parameter expansion and activation sparsity. A common structure is “routed experts + shared experts”: each token selects top-k routed experts and also passes through shared experts to retain general capability. Expert intermediate dimensions are often in the 2K–3K range. As expert count increases, each expert can be narrower and routing combinations more flexible.

    The benefit of fine-grained routing is stronger specialization and more controllable activated parameters. The cost is routing overhead, load imbalance, and greater sensitivity in expert-parallel communication. More experts increase the scale of all-to-all communication; finer top-k complicates routing matrices and token reordering. During planning, focus on expert-parallel degree, the number of experts that fit on a single card, communication domain size, and whether the hottest expert creates a long tail. Shared experts can mitigate degradation of general capability, but they also add resident weights.

    6. Residual Mixing and Numeric Formats

    Residual connections appear simple, but in deep large models, the numerical stability of residual paths, multi-path residual mixing, and normalization placement affect training convergence and inference accuracy. Multi-path residual mixing can improve information flow, but it increases intermediate tensors and the difficulty of operator fusion. For numeric formats, low-precision schemes such as FP16, BF16, FP8, and INT8 can reduce weight volume and improve compute utilization, but attention must be paid to accumulation precision, overflow risk, and quantization calibration.

    The choice of numeric format in turn affects hardware planning. A high low-precision compute peak does not mean high actual throughput. If attention or routing still requires higher precision, mixed precision introduces type conversion and bandwidth overhead. Numeric format should therefore be reviewed together with operator-level decomposition, not by looking at peak metrics alone.

    7. Operator-Level Accounting: From Formulas to Capacity Planning

    For each candidate architecture, it is advisable to build an operator-level breakdown table that lists operators, formulas, and tensor dimensions segment by segment along the attention path and the feed-forward/expert path. For example, with input [B,S,H], the QKV projection yields [B,S,H_q] and [B,S,H_kv]; attention scores are [B,heads,S,S] (or a sparse index form); the feed-forward path goes from [B,S,H] to [B,S,I] and back to [B,S,H]; expert routing goes from [B*S,H] to top-k experts. The following formulas summarize cache, compute, and weight-volume accounting:

    1. weights_bytes = total_params * bytes_per_param
    2. activated_flops ≈ 2 * active_params * tokens
    3. kv_bytes = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem
    4. expert_comm ≈ top_k * tokens * hidden * bytes_per_elem * parallel_degree # illustrative

    When planning general-purpose accelerator resources on a cloud platform, you can break the accounting into four budgets: memory capacity, memory bandwidth, interconnect bandwidth, and low-precision compute. Then leave headroom for long-context caching, expert-communication peaks, and batching fragmentation. This is closer to real bottlenecks than looking only at total parameters or peak compute.

    8. Trend Judgment and Planning Headroom

    Taken together, the structural trends of next-generation models can be summarized as follows: total parameters continue to rise, but activated parameters remain restrained; context continues to expand, but KV cache is controlled through compression and tiered caching; expert counts continue to increase, but routing becomes finer and shared experts become more common; layer order becomes more hybrid, and residual and numeric formats become more refined. For accelerator planning, priority should go to sufficient memory capacity and bandwidth, followed by the interconnect topology for expert parallelism, and finally peak compute. Only by translating structural trade-offs into an operator-level ledger can architecture evolution be given verifiable headroom.