Modern LLM Architecture and Parameter Scaling: A Technical Deep Dive

    This article breaks down the internal architecture of modern large language models (LLMs), explaining how decoder-only backbones, compressed attention mechanisms, and mixture-of-experts (MoE) layers work together. It focuses on the trade-offs between model size, activation parameters, memory bandwidth, and compute costs, and provides practical formulas for estimating KV cache usage and inference overhead—helpful for engineers planning deployment and capacity.

    Introduction: Why Architecture Matters

    Recent LLM iterations may appear diverse, but comparing public technical specifications reveals a counterintuitive trend: model changes are scattered, while hardware constraints converge. Regardless of the vendor or model family, architectural evolution consistently gravitates toward the same physical bottlenecks: compute, memory bandwidth, and inter-chip communication. Recognizing this is essential to understanding architecture rather than merely reading parameter tables.

    Three typical phenomena help frame the discussion:

    • Total parameters surge, while activated parameters stay flat. Total model parameters have climbed from hundreds of billions to trillions, but the parameters actively involved in a single forward pass remain in the tens of billions. This is not coincidental—it is the result of deliberate design under the mixture-of-experts (MoE) paradigm, using sparse activation to achieve “large capacity at low inference cost.”
    • Context windows grow, while caches are compressed. While context support expands from 128K to 1M tokens, KV cache size is repeatedly squeezed. Various low-rank and compression-based attention modifications are fundamentally battles against memory bandwidth.
    • More experts, finer routing. Total expert counts have increased from a few hundred to nearly a thousand, but the number of active experts per token has not scaled proportionally. Routing granularity has become finer, and designs like shared experts have been introduced.

    These three seemingly contradictory trade-offs form the central thread for understanding modern LLM architecture.

    The Backbone: Decoder-Only Framework and Layer Order

    Virtually all modern LLMs are built on a decoder-only Transformer skeleton. The differences primarily emerge in layer organization:

    1. Dense opening + sparse main body. A common practice is to keep the first few layers as dense feed-forward networks to stabilize low-level representation learning, then switch to MoE layers for the remaining stack. For example, one typical configuration uses 61 layers: 3 dense layers followed by 58 expert layers.
    2. Layer-wise heterogeneous attention. Advanced designs use different attention types per layer. A few layers handle global long-range dependencies, while the rest use sliding window or compressed attention—maintaining long-context capability while controlling compute and cache overhead.
    3. Depth vs. width allocation. Larger parameter counts do not necessarily mean more layers. What matters is the ratio of hidden dimensions, attention heads, and feed-forward intermediate dimensions. A useful rule of thumb: total parameters ≈ per-layer parameters × number of layers + embedding table. In ultra-large vocabulary scenarios, the embedding table’s share becomes non-negligible.

    Attention Mechanisms: From Full Attention to Compression

    Attention is the primary battleground for architectural differences. The core tension: attention compute scales quadratically with sequence length, KV cache grows linearly, and memory bandwidth lags far behind.

    Low-Rank Compressed Attention

    The representative approach applies low-rank decomposition to Key/Value projections. Hidden states are first compressed into a latent vector far smaller than the original dimension (e.g., from thousands down to 512), cached at inference time, and projected back up only when needed. This cuts KV cache volume by an order of magnitude, at the cost of a small amount of extra projection compute. Query-side decomposition is often applied similarly to balance expressiveness.

    Sparse and Dynamically Selected Attention

    Another direction is “select, don’t compress”: for each query token, dynamically pick the most relevant historical tokens for attention—e.g., using a lightweight indexer to choose the top-2048 positions out of hundreds of thousands. This approach is critical for ultra-long contexts (1M+ tokens), where full attention is infeasible in both compute and cache.

    Hybrid Attention Layout

    In practice, mixed attention is common: a few layers retain strong global attention, while the majority use compressed or sliding-window attention. The hardware implication is that different layers have distinct bandwidth and compute demand profiles, so inference frameworks must support layer-aware scheduling strategies.

    Mixture of Experts (MoE): Separating Capacity from Cost

    MoE is currently the dominant method for scaling parameters. Its design principles can be summarized in four points:

    • Fine-grained expert splitting. Instead of a few large experts, models are split into hundreds of smaller experts (intermediate dimension around 2048–3072), with 6–8 selected per token. This yields a much larger combinatorial space than coarse-grained designs, improving capacity utilization.
    • Shared experts. A shared expert that remains always active is added alongside routed experts to capture general knowledge, reducing redundancy among routed experts.
    • Routing stability. Load-balancing losses and auxiliary bias adjustments are used to prevent “starved experts”—one of the most challenging parts of training engineering.
    • Constant activated parameters. With top-k selection: activated parameters ≈ total parameters × (k/N) + dense portion. For a model with 671B total parameters and ~37B activated per token, inference compute cost is comparable to a 37B dense model, but knowledge capacity approaches that of a dense 671B model—this is the mathematical foundation of “total parameters rise, activated parameters don’t.”

    MoE’s direct hardware constraint is communication: when experts are distributed across devices, each token’s hidden state must be exchanged via all-to-all communication, and interconnect bandwidth often becomes the bottleneck before compute does.

    Numeric Formats and Feed-Forward Design

    Feed-forward layers commonly adopt gated linear units (e.g., SwiGLU structures)—using three matrices: two up-projections (one gated, one value), element-wise multiplication, followed by a down-projection. The intermediate dimension is typically several times the hidden dimension, prorated per expert under MoE.

    On numeric formats, FP8 mixed precision has become a practical option for large-scale training, using block-wise scaling to manage dynamic range. INT8/FP8 weight quantization with dynamic activation quantization is standard for inference. The implication for hardware planning: peak compute metrics must be qualified by precision. “TFLOPS@FP16” and “TFLOPS@FP8” are entirely different numbers for the same chip.

    Residual pathways have also seen multi-branch attempts: splitting the residual stream into multiple channels with weighted fusion to improve deep-layer gradient propagation. These designs significantly increase per-layer memory traffic and require targeted optimization at the fused-operator level.

    Estimating KV Cache and Inference Cost

    Here is a general estimation framework for auditing any configuration:

    1. KV cache per token = layers × KV heads × dim per head × 2 (K and V) × bytes
    2. Attention compute per token ≈ layers × seq_len × heads × dim × constant
    3. Weight memory = total params × bytes per param
    4. Activation compute per token ≈ 2 × activated params (FLOPs)

    Example: compressing KV to a latent vector. If cached objects go from “64 heads × 128 dim” to a “512-dim latent vector,” per-token cache drops from 16,384 elements to 512—a compression ratio of about 32×. This is why long-context serving is feasible at all. When selecting an inference system, prioritize estimating memory usage at the target context length using the formulas above, then work backward to determine GPU count and concurrency limits.

    Summary of Hardware and Infrastructure Constraints

    Translating architectural features into infrastructure requirements yields four planning guidelines:

    1. Compute and bandwidth are equally important. MoE layers have low arithmetic intensity, making interconnect and memory access the bottleneck; attention layers become compute-dense at long sequence lengths. Procurement and scheduling must assess both metrics.
    2. KV cache determines memory capacity. For long-context serving, per-GPU concurrency ≈ available memory / (per-token cache × context length). Compressed attention directly increases this value.
    3. Interconnect topology sets the MoE ceiling. Expert parallelism is bounded by all-to-all bandwidth; high-speed intra-node interconnect provides greater gains than inter-node links.
    4. Precision flexibility. Training and inference pipelines should reserve the ability to switch between multiple precision tiers to match evolving numeric formats.

    For teams looking to deploy LLM inference and training in production, Baidu AI Cloud offers high-performance computing instances, distributed training frameworks, object storage, and monitoring and alerting services, enabling rapid infrastructure setup that meets the constraints outlined above.

    Returning to the three contradictions introduced at the start—the split between total and activated parameters, the gap between context length and cache, and the inverse movement of expert count and routing granularity—one trend is clear: the next generation of competition is no longer about “more parameters,” but about “more knowledge per unit of activated compute and longer context per unit of memory.” With the estimation methods and architectural framework presented in this article, you can quickly parse any new model’s spec sheet to understand its real costs and hardware requirements, leaving ample margin for technical selection and capacity planning.