Efficient LLM Architecture: A Deep Dive into Three Key Technologies

    This article examines three architectural techniques that are increasingly discussed in large language model design: lower-bound decay parameterization for linear attention, auxiliary-loss-free quantile load balancing for mixture-of-experts (MoE) routing, and block-level attention residuals (Block AttnRes) for deep networks. For each technique, it explains the problem being solved, how the mechanism works, what the costs and trade-offs are, and how to evaluate whether to reuse it. It then covers how the three techniques reinforce one another along the sequence, width, and depth dimensions, offers practical guidance for moving from prototype validation to production deployment, and lists common misconceptions to watch out for.

    1. Introduction: Efficiency innovation is the main thread of LLM architecture evolution

    When a new model is released, people tend to look first at parameter count and benchmark scores, while easily overlooking the architecture efficiency that is equally critical. What actually determines whether a model can be deployed in real engineering is often how many effective tokens each card can process per second, how long a context it can support under a given amount of memory, whether training can run stably over long periods, and whether inference cost can fall as the business grows.

    Looking at recently published large model architecture designs, efficiency innovation has been unfolding along three dimensions:

    • Sequence dimension: the tension between the quadratic complexity of attention computation and the demand for long context;
    • Width dimension: the routing skew and communication bottlenecks introduced by increasing the number of experts in ultra-large-scale mixture-of-experts (MoE);
    • Depth dimension: gradient propagation and representation homogenization in deep networks, along with the memory and communication overhead from inter-layer activations.

    In public technical discussions, three designs have been mentioned repeatedly and are considered to have high cross-project reuse value:

    1. Lower-bound decay parameterization for linear attention: constructing a forget gate in a numerically bounded way, which eliminates special computation branches;
    2. Auxiliary-loss-free MoE quantile balancing strategy: dynamically adjusting routing based on quantile statistics, without relying on auxiliary loss;
    3. Block-level attention residual (Block AttnRes): introducing block-level summarization and attention to open up gradient and representation pathways in deep networks.

    This article does not evaluate any specific model. Instead, it extracts these three techniques and analyzes them across four levels: what problem they solve, how the mechanism works, where the costs lie, and how to evaluate reuse.

    2. Technique One: Lower-bound decay parameterization for linear attention

    2.1 Root of the problem: a reciprocal that creates a branch

    In softmax attention, every query interacts with all historical keys, so computation and memory overhead rise sharply for long sequences. Linear attention replaces explicit access to the full history with a fixed-size state matrix. Its recurrence can be written as:

    1. S_t = Diag(α_t) · S_{t-1} + k_t · v_tᵀ
    2. o_t = q_tᵀ · S_t

    where α_t ∈ (0, 1) is the step-wise decay (forget) factor. With the common parameterization g_t = −softplus(f(x_t)) and α_t = exp(g_t), g_t can take any negative value. This is especially noticeable early in training, when the forget gate is small and g_t may reach −20 or even lower.

    Engineering implementations typically unroll the recurrence block by block (chunk by chunk). The intra-block part can be written in matrix form:

    1. O = (Q · b) · tril[(K / b) · Vᵀ] + (Q · b) · S_prev
    2. b_i = exp(Σ_{j≤i} g_j)

    The key term is 1/b. The weight between any pair of positions (i, j) within a block is bi / b_j, i.e., exp(−Σ{j<l≤i} g_l). Once the decay is too strong, this reciprocal overflows to inf in low-precision formats such as BF16, or underflows to 0 and then triggers division by zero. Taking BF16 as an example, its representable range is about 1.2×10⁻³⁸ to 3.4×10³⁸, corresponding to an exponential safe interval of about [−87, 88].

    As a result, actual implementations can only design a separate path for this (the “position-pair path”). For each (i, j), the interval decay is accumulated directly and then multiplied by the gate in an add-multiply fashion, all element-wise, so matrix-multiply instructions cannot be used. This path is usually the slowest part of the entire linear attention operator.

    2.2 Mechanism: locking the decay into a finite interval

    The approach of lower-bound decay parameterization is straightforward: change the output of the decay exponent from “unbounded below” to “bounded below.”

    1. g_t = g_min · σ(wᵀ h_t) g_min = −5
    • σ is the sigmoid function, h_t is the current hidden state, and w is a learnable parameter.
    • g_min is a fixed hyperparameter that does not change during training.
    • The output always satisfies g_min ≤ g_t < 0: at the fastest decay, α is about 0.0067, and at the slowest it approaches 1 (approximately no forgetting).

    Once the lower bound is fixed, the upper bound of the reciprocal scaling between any L consecutive positions becomes exp(−L · g_min). Taking a 16-token block as an example, exp(80) ≈ 5.5×10³⁴, still within BF16’s representable range. The computation between blocks only involves the forward cumulative decay b, which always satisfies 0 < b < 1, so no overflow occurs.

    Numerical form Unbounded (e.g., negative-softplus) Lower-bound parameterization (g_min = −5)
    Decay exponent g_t (−∞, 0), can be arbitrarily small [−5, 0)
    Intra-block reciprocal 1/b (L = 16) No upper bound; overflows/underflows in low precision ≤ exp(80) ≈ 5.5×10³⁴
    Inter-block scaling b Always in (0, 1) Always in (0, 1)
    Intra-block computation path Position-pair path Shares dense matrix multiply with ordinary blocks

    The engineering benefits are mainly twofold:

    1. Eliminating special branches. All blocks (including diagonal blocks) go through dense matrix multiply (GEMM) plus a causal mask uniformly, which naturally fits the chunking and warp-level matrix multiply (MMA) primitives commonly used in attention. The operator logic is more regular, and both hardware utilization and cross-platform portability on mainstream hardware such as GPUs are better; the specific improvement depends on the hardware and implementation.
    2. More controllable numerical semantics. The decay behavior is analyzable and predictable, making error analysis under mixed precision easier. Because position information is implicitly carried by decay accumulation, this kind of structure usually needs no positional encoding to handle long sequences and is also easier to extrapolate to longer lengths.

    2.3 Boundaries and costs

    A lower bound means there is an upper limit on the “forgetting ability.” At the fastest decay level, it takes about 150 tokens for the weight of historical information to decay to 1%. If the forget gates of all layers hit the lower bound, the network loses the channel for retaining distant information. Therefore:

    • Multiple time scales must form: some layers or branches remain approximately identity (slow decay) to carry long-range memory, while the rest handle fine-grained local modeling;
    • Targeted evaluation is needed: long-context retrieval tasks (such as locating a specific fact within a long document) are extremely sensitive to “whether it can remember” and should be a mandatory test for this modification;
    • Multi-layer decoupling is feasible: if different layers are allowed to use different g_min values, shallow layers can focus on the local and deep layers on the global, but this introduces extra hyperparameters that require more ablation experiments.

    2.4 Illustrative implementation

    1. # Illustrative pseudocode: block-level linear attention (normalization and output transform omitted)
    2. G_MIN = -5.0 # Fixed decay lower bound
    3. def gated_linear_attention(q, k, v, h, w, tile=16):
    4. # q, k: [B, T, Dk]; v: [B, T, Dv]; h: [B, T, H]
    5. g = G_MIN * torch.sigmoid(h @ w) # [B, T, 1], always -5 <= g < 0
    6. state = q.new_zeros(B, Dk, Dv) # Cross-block state
    7. outputs = []
    8. for s in range(0, T, tile):
    9. qs = q[:, s:s + tile]
    10. ks = k[:, s:s + tile]
    11. vs = v[:, s:s + tile]
    12. gs = g[:, s:s + tile]
    13. # Compute cumulative decay in the log domain, in FP32, to avoid BF16 chained-multiply error
    14. log_b = torch.cumsum(gs.float(), dim=1) # [B, tile]
    15. b = log_b.exp().to(q.dtype) # (0, 1)
    16. inv_b = (-log_b).exp().to(q.dtype) # <= exp(-tile * G_MIN)
    17. # Diagonal and off-diagonal blocks use the same dense matrix multiply
    18. qt = qs * b
    19. kt = ks * inv_b
    20. attn = qt @ kt.transpose(-1, -2) # [B, tile, tile]
    21. mask = torch.ones(tile, tile, device=q.device).tril()
    22. attn = attn * mask # Causal mask
    23. # Interaction with the previous block's state
    24. o_intra = attn @ vs
    25. o_cross = qt @ state
    26. outputs.append(o_intra + o_cross)
    27. # State update (illustrative: actual implementation should maintain FP32 or renormalize periodically)
    28. kd = ks.transpose(-1, -2) * b.unsqueeze(-2)
    29. state = state * b[:, -1, None, None] + kd @ vs
    30. return torch.cat(outputs, dim=1)

    3. Technique Two: Auxiliary-loss-free MoE quantile load balancing

    3.1 Load balancing: the hidden cost of ultra-large-scale MoE

    MoE uses sparse activation to scale parameter capacity under approximately constant per-token computation. But expert parallelism (EP) requires that expert load be as balanced as possible across devices. If tokens concentrate on a few “hot experts,” a chain of reactions follows:

    • Devices hosting hot experts queue up their computation while idle devices spin idle; All-to-All communication is dragged down by the slowest device, and overall throughput drops;
    • The capacity factor must be increased to accommodate overflow, adding redundant computation;
    • If token dropping is enabled, tokens that miss capacity are skipped directly, losing effective training signal.

    The traditional approach is to introduce an auxiliary load-balancing loss that constrains the routing distribution with the goal of minimizing total cost. But it has two inherent problems:

    • Conflict with the main task objective: the auxiliary loss directly applies gradients to the routing parameters; a large coefficient harms language modeling quality, while a small one gives insufficient balancing, requiring repeated tuning;
    • Harder to balance as scale grows: when the number of experts reaches hundreds or even thousands, the variance of each expert’s probability estimate increases, and overloading of just a few experts is enough to slow down the entire EP group.

    3.2 Mechanism: quantile balancing

    The core idea of quantile balancing is to turn balancing from a “training objective” into “post-routing processing”: instead of relying on auxiliary loss, it dynamically adjusts based on quantile statistics of the routing output. The typical flow is as follows:

    1. Statistics: maintain the recent sum of times each expert is actually selected (or its routing probability), usually via a sliding window or exponential moving average;
    2. Compute quantiles: take quantiles of the expert load vector (such as the median or a high quantile) as a reference for a “reasonable load level”;
    3. Feedback correction: multiply the difference between the reference and each expert’s actual load by a scaling coefficient, then add it as a bias to the routing scores; overloaded experts are down-weighted and underloaded ones up-weighted, and Top-k selection is redone;
    4. Cooperate with capacity management: work together with the capacity factor and fallback paths for overflow tokens (such as falling back to shared experts).

    Quantiles are more robust than the mean and insensitive to a few extreme hot experts. Anchoring on quantiles can avoid the large-scale overflow caused by a “one-size-fits-all” capacity limit.

    3.3 Benefits, costs, and engineering essentials

    Benefits:

    • Routing gradients come only from the main task loss, free of auxiliary-loss interference, which eliminates coefficient tuning;
    • The load distribution is more stable, reducing capacity redundancy and token dropping, and lowering the variance of EP communication latency;
    • Statistics and bias adjustments are all incremental operations on the existing routing scores and require no changes to the forward-computation backbone.

    Costs:

    • Statistics must be synchronized across devices (periodic All-Reduce or asynchronous aggregation can be used to control overhead);
    • The choice of quantile and scaling coefficient (quantile level, window length, scaling strength, target bias) affects convergence speed and quality and needs to be tuned by model scale;
    • In early training the routing itself is unstable, so the balancing feedback needs a smooth start to avoid oscillation;
    • It acts only during training and has no effect on inference; but it also means inference cannot “compensate” for uneven expert quality left over from training.

    Diagram:

    1. Routing scores ──► Top-k selection ──► Count recent load per expert
    2. ▲ │
    3. │ ▼
    4. Bias adjustment ◄── Quantile reference ◄── Load vector

    3.4 Illustrative implementation

    1. # Illustrative pseudocode: quantile-based, auxiliary-loss-free load balancing
    2. ALPHA = 0.8 # Quantile level
    3. BETA = 1e-2 # Scaling strength
    4. MOMENTUM = 0.99 # Moving-average coefficient for the statistics
    5. def balanced_topk(logits, k, load_ema):
    6. # logits: [N_tokens, E], load_ema: [E]; both are global statistics across all EP ranks
    7. ref = torch.quantile(load_ema, ALPHA) # Quantile reference value
    8. bias = BETA * (ref - load_ema) # Down-weight overloaded, up-weight underloaded
    9. scores = logits + bias # Affects routing only; does not change expert parameters
    10. topk_idx = scores.topk(k, dim=-1).indices
    11. # Update global load statistics (actual implementation needs periodic cross-device sync)
    12. load_ema = MOMENTUM * load_ema + (1 - MOMENTUM) * count_tokens(topk_idx, E)
    13. return topk_idx, load_ema

    4. Technique Three: Block-level attention residual (Block AttnRes)

    4.1 Two bottlenecks of deep networks

    Representation bottleneck: in a deep Transformer, the inter-layer residual is an identity pass-through, so the input to layer L is determined entirely by the output of layer L−1. Deep layers mainly do local refinement, and inter-layer outputs tend to become homogeneous, i.e., effective rank decreases. Prior research has analyzed similar phenomena from perspectives such as “representation collapse,” whose direct manifestation is diminishing returns from deep parameters, while the memory and communication overhead brought by layers grows linearly.

    Engineering bottleneck: every layer must retain forward activations for backpropagation. Activation memory is proportional to the number of layers; in pipeline parallelism, the amount of inter-layer activation transfer is directly related to the number of layers.

    4.2 Mechanism: summarization, block-level attention, and residual bridging

    Block-level attention residual draws on the ideas of “token merging/pooling” in vision Transformers and “cross-layer access” in general Transformers. It can be understood in three steps:

    1. Chunking and summarization: divide the sequence (or layer stack) into several blocks, perform learnable pooling within each block (such as position-weighted mean), and obtain one compact summary vector per block;
    2. Block-level attention: perform a single attention over all summary vectors so that each block can directly “see” the global information of other blocks and even earlier-layer blocks, then map it back to the token dimension via projection;
    3. Residual bridging: add the block-level attention output to the original token representation, forming a “cross-layer highway” that bypasses the intermediate layers.

    Diagram:

    1. Block 0 ──► Summary ──┐
    2. Block 1 ──► Summary ──┼──► Block-level attention ──► Fill back to blocks ──► Deep layers
    3. ⋮ │
    4. Block N ──► Summary ──┘

    4.3 Benefits and costs

    Benefits:

    • Shorter gradient path: deep layers can directly obtain information from early blocks, alleviating gradient dilution and representation homogenization and improving the utilization of deep parameters;
    • Significantly reduced overhead: the number of summary vectors is only the number of blocks, with length O(L/B). The computation and memory complexity of block-level attention is about O((L/B)²), roughly B² times lower than the O(L²) of token-level global attention (B is the block size), and inter-block communication is reduced by the same ratio;
    • Inference-friendly: block summaries can serve as a compressed memory cache for cross-layer information retrieval during long-context inference.

    Costs:

    • Intra-block information is still limited by single-layer modeling ability: cross-block relies on attention, while intra-block relies on the modeling ability of the layer’s attention or linear attention;
    • Summaries and block boundaries are hyperparameters: too large a block causes intra-block information loss, while too small increases summary overhead;
    • Implementation constraints: the pooling order and positional information of summaries must stay consistent, and the causality design of attention must be handled carefully; block-level attention, as a new structural layer, needs support from model-parallel partitioning and inference engines;
    • Inference caching: summaries and key-value caches must be managed together, adding complexity to inference-side memory planning.

    4.4 Illustrative implementation

    1. # Illustrative pseudocode: block-level attention residual bridging
    2. def block_att_res(tokens, blocks, proj, block_attn):
    3. # tokens: [B, L, D]; blocks: [B, n_block], token-to-block assignment
    4. # proj: summary projection matrix [B, D, n_block]
    5. summaries = einsum("bld,dbn->bnd", tokens, proj) # Block-level summaries [B, n, D]
    6. summaries = block_attn(summaries) # Attention between blocks
    7. # Broadcast block-level results back to tokens: gather by blocks
    8. broadcast = torch.gather(summaries, 1, blocks.unsqueeze(-1).expand(-1, -1, D))
    9. return tokens + broadcast # Cross-layer residual bridging

    5. Synergy of the three techniques: unified optimization of length, width, and depth

    Viewed individually, each of the three techniques solves a problem in one dimension; combined, they jointly form an efficient training framework for “long sequence × ultra-large scale × deep network”:

    • Linear attention (long): trading bounded decay for unified block-level matrix multiply, freeing long-sequence training from the per-position path;
    • Quantile balancing (wide): replacing auxiliary loss with statistical feedback, keeping ultra-large-scale MoE balanced under EP;
    • Block residual (deep): bridging deep layers with summary attention, opening up the gradient path while controlling activation overhead.

    From the engineering-constraint perspective, the three techniques clearly complement and reinforce each other:

    • Both linear attention and block residual use “blocks” as the basic scheduling unit, which facilitates unified block pipelining, operator fusion, and activation-recomputation strategies;
    • Block summaries can serve as additional context for routing (for example, concatenating block summaries into the routing-score input) and work together with quantile balancing;
    • The load statistics produced by quantile balancing during distributed training can also guide the scheduling of block-level tasks.

    When evaluating whether to adopt them, it is recommended to judge uniformly by a “four-question checklist”:

    1. Can the benefit be quantified: measure it with business metrics (throughput, memory, long-context task accuracy, training stability) rather than benchmarks alone;
    2. Is the cost controllable: the amount of change, the hardware characteristics relied upon, and the tuning complexity;
    3. Can it be rolled back: keep the original path as a switch for easy comparison and rollback;
    4. Long-stability validation: observe numerical stability, routing balance, and communication efficiency at the thousands-of-cards level.

    6. Implementation recommendations: from prototype validation to production deployment

    1. Do small-scale validation first: run controlled experiments below tens of billions of parameters and at a scale of billions of tokens, focusing on comparing the perplexity curves and long-context retrieval metrics “with and without”;
    2. Unify the operator abstraction: linear attention needs to share block scheduling with block residual and MoE routing; it is recommended to abstract attention uniformly into a “block interface” to reduce special branches;
    3. Pay attention to numerical precision: compute cumulative decay in the FP32 log domain; use low precision only for the main body of the matrix multiply; periodically renormalize the state;
    4. Put observability first: add metrics such as routing load distribution (Gini coefficient / entropy), attention numerical range, and block-level attention distribution to training monitoring;
    5. Evaluate the inference side synchronously: block-summary caching strategy, linear-attention state management, and MoE expert caching and routing strategy all affect inference cost and should be considered together at the model design stage.

    If cloud training is used, you can leverage the general capabilities offered by cloud platforms such as Baidu AI Cloud to reduce engineering complexity, for example:

    • Use a container platform to orchestrate distributed training and inference tasks uniformly;
    • Use object storage to persist training data and checkpoints;
    • Use monitoring and alerting services to track key metrics such as sudden loss changes, routing skew, and communication bottlenecks.

    7. Common misconceptions and risk warnings

    Misconception Analysis Recommendation
    A fixed lower bound means “infinite extrapolation” The lower bound ensures numerical stability, but long-range memory depends on slow-decay channels and inter-layer stacking; extrapolation ability needs dedicated evaluation Always test long-context retrieval tasks
    Linear attention is “naturally fast” Training benefits from dense matrix multiply, but inference decoding is constrained by state updates and memory bandwidth Benchmark training and inference separately
    Quantile balancing is “set and forget” It is only post-routing processing and still needs capacity management, statistic synchronization, and parameter tuning Establish load monitoring and rollback mechanisms
    Block residual “definitely saves memory” Block size and summary granularity determine the actual overhead, and inference-engine support is required Run ablation experiments across block sizes
    Only architecture modification, no engineering support Missing operator implementation, numerical precision, monitoring, and rollback mechanisms means architecture benefits are hard to realize Advance architecture and engineering together

    8. Conclusion

    These three techniques correspond to the three dimensions of length, width, and depth: lower-bound decay parameterization makes linear attention numerically “controllable,” quantile balancing makes ultra-large-scale MoE “balanced” in load, and block-level attention residual “connects” deep networks in gradient and representation. For developers, the real reuse value lies not in copying any single implementation, but in understanding the constraints behind its design ideas — numerical boundaries, load balancing, and gradient pathways — and combining them with the scale, hardware, and evaluation system of their own business to make technical choices that are quantifiable, rollback-able, and long-stable.