1. Background: Three Bottlenecks on the Path to Scale

Context windows for large language models are moving from tens of thousands of tokens toward million-token ranges. At the same time, MoE models are expanding from dozens to hundreds or even thousands of experts, and transformer depth continues to grow. Along this scaling path, three problems appear repeatedly:

1. Numerical and hardware-efficiency bottlenecks in linear attention operators
   Linear attention and incremental update mechanisms such as delta-rule variants use fixed-size state matrices to achieve O(N) complexity. However, their block-parallel implementations rely on cumulative decay factors Γ. The reciprocal of Γ easily underflows in limited-precision formats such as BF16, forcing the diagonal blocks to fall back to position-by-position scalar computation and losing the benefit of hardware matrix multiplication.

2. The MoE load-balancing dilemma
   Sparse routing often collapses to a small set of popular experts. A common remedy is to add an auxiliary balancing term to the loss function, but this couples the balancing objective with the main-task gradient and can lower the ceiling of model quality.

3. Representation and overhead problems in deep stacks
   As layer count increases, representation capacity tends to saturate. Meanwhile, per-layer residuals cause activation memory and cross-device communication costs to grow linearly with depth.

Recent public technical discussions have highlighted three designs that deserve closer study: lower-bound parameterization of decay factors, quantile-based auxiliary-loss-free dynamic load balancing, and block-wise attention residuals. The following sections examine each in turn.

2. Technique 1: Lower-Bound Parameterization of Decay Factors

2.1 Problem: The Numerical Cliff of Reciprocal Scaling Factors

Block-parallel execution is essential for achieving high throughput when training linear attention models. The sequence is partitioned into tiles. Across tiles, states are accumulated in parallel along the time dimension. Within a tile, local interactions are modeled through diagonal blocks.

The problem lies precisely in the diagonal blocks: they require the reciprocal of the intra-tile cumulative decay Γ as a scaling factor.

Traditional implementations parameterize log-decay as unbounded negative-softplus:

# Traditional parameterization: unbounded
g_t = -softplus(alpha * h_t)   # range: (-inf, 0)

The tail of softplus diverges linearly. When the input response is large, the per-step decay can approach zero arbitrarily closely. Under BF16, after accumulating log-decay over multiple steps and taking the exponential, the reciprocal scaling factor either underflows to zero, erasing local information, or the compensation scaling itself overflows. As a result, diagonal blocks cannot safely use the matrix-multiplication path and must explicitly expand position-wise computation, falling back to a low-throughput slow branch.

2.2 Mechanism: From Unbounded to Lower-Bounded

The fix is straightforward: enforce a hard lower bound on log-decay by replacing the unbounded negative-softplus with a scaled sigmoid parameterization. The following is schematic; internal linear transformation details are omitted:

# Lower-bound parameterization: bounded
g_min = -5.0
g_t = g_min * sigmoid(alpha * h_t)   # range: [g_min, 0)

Sigmoid saturates naturally and its output is confined to (0, 1). After multiplying by the fixed lower bound g_min = -5, the per-step log-decay falls within [-5, 0), which corresponds to decay factors in [e^-5, 1). In other words, the decay cannot become arbitrarily fast, eliminating uncontrolled cumulative-scale explosion at the source. For a typical 16-token tile, the reciprocal scaling factor remains within a safe numerical range.

2.3 Benefits: One Numerical Path for All Tiles

• Eliminates special branches
  Diagonal and off-diagonal blocks can uniformly use dense matrix multiplication. The underlying GPU operator implementation no longer needs a separate position-wise path for diagonal blocks, which significantly improves hardware accelerator utilization and throughput.

• Numerically robust
  The approach no longer depends on complicated log-space compensation tricks, making extremely long-sequence training more stable.

• Minimal engineering change
  Only one parameterization function is replaced. The training-side cost is nearly zero, and inference only requires updating the decay computation logic in the corresponding kernel.

• Naturally compatible with NoPE
  For million-token contexts, it can be extrapolated directly without position-encoding-related adaptation overhead.

2.4 Reuse Recommendations

For any team implementing custom linear attention or incremental attention operators, this is a classic “one change, global benefit” modification. A more general lesson is: before optimizing kernels, audit the boundaries of the numerical path first. Many “slow operator” issues are not caused by insufficient parallelism, but by numerical ranges that force the implementation onto a slow path.

3. Technique 2: Auxiliary-Loss-Free Dynamic Load Balancing for MoE

3.1 Problem: The Cost of Auxiliary Losses

MoE models use top-k routing to dispatch tokens to experts. However, early in training, routing scores are close to each other and exploration is limited, so routing easily collapses to a few popular experts while the remaining experts receive almost no gradient.

The mainstream solution is to add an auxiliary balancing loss to the training objective, such as a uniformity penalty on expert load. But this introduces three problems:

1. Gradient coupling
   The balancing term competes with the main-task objective. Experts are forced to balance for the sake of balancing, which can hurt specialization.

2. Hyperparameter sensitivity
   The balancing coefficient must be repeatedly adjusted across training stages and expert scales.

3. Scaling difficulty
   When the number of experts grows to hundreds or thousands, a static coefficient can no longer cover all routing states.

3.2 Mechanism: Internalizing Balance into Routing Decisions

Based on its naming and public descriptions, quantile balancing moves balance from a loss constraint into the routing decision itself: instead of selecting experts purely by absolute scores, it uses quantiles of the routing-score distribution as dynamic gating thresholds.

The intuition is as follows: within each batch, compute quantile statistics over all routing scores and use the q-th quantile as the admission threshold. Only tokens whose scores exceed the threshold are admitted to the corresponding expert. When an expert remains persistently overloaded, its relative ranking threshold rises, causing overflow tokens to go to the next-best experts. Conversely, the threshold for underused experts drops. Load state is maintained through low-overhead online statistics, making balance an intrinsic property of routing rather than an external penalty attached to the loss function.

3.3 Benefits

• Cleaner main objective
  Removing the auxiliary loss prevents expert specialization from being suppressed by the balancing term, leading to a higher model-quality ceiling.

• Adaptive and tuning-free
  Thresholds adjust in real time based on batch statistics, eliminating manual balancing-coefficient sweeps.

• Friendly to very large models
  The cost of statistics and threshold adjustment is far lower than multi-round routing retries or global rebalancing schemes, supporting scenarios with very large expert counts.

• Stable training
  It suppresses both early routing collapse and late-stage balancing oscillation.

3.4 Reuse Recommendations

This approach is best suited to training scenarios with a large number of experts and a strong emphasis on MoE model quality. Key implementation considerations include the statistics window, such as per-step, per-microbatch, or EMA smoothing, and a warmup strategy during early training to prevent threshold jitter from affecting convergence.

4. Technique 3: Block-Wise Attention Residuals

4.1 Problem: The Triple Cost of Per-Layer Residuals

In a standard transformer, each attention output is injected into the main path through an independent residual connection. When depth reaches several dozen layers or more, this structure incurs three costs:

1. Representation cost
   Too many identity branches create long signal-return paths for deep layers, and representation capacity improves with depth at a diminishing rate.

2. Memory cost
   Attention activations for every layer must be retained for backpropagation, so memory usage grows linearly with depth.

3. Communication cost
   Under sequence parallelism and related strategies, residual-related quantities participate in cross-device communication. More layers mean more synchronization events.

4.2 Mechanism: Organizing Residual Paths at Block Granularity

Block-wise attention residuals can be summarized as upgrading per-layer residuals to block-level aggregation. Several adjacent layers are grouped into a block, and the attention contributions within the block are merged into the main path at a coarser granularity. In effect, this structurally reorganizes the residual path along the depth dimension.

As a result:

• Deep-layer signals flow back through shorter paths, alleviating the depth representation bottleneck.
• Intermediate activations inside a block no longer need to be fully saved layer by layer, reducing peak activation memory.
• Communication is merged from per-layer granularity to block granularity, reducing the number of cross-device synchronization events.

This technique is orthogonal to general memory-saving methods such as gradient checkpointing and activation recomputation, so it can be combined with them.

4.3 Reuse Recommendations

This is suitable for architectures targeting extremely deep stacks and long contexts. If a team observes that validation loss no longer improves after adding depth, or if memory and communication become the scaling bottleneck, this type of structural change should enter the candidate list.

5. Side-by-Side Comparison and Adoption Priorities

Technique	Change Cost	Core Benefit	Applicable Scenario
Lower-bound decay factor parameterization	Minimal (parameterization-level)	Operator throughput, numerical stability	Linear attention / incremental attention implementations
Quantile-based dynamic load balancing	Medium (routing module rework)	Auxiliary-loss-free, adaptive balancing	Large-scale MoE training
Block-wise attention residuals	Medium (structural adjustment)	Deep-layer representation, memory and communication reduction	Deep / long-context architectures

Suggested evaluation order:

1. Teams already on the linear attention path
   Prioritize lower-bound parameterization. It has the lowest cost and immediate benefits.

2. When MoE expert counts reach hundreds
   Evaluate replacing auxiliary losses with routing-internal balancing mechanisms. Validate convergence behavior at small-to-medium scale before scaling up.

3. When depth scaling hits representation saturation or memory walls
   Consider structural changes such as block-wise residuals.

6. Three General Lessons for Engineering Teams

1. Design numerical ranges before optimizing operators.
   First introduce boundedness and normalization so that all tiles follow the same fast path. Only then consider kernel fusion and scheduling.

2. Do not externalize constraints that the system can internalize.
   Global constraints such as load balancing should be moved into the decision mechanism rather than attached to the loss function, where they compete with the main objective for gradient.

3. Solve structural overhead with structural changes.
   Beyond framework-level optimization, memory and communication issues can also be addressed at the architecture level through blocking, sparsification, and coarse-grained aggregation.

Whether a team builds its own training pipeline or uses general-purpose GPU compute and training acceleration services from cloud platforms such as Baidu AI Cloud, these three techniques are worth adding to the architecture review checklist. They point in the same direction: making model architectures capable of bearing the engineering load of scale across numerical stability, load balancing, and depth extension.