A Verifiable Analysis of Two Optimizations: Linear Attention and MoE Load Balancing
This article examines two proposed changes to Transformer training: a lower bound on the decay factor in linear attention, and auxiliary-loss-free load balancing for Mixture-of-Experts (MoE) models. It derives the formulas and numerical limits for the first, describes a bias-based method for the second, and lists monitoring, rollback, and small-model validation steps. It also identifies claims that the source does not support in enough detail to implement, including deep residual connections and NoPE length extrapolation.
1. Scope and Verification Boundaries
The source describes three changes: a lower bound on the linear attention decay, auxiliary-loss-free load balancing for MoE, and deep residual connections. The first two can be derived from general principles, so this article provides formulas, pseudocode, and monitoring methods for them. The source does not define the deep residual mechanism, including its “cross-chunk residuals,” to the level needed for implementation. It also gives no paper, code, or experimental setup for it. This article therefore does not treat deep residuals as a conclusion. Section 5 lists the information needed to verify them.
This article does not compare performance numbers for specific models, and it does not endorse the benefit figures in the source. Readers should be familiar with Transformers, BF16 numerical formats, and expert parallelism.
Each conclusion is labeled with its source. “Derived” means the result follows directly from a formula or numerical rule. “Literature” means the result is cited from public work and should be checked against the original publication. “To be verified” means the source gives no basis for the claim, so it should not be adopted directly.
2. Linear Attention: Lower-Bounded Decay
2.1 Parameterization and Numerical Risk
The decay factor γ_t at each step lies between 0 and 1. Let g_t = log γ_t, so g_t ≤ 0. A common parameterization is g_t = -softplus(a_t). This form has no lower bound, so γ_t can approach 0 arbitrarily closely.
Within a chunk, the computation requires exp(-b_i), where b_i is the prefix sum of g within the chunk. When |b_i| becomes large, exp(-b_i) exceeds the representable range of finite-precision formats.
(Derived) One way to impose a lower bound l on γ_t is:
γ_t = l + (1 - l) * sigmoid(a_t), where l ∈ (0, 1)
This gives log l < g_t < 0. The bound l limits the minimum decay at each step. For a chunk length C, the prefix sum within a chunk satisfies |b_i| ≤ C·|log l|, so exp(-b_i) ≤ exp(C·|log l|). This upper bound determines whether the reciprocal is safe.
Prefix-sum convention: b_i = Σ g_t, summed from the start of the chunk to position i, including the current step. Each chunk restarts the accumulation from 0. The pseudocode below uses this convention. If an implementation uses a prefix sum that excludes the current step, the exponents of q_s and k_s must shift accordingly. The two conventions must not be mixed.
2.2 Numerical Safety Ranges
The table below gives the limits of two numerical formats. The suggested conditions are empirical values with a margin, not conclusions from the source paper.
| Format | Maximum value | Natural log upper bound | Suggested condition | ||
|---|---|---|---|---|---|
| FP16 | 65504 | about 11.1 | C·\ | log l\ | < 10 |
| FP32 | about 3.4e38 | about 88.7 | C·\ | log l\ | < 40 |
| BF16 | Same exponent range as FP32 | about 88.7 | C·\ | log l\ | < 40, and monitor precision separately |
BF16 has the same exponent range as FP32, so it does not fail from exponent overflow. However, it has only a 7-bit mantissa. Errors grow in multiply-add operations that involve large values, so this must be confirmed through monitoring.
The table below gives values of C·|log l| (derived):
| l | C=32 | C=64 | C=
