Bounded Decay for KDA: Preserving Tensor Core Acceleration Under BF16

    This article examines bounded decay parameterization for KDA (Kimi Delta Attention-style linear attention), which replaces unbounded Negative-Softplus parameterization with a scaled Sigmoid that guarantees a clear lower bound on per-step decay values. It walks through the numerical verification at block size 16 under BF16, showing why the accumulated within-block decay and its reciprocal stay within representable range, explains the conditions under which the result holds, and outlines practical checkpoints for adoption, including FP16 caveats and monitoring recommendations. Readers will also find a comparison table between unbounded and bounded approaches and a checklist for evaluating whether this technique fits their linear attention workloads.

    What This Article Covers

    The source material references three techniques: bounded decay parameterization for KDA, Quantile Balancing, and Block Attention Residuals. Only the first one comes with formulas, parameters, and numerical conditions in the source, so this article fully develops only that technique. For the other two, readers should consult the original paper or official technical report; here they are described only at the level of the problems they solve, without deriving their implementations.

    Background: Three Bottlenecks in Ultra-Large Models

    As parameter counts push toward the trillion scale and context windows extend to a million tokens, standard Transformers run into three classes of problems.

    The first is numerical stability of attention computation. Standard Softmax attention scales quadratically with sequence length, which motivates many linear attention variants. When computation is chunked, decay factors are multiplied repeatedly. In BF16 precision, these products can easily overflow or underflow. Once values go out of range within a block, the kernel must fall back to position-by-position iteration, losing access to Tensor Core matrix multiplication and dropping throughput.

    The second is MoE load imbalance. With hundreds or even thousands of experts, uneven routing slows down training. Common auxiliary losses interfere with gradients of the main task and require synchronizing global statistics across devices.

    The third is representational collapse in deep networks. Once the layer count reaches the hundreds, representations across layers grow increasingly similar, and residual connections alone share information between layers inefficiently.

    State Updates in Linear Attention

    Linear attention compresses historical information into a state matrix, updated recursively:

    St = γ_t · S{t-1} + k_t v_t^T

    Here S_t is a d_k × d_v state matrix, k_t and v_t are the key and value vectors, and γ_t is the decay factor that controls how much old information is retained. At output time, a query vector q_t reads the state, yielding o_t = S_t^T q_t. The original source does not specify the values of d_k and d_v.

    The Problem with Unbounded Parameterization

    Traditional approaches (such as Delta Net) parameterize the logarithm of the decay with an unbounded Negative-Softplus function. This is flexible early in training, but when input features have a large dynamic range, decay factors can approach extreme values. Once within-block products go out of range, the kernel must take a separate explicit position-wise traversal branch for diagonal blocks, and efficiency drops noticeably.

    KDA’s Bounded Decay Parameterization

    KDA’s core idea is to replace the unbounded function with a scaled Sigmoid that has explicit upper and lower bounds:

    g_t = g_min · Sigmoid(Linear(x_t)) + b

    The source sets g_min = -5. With the bias b = 0, the Sigmoid output falls in (0, 1), so g_t lies in (-5, 0). The decay factor γ_t = exp(g_t) therefore lies in (e^-5, 1), approximately (0.0067, 1). Regardless of the input, every per-step decay value has a definite lower bound.

    This formulation turns the numerical range into a constant that can be computed in advance, rather than something observed empirically during training.

    Numerical Verification at Block Size 16

    With 16 tokens per block, the minimum per-step log-decay is -5, so the minimum accumulated log-decay within a block is 16 × (-5) = -80. The corresponding accumulated decay is e^-80, approximately 1.8×10^-35.

    In chunked parallel implementations, the reciprocal of the accumulated within-block decay product P, namely 1/P, is often used for normalization or rescaling. The original source does not provide an explicit formula; the derivation below follows the general chunked linear attention formulation and should be checked against the original paper. If the implementation requires 1/P, its worst case equals e^80, approximately 5.5×10^34. Checking against BF16 limits:

    • The maximum finite BF16 value is about 3.39×10^38, sharing the same exponent range as FP32.
    • The minimum normal BF16 value is about 1.18×10^-38.
    • Both e^-80 ≈ 1.8×10^-35 and 5.5×10^34 fall within these ranges, so neither overflows nor underflows to zero.

    Therefore, with the source’s parameters, the worst-case accumulated decay within a block and its reciprocal both stay in range under BF16.

    Conditions Under Which the Verification Holds

    The source’s conclusion is that once numerical ranges are bounded, both diagonal and off-diagonal blocks can directly use dense Tensor Core GEMM, with no separate branch needed for diagonal blocks. This conclusion depends on the following conditions:

    1. Block size is 16.
    2. Precision is BF16.
    3. The CUDA kernel corresponding to the original paper is used.

    The source does not name the kernel or describe its implementation details, so this article does not fill that gap. If any condition changes, the verification must be redone.

    Comparison with Unbounded Approaches

    Dimension Unbounded Negative-Softplus KDA Bounded Parameterization
    Decay value range No lower bound; can approach extreme values when input dynamic range is large (e^-5, 1), with an explicit lower bound
    Accumulated within-block decay Can go out of range At block size 16, minimum is about e^-80
    BF16 overflow risk Falls back to traversal when out of range No overflow under the source’s parameters, verified above
    Tensor Core availability Uses the traversal path when out of range The source claims direct Tensor Core GEMM at block size 16, BF16, and with the corresponding kernel; the fallback branch should be confirmed by testing

    Model Specifications: Which Numbers Need Verification

    The source material describes a multimodal MoE model. All figures in the table below come from the source’s abstract and body; they have not been verified against any official release page, so check them before citing.

    Metric Claimed in Source Meaning
    Total parameters 2.8 trillion Upper bound on the knowledge the model can hold
    Activated parameters 104 billion Parameters actually used in a single forward pass
    Context window 1 million tokens Supports very long documents and long-range reasoning
    Routed experts 896 Number of experts under sparse activation
    Modalities Vision and text Supports joint image-text understanding

    The activation ratio is roughly 104 ÷ 28000 ≈ 3.7%, or about 1:27. Total parameters determine how much knowledge the model can hold, while activated parameters determine the actual cost of each forward pass. The source does not state whether the vision encoder is counted in the total; note the accounting basis when citing.

    Adoption Checklist

    When evaluating KDA for a technology selection, check the following:

    1. Record the minimum accumulated log-decay within each block. Under the source’s parameters, its lower bound is 16 × (-5) = -80.
    2. Keep the block size fixed at 16. If it changes, redo the verification.
    3. Confirm the precision. BF16’s exponent range is sufficient, but FP16 behaves differently: FP16’s maximum is about 65504, its minimum normal value about 6.1×10^-5, and its minimum subnormal about 6×10^-8. In FP16, e^-80 would underflow to 0, and 5.5×10^34 would overflow to infinity. The conclusions do not transfer directly to FP16.
    4. Monitor whether the kernel takes the fallback branch. When throughput drops suddenly, check this first.
    5. If you adjust the lower bound g_min, recompute the relationship between 16·|g_min| and the numerical range of your target precision.

    Risks and Selection Recommendations

    Long-context training carries three main categories of risk, each requiring separate monitoring:

    • Memory: grows with context length. Track peak memory usage and consumption at the maximum context length.
    • Communication: as the expert count grows, cross-device communication becomes heavier. Track all-to-all latency per step and the communication share.
    • Numerical stability: issues may only surface late in training. Track the minimum decay values per layer and the frequency of fallback branch triggers.

    Run stress tests covering maximum context length and peak concurrency before launch, and prepare a rollback plan, such as keeping the previous checkpoint and kernel configuration.

    If your model uses linear attention and decay factors are multiplied within blocks, run a numerical verification first before deciding whether to adopt bounded decay. The decision should rest on your own precision, block size, and kernel conditions—not solely on the source’s conclusion.