KDA Hybrid Attention, Residual Attention, and Latent MoE: Inside the New LLM Architecture Paradigm

    This article breaks down three architecture mechanisms popularized by Moonshot AI's Kimi series—KDA (Kimi Delta Attention) hybrid attention, residual attention, and Mixture of Latent Experts (LatentMoE)—and explains how each targets a different inference cost: attention memory, inter-layer information flow, and expert parameter overhead. It covers the cost structure of standard attention, how gated delta-rule linear attention compresses the KV cache, how residual attention replaces fixed layer addition with data-driven cross-layer aggregation, and how latent experts factorize expert weights through a shared latent space. A comparison table, a deployment verification checklist, and notes on engineering trade-offs are included. Explanations are based on the authors' public technical materials and general research in linear attention and sparse experts; actual adoption, ratios, and dimensions should be confirmed against official technical reports and benchmarks.

    $$# KDA Hybrid Attention, Residual Attention, and Latent MoE: Inside the New LLM Architecture Paradigm

    1. Background: Inference Cost Is Now Driving Architecture Evolution

    For the past few years, discussions about large-model performance centered on “parameter scale” and “data scale.” After 2025, however, the focus has clearly shifted toward “inference efficiency at equal quality.” The reasons are straightforward:

    • Contexts keep getting longer. Long-document understanding, codebase-level question answering, and multi-turn agent tasks have pushed context windows from a few thousand tokens to hundreds of thousands, or even over a million tokens.
    • Interaction has moved to agents. A single task may require dozens of consecutive reasoning steps, each carrying the full history. Repeated memory access during the decoding phase has become the primary bottleneck.
    • Commercial constraints are stricter. The per-token cost directly determines whether a model can be embedded at scale into production pipelines.

    Against this backdrop, the marginal returns of “just scaling up” are diminishing, while the room for “changing the structure” is being re-examined. Since 2025, the architecture terms most discussed around Moonshot AI’s open-source Kimi series have been, in order:

    • KDA: Kimi Delta Attention, a fine-grained gated delta-rule linear attention;
    • AttnRes: Residual attention, which reworks the ordinary residual connection into dynamic cross-layer aggregation;
    • LatentMoE: Mixture of Latent Experts, which represents expert weights in a shared latent space.

    The three act on the attention layer’s “memory cost,” the inter-layer connection’s “information flow,” and the expert layer’s “parameter cost,” respectively, forming a coordinated design aimed at inference efficiency. Their shared logic can be summarized in one sentence: instead of unconditionally adding compute and parameters, the model learns to “compute on demand, remember on demand, and fetch parameters on demand.”

    One boundary needs to be drawn first. The explanations of these three mechanisms below come from the model authors’ public technical materials and from general findings in areas such as linear attention and sparse experts—not from hands-on testing of any specific release. In a concrete version, the ratios, dimensions, and enabling conditions of each mechanism may be adjusted. Whether they are adopted, and to what extent, should be confirmed against official technical reports and actual benchmarks.

    2. Attention Layers: KDA Hybrid Attention

    2.1 The Cost Structure of Standard Attention

    The core of standard multi-head attention is a key-value cache (KV Cache) that grows with sequence length. When generating the t-th token, the model must read the K and V vectors of all previous tokens:

    1. KV memory ≈ 2 × layers × KV heads × head dim × sequence length × bytes per precision × concurrent requests

    This expression reveals two facts:

    1. In long contexts, the KV Cache often occupies more GPU memory than the model weights, and it grows linearly with concurrency;
    2. During decoding, every generated token requires moving the entire KV Cache from GPU memory into the compute units. Decoding is therefore a typical memory-bound workload.

    2.2 Linear Attention and the Gated Delta Rule

    Linear attention removes the softmax normalization and reorders the computation using matrix operations. When the kernel can be decomposed into feature maps, we have:

    1. o_t = Σ_{i<t} (φ(q_t)ᵀφ(k_i)) v_i / Z
    2. = φ(q_t)ᵀ · [ Σ_{i<t} φ(k_i)v_iᵀ ] / Z

    The quantity in brackets depends only on the history and can be maintained incrementally. As a result, the model no longer needs to store the K and V of every historical token; it only maintains a state matrix of “key dim × value dim.” No matter how long the context, the state size stays fixed. The cost is that compressing history into a fixed-size state is necessarily lossy. When precisely retrieving a single piece of historical information, linear attention is clearly weaker than full attention.

    The gated delta rule introduces two modifications on top of ordinary linear attention, which can be written as:

    1. S_t = α_t · (I − β_t · k_t k_tᵀ) · S_{t−1} + β_t · v_t k_tᵀ
    2. o_t = S_tᵀ q_t
    • The delta-rule part (I − β_t k_t k_tᵀ): before writing in a new key-value pair, it first erases the old content in the state that is related to the current key k_t, resembling a single step of online gradient descent. This alleviates the memory interference caused by ordinary linear attention’s “only accumulate, never modify” behavior.
    • The forget gate α_t: controls how much of the old state is retained, allowing the model to decay historical information by distance.
    • The key difference of KDA: the forget gate is changed from “one shared scalar per channel” to “independent per feature dimension,” i.e., fine-grained, data-dependent gating. For example, a dimension representing temporal information can decay slowly, while a dimension representing a concrete fact can be more precisely retained or cleared.

    Below is a sketch implementation of a single decoding step:

    1. import torch
    2. def kda_decode_step(q, k, v, beta, alpha, state):
    3. """
    4. Gated delta-rule update for a single decoding step (illustrative, not runnable as-is)
    5. q, k, v : [batch, heads, head_dim]
    6. beta : [batch, heads] write strength, usually obtained via sigmoid
    7. alpha : [batch, heads, head_dim] per-dimension forget gate
    8. state : [batch, heads, head_dim, head_dim] memory state reused across steps
    9. """
    10. kt = k.unsqueeze(-1) # [B, H, D, 1]
    11. vt = v.unsqueeze(-2) # [B, H, 1, D]
    12. # 1. delta rule: first erase old memory related to the current key, then write the new key-value association
    13. state = (state - beta[..., None, None]
    14. * (state @ kt) @ kt.transpose(-1, -2))
    15. state = state + beta[..., None, None] * vt @ kt.transpose(-1, -2)
    16. # 2. per-dimension forget gating
    17. state = state * alpha[..., None, :]
    18. # 3. read from the state using the query
    19. return torch.einsum('bhd,bhde->bhe', q, state)

    In practice, this recursion is not executed token by token. During training, it typically takes a chunkwise parallel form: matrix multiplication runs in parallel within a chunk, and only the state is passed between chunks. During inference, it is coordinated via “computing the state in the prefill phase and updating it step by step in the decoding phase.”

    2.3 Why “Hybrid” Instead of “Full Replacement”

    Using linear attention alone loses the ability to precisely retrieve. The common compromise is to interleaved stack the two types of layers by ratio. Public technical materials indicate that the Kimi series adopts a scheme of inserting a multiple number of KDA layers between a few full-attention layers, driving the KV Cache overhead of the vast majority of layers down to constant cost while keeping a small number of full-attention layers responsible for precise retrieval.

    Three points are worth noting in engineering practice:

    • Positional encoding: KDA reuses the same type of rotary positional encoding as the full-attention layers, ensuring the two types of layers express sequence positions consistently;
    • Precision selection: gated computation is sensitive to numerical stability, and full high precision would sacrifice efficiency. The design typically keeps higher precision only for key operators such as the forget gate, using low precision for the rest;
    • Scheduling coordination: in the prefill phase, KDA layers perform a linear scan and produce the state; in the decoding phase, they update the state at constant cost. Combined with a deployment style that “separates prefill from decoding,” the two phases can be optimized independently.

    3. Connection Layers: Residual Attention

    3.1 Two Limitations of Traditional Residuals

    Inter-layer connections in Transformers have long used the fixed summation “current-layer output + sublayer output.” The advantage is simplicity and optimization-friendliness; the limitations are equally clear:

    1. Each layer can only receive the output of its immediately preceding layer, with a fixed information flow;
    2. At depths of dozens or even a hundred layers, the residual stream tends to homogenize, and fine-grained information from shallow layers gradually decays as it passes layer by layer.

    Prior work has tried to alleviate this with gating (Highway Networks) or dense connections (DenseNet), but these are either static gates or introduce substantial memory overhead.

    3.2 The Basic Idea of Residual Attention

    The core idea of residual attention is to change each token’s “residual reception target” from a fixed tensor to a data-determined weighted average. For layer l:

    1. H_l = Σ_{i=0}^{l−1} α_{l,i} ⊙ (H_i + F_i(H_i))
    2. α_{l,i} = Softmax_i( q_l^T · LN(H_i) )

    where F_i is the attention and feed-forward sublayers of layer i, q_l is dynamically generated from the current state, and α is the normalized weight over historical layer outputs. Notice that:

    • when α degenerates into uniform weights, it approximates a dense connection that “averages the outputs of all layers”;
    • when α concentrates only on adjacent layers, it degenerates back into an ordinary residual.

    Therefore, the ordinary residual can be viewed as a special case of residual attention.

    Below is a sketch implementation:

    1. import torch
    2. import torch.nn as nn
    3. class AttentionResidual(nn.Module):
    4. """Illustrative residual attention: replace fixed summation with cross-layer attention"""
    5. def __init__(self, dim, heads, head_dim, eps=1e-6):
    6. super().__init__()
    7. self.proj = nn.Linear(dim, 3 * heads * head_dim)
    8. self.out = nn.Linear(heads * head_dim, dim)
    9. self.heads, self.head_dim = heads, head_dim
    10. self.norm = nn.LayerNorm(dim, eps=eps)
    11. def forward(self, x, history):
    12. # x: [B, T, D] current layer output
    13. # history: [B, T, D, L] outputs of all previous layers (illustrative)
    14. b, t, d, l = history.shape
    15. q, k, v = self.proj(x).chunk(3, dim=-1)
    16. # treat each layer's output as a "key-value position" and do cross-layer attention
    17. q = q.view(b, t, self.heads, self.head_dim)
    18. k = k.view(b, t, self.heads, self.head_dim).permute(0, 2, 1, 3)
    19. v = v.view(b, t, self.heads, self.head_dim).permute(0, 2, 1, 3)
    20. # each historical layer output x_i + F_i(x_i) provides a set of K, V
    21. kk = torch.cat([k] + [self.proj(h).view(b, t, self.heads,
    22. self.head_dim).transpose(1, 2) for h in history.unbind(-1)], dim=2)
    23. vv = torch.cat([v] + [self.proj(h).view(b, t, self.heads,
    24. self.head_dim).transpose(1, 2) for h in history.unbind(-1)], dim=2)
    25. weights = torch.softmax(q @ kk.transpose(-1, -2) / self.head_dim ** 0.5, dim=-1)
    26. out = (weights @ vv).flatten(-2)
    27. return x + self.out(out)

    3.3 Benefits and Costs

    The benefit of residual attention is that shallow token-level local features and deep semantic representations can be combined dynamically per token, rather than being averaged and diluted at every layer. This is especially important under long contexts and noisy inputs, because the model can actively “pick out” the more useful layers.

    The costs are equally obvious:

    • Memory: all historical layer outputs must be cached. In practice, this is usually projected down to a lower dimension, or the historical layers are summarized;
    • Compute: cross-layer attention is an extra set of matrix multiplications per layer, whose cost-effectiveness must be evaluated under high concurrency;
    • Implementation: it breaks the standard layer-stacking structure, affecting model-parallel sharding and the layer scheduling of inference frameworks.

    4. Expert Layers: Mixture of Latent Experts (LatentMoE)

    4.1 The Basic Structure of Sparse MoE

    A sparse Mixture-of-Experts uses a gating network to select a few experts for each token (usually 1 to 2), activating only the parameters of the selected experts. Thus:

    1. Total parameters ≈ attention layers + N experts; activated parameters ≈ attention layers + top-k experts

    Taking the “fine-grained experts + shared experts” combination as an example: there are many routing experts, but each is small, plus a few shared experts that always participate in computation to absorb general knowledge.

    Its bottleneck is that although the activated amount is small, all expert parameters must remain resident in GPU memory; and under expert parallelism, tokens must be sent to the device where the expert resides via all-to-all communication.

    4.2 Latent Experts: Representing Weights in a Shared Space

    The idea of latent experts is to rewrite each expert’s forward computation as “first project into a shared latent space, then map back to the original space via that expert’s dedicated decoder”:

    1. Standard expert: y = W_i · x, W_i ∈ R^{h×d}
    2. Latent expert: y = D_i · (P · x), P ∈ R^{r×d}, D_i ∈ R^{h×r}, r ≪ d
    • P is a down-projection matrix shared by all experts, compressing the input into an r-dimensional latent space;
    • D_i is the up-projection matrix dedicated to expert i, reduced in size from h×d to h×r;
    • when r is far smaller than d, each expert’s parameter count drops to roughly r/d of the original.

    This can be understood as a “shared-factor” low-rank decomposition of the expert weight matrix W_i ≈ D_i·P. Experts no longer each store a complete set of weights; instead, they share the same latent basis and keep only their own coefficients. This idea is in the same lineage as low-rank compression in attention (such as representing keys and values with latent variables).

    Public materials indicate that Kimi introduced the latent expert design in an earlier version. Its motivation is not only saving GPU memory: there is parameter redundancy between standard experts and shared experts, and deep layers are prone to “feature collapse.” A shared latent space can constrain the representational differences between experts and reduce ineffective redundancy.

    4.3 Engineering Value

    • Memory: total expert parameters shrink significantly, and the number of experts a single GPU can hold multiplies;
    • Communication: the all-to-all transfer volume under expert parallelism decreases accordingly, making cross-machine deployment friendlier;
    • Training: the latent space limits each expert’s degrees of freedom, reducing the risk of overfitting and representational collapse;
    • Combined optimization: the structure of latent experts can share the same compression paradigm as low-rank compression on the attention side, making the model easier to maintain as a whole.

    4.4 Risks and Boundaries

    Latent experts are not a free lunch:

    1. Capacity loss: too small an r loses fine-grained knowledge and must be validated on large-scale corpora;
    2. Routing stability: after the expert size changes, the load-balancing strategy and auxiliary-loss weights must be re-tuned;
    3. Low-rank assumption: when experts are highly functionally differentiated, the shared basis may limit their expressiveness, requiring a moderate number of expert-private dimensions.

    5. Working Together: A “Combo” Aimed at Inference Efficiency

    Looking at the three designs together, each solves one bottleneck:

    Mechanism Where it acts Core technique Main benefit Main cost
    KDA hybrid attention Attention layer Gated delta-rule linear attention + a few full-attention layers Sharply compresses the KV Cache, reducing long-context memory and bandwidth Limited precise-retrieval ability of linear attention
    Residual attention Inter-layer connection Per-token dynamic weighting over historical layer outputs On-demand combination of fine-grained and semantic information Extra memory and compute; breaks the standard layer structure
    Latent experts Expert layer Shared latent space + expert-specific decoder Expert parameter compression; more experts become feasible Low-rank capacity constraint; routing needs re-tuning

    Stacked together, the overall benefit is not a simple sum but mutual amplification: KDA compresses the KV Cache, while residual attention needs to cache multiple layers of output—together they determine the memory budget per request; latent experts compress the weights, making “more experts + narrower activation” feasible, which is consistent with the “compute on demand” logic of linear attention.

    6. From Architecture to Deployment: A Verification Checklist for Enterprise Adoption

    Architectural innovation must ultimately stand up to engineering scrutiny. It is recommended to confirm the following item by item before going live:

    1. Context budget: estimate the KV Cache and residual-attention history cache based on actual business lengths, stack the latent-expert weight footprint on top, and reconcile against single-GPU memory and concurrency;
    2. Framework support: pay attention to model weight formats, quantization schemes, and support status in mainstream inference frameworks. New architectures can often run only through community adaptation before official support matures;
    3. Task evaluation: focus on testing precise-retrieval abilities such as long-document extraction and cross-section retrieval, and compare against a same-scale full-attention baseline;
    4. Deployment shape: for long-context scenarios, consider a deployment that separates prefill and decoding; for expert-parallel deployment, assess whether latent experts reduce cross-machine communication pressure;
    5. Stability and observability: set timeouts and fallback strategies for long requests, and track metrics such as time-to-first-token, per-token generation latency, and GPU memory usage through logging services and monitoring alerts;
    6. Reproducibility of evaluations: community-circulated conclusions like “task X finished in Y minutes for Z dollars” usually lack reproducible details (whether the prompt was frozen, whether scoring was independent, whether failure counts were recorded) and should not be used as a basis for decisions. A more valuable approach is to fix the evaluation set and prompts, sample multiple times to obtain a distribution, rather than reporting only the single best result.

    7. Conclusion: The “Spear” and “Shield” of Architectural Innovation

    KDA hybrid attention, residual attention, and latent experts are, in essence, all answering the same question: in an era when compute is no longer free, how do we make every bit of parameter and every byte of GPU memory count where it matters most? Their direction is aligned, but their maturity differs:

    • Linear attention and hybrid stacking already have relatively complete theoretical groundwork, with the main challenge in engineering implementation;
    • Residual attention relies more on training strategy and framework coordination;
    • Latent experts must repeatedly balance capacity against efficiency.

    For enterprise developers, the most pragmatic attitude is to pay attention to architectural trends, but validate against real business workloads as the yardstick, rather than being led by a single leaderboard or community summary. As open weights keep iterating and inference-framework adaptation gradually catches up, understanding the “why” behind these mechanisms has more long-term value than memorizing the benchmark scores of a particular model.