Hybrid Linear Attention and Attention Residuals: Architecture Choices in Trillion-Parameter Models

    This article examines architecture choices behind trillion-parameter LLMs, focusing on sparse MoE scaling, hybrid linear attention (KDA), cross-layer attention residuals, native multimodal design, and deployment cost. It explains how capacity and per-token compute separate in MoE, why hybrid attention combines fixed-size linear states with local Softmax windows, and how cross-layer residual connections can preserve shallow features. It also covers native multimodal token integration, cloud deployment constraints, and a KV cache memory formula with a 1M-token example.

    Introduction: Architecture Choices Reveal More Than Scores

    Competition among large language models has moved beyond simply stacking parameters toward a combined contest of architecture, data, and engineering capability. To assess a new model, look not only at benchmark scores but also at which technical route it has chosen. Recently, several model teams have been pursuing a route of “very large parameters + sparse experts + hybrid attention.” This article focuses on three key designs in that route: hybrid linear attention, cross-layer attention residuals, and native multimodal architecture. The parameter scale used in this article is 2.8 trillion (roughly 3T-class).

    1. Scale and Engineering Cost

    1.1 Sparse Experts Balance Capacity and Cost

    An MoE model has a large total parameter count, but each token activates only a small number of experts, commonly 1 to 2, depending on the model configuration. Therefore, per-token compute is no longer proportional to total parameters. Knowledge capacity comes from total parameters, while per-inference compute comes from activated parameters. Separating the two is the core benefit of this design.

    1.2 Three Training Bottlenecks

    • Data: High-quality text and multimodal data cannot grow as fast as compute. Synthetic data can supplement the shortfall, but it must undergo strict quality filtering; otherwise, it can amplify model bias.
    • Parallelism: Very large-scale training usually combines tensor parallelism, pipeline parallelism, sequence parallelism, context parallelism, and expert parallelism. The combination directly determines cluster utilization (MFU).
    • Stability: Loss spikes can occur during training. Common practices include gradient clipping, learning rate warmup, and detection and rollback of anomalous steps. Optimizer state sharding (such as ZeRO-like methods) is used to reduce per-GPU memory usage.

    2. Hybrid Linear Attention

    2.1 Why Hybrid Attention Is Needed

    The compute and memory overhead of standard Softmax attention grow quadratically with sequence length L. Linear attention uses kernel decomposition to write attention in recursive form, with a fixed state size independent of sequence length. The cost is weaker exact retrieval capability; early linear attention often showed clear losses in exact recall and complex reasoning. The idea of a hybrid architecture is to use linear modules for most long-range information processing and a small number of Softmax layers or local windows for precise modeling.

    2.2 Division of Work Between Two Modules

    The KDA (Kimi Delta Attention) mentioned in this context can be understood as two modules:

    • Global linear compression module: It uses linear recurrence with a delta update rule to compress historical information into a fixed-size state matrix for each attention head. The state dimension is d_k × d_v and does not grow with context.
    • Local Softmax window module: It retains exact Softmax attention within a sliding window. Information outside the window is passed only through the compressed state.

    Routing can be understood as fixed allocation by layer or module, rather than dynamic per-token selection. Window size and the proportion of linear layers are engineering configurations; this article does not provide official values. If the window is set to 4096, the KV cache of the local module depends only on the window length. This value is an example and does not represent an official configuration.

    2.3 Applicable Boundaries of Accuracy Loss

    Hybrid structures perform well in scenarios such as long-document summarization, topic maintenance in long conversations, and repository-level code understanding. The following scenarios require empirical testing: tasks that require exact recall of a number or entity from hundreds of thousands of tokens earlier, and tasks that depend on strict character-level alignment. When selecting a model, validate it on your own dataset rather than relying only on aggregate benchmarks.

    3. Attention Residuals

    3.1 Review of Standard Residuals

    The standard residual in a Pre-Norm structure can be written as:

    x_{n+1} = x_n + F_n(LN(x_n))

    Each layer reads only the output of the previous layer, and gradients flow back along the identity path.

    3.2 Cross-Layer Attention Residuals

    A general modeling approach is to add the attention outputs of the previous k layers to the attention input at layer n:

    an = Attn(LN(x_n)) + g_n · Σ{j=n-k}^{n-1} W_{n,j} · a_j

    Here, gn is a learnable gate coefficient, W{n,j} is a projection matrix, and k is the cross-layer interval. This is a generalized description of cross-layer attention residuals, not the exact official form. Specific implementations should follow the official technical report.

    Compared with standard residuals, this design has two main differences. First, shallow attention features can be passed directly to deep layers, reducing information decay after multiple transformations. Second, the gate lets the model learn whether to use cross-layer information, thereby avoiding the introduction of noise in early training.

    3.3 Benefits That Need Validation

    In theory, this type of design shortens the gradient path and preserves shallow features. Actual benefits require ablation experiments. It is advisable to compare three groups: standard residuals, fixed-weight cross-layer residuals, and gated cross-layer residuals. Under the same compute budget, observe convergence speed and long-context performance.

    4. Native Multimodal Design

    4.1 Concatenated vs. Native Multimodal

    Concatenated multimodal approaches first train a vision encoder and a language model separately, then use a projection layer to map visual features into the word embedding space. This method is simple to implement, but the semantic spaces of visual tokens and text tokens are not consistent, and alignment effectiveness is limited by the capacity of the projection layer.

    Native multimodal design places image, text (and audio) tokens into the same representation space during pretraining and shares attention and residual structures. Visual information can therefore participate in attention computation rather than merely serving as a substitute for textual descriptions.

    4.2 Key Points for Integrating Visual Tokens

    Vision encoders usually adopt ViT-like structures and split images into sequences through Patch Embedding. The dimension of visual tokens must align with the hidden dimension of the language model; otherwise, an additional projection layer is required. Specific encoder structures and dimension configurations should follow official public information; this article does not make inferences.

    4.3 Usage Scenarios

    For example, when a user uploads a system architecture diagram and asks how it is implemented, a native multimodal model can simultaneously read the text labels and arrow directions in the diagram and associate them with the question text. The strength of this capability needs to be evaluated through benchmarks such as chart question answering and document understanding, as well as on proprietary datasets.

    5. Developer and Cloud Service Perspective

    5.1 Deployment Form

    When the parameter scale reaches the trillions, local deployment is usually infeasible, and most requests are completed through cloud APIs. This imposes three requirements on service providers: elastic scaling and high concurrency, low-latency streaming output, and prefix caching and reuse.

    5.2 Impact on Model Selection

    When choosing a model, developers should rely on task metrics and cost data rather than brand impressions. It is advisable to build your own evaluation set covering long context, multimodality, code generation, and refusal boundaries, and to record the actual cost per million tokens and latency.

    6. Inference Cost and Memory Estimation

    6.1 KV Cache Estimation Formula

    The KV cache occupied per token can be estimated as follows:

    Per-token memory = 2 × number of layers × number of KV heads × head_dim × bytes per element

    The coefficient 2 represents the two caches for K and V. Bytes per element are 2 for FP16 or BF16 and 1 for INT8.

    Assume an example configuration: 60 layers, 8 KV heads, head_dim of 128, and BF16 storage. Then per token is approximately 2 × 60 × 8 × 128 × 2 = 245,760 bytes, or about 245 KB. With a context of 1 million tokens, the cache for a single sequence is approximately 246 GB (decimal). If INT8 quantization is used, this is roughly halved.