Long-Context and Deep Network Information Flow Optimization: Techniques and Trade-offs
This article analyzes two distinct bottlenecks in large-model training and inference: growing sequence length and increasing network depth. It explains the cost model of standard attention, fixed-size state alternatives such as linear attention, the delta update rule, and practical trade-offs. It then examines depthwise information flow, including the limits of standard residual connections and learnable aggregation across layers. The article also provides illustrative code, evaluation guidance, and deployment recommendations for long-context and deep-network optimization.
1. Two Bottlenecks: Longer Sequences and Deeper Networks
The training and inference costs of large models are mainly affected by two dimensions. The first is the sequence dimension, that is, the length of the input context, n. The second is the depth dimension, that is, the number of network layers, L. The two bottlenecks have different sources and require different optimization approaches.
On the sequence dimension, standard scaled dot-product attention computes the relevance between every position and every other position, so the amount of computation is proportional to n². During inference, to avoid repeated computation, the model must cache historical keys and values, and memory usage grows linearly with n. When the context reaches the million-token scale, both computation time and memory pressure become difficult to bear.
On the depth dimension, the problem appears as information gradually decaying or being overwritten as it passes between layers. Residual connections allow each layer to learn only an increment, so signals can pass directly through multiple layers. However, when there are many layers, the outputs of all layers are continuously accumulated, and the relative share of early-layer features in the residual stream becomes smaller and smaller. Deep networks therefore struggle to effectively use shallow representations, and training also relies more on careful initialization and normalization design.
2. Sequence Dimension: Replace Full Cache with a Fixed-Size State
2.1 Cost Model of Standard Attention
Let the sequence length be n and the feature dimension be d. The main computation of standard attention is approximately O(n²d), and the KV cache size is approximately O(nd). When the context length increases tenfold, the amount of computation increases by roughly one hundredfold.
2.2 Basic Idea of Linear Attention
Linear attention compresses historical information into a fixed-size state matrix S whose dimension is independent of n. When each new token arrives, the model reads and updates the state, and the overall computational complexity drops to O(nd²). When n is greater than d, this complexity is clearly lower than O(n²d).
The cost is that compression inevitably loses some information. The state capacity is limited. If the model needs to accurately recall a detail from a long time ago, the compressed state may not retain it.
2.3 Delta Update Rule
Directly accumulating all new information into the state will cause old memories to be repeatedly overwritten or added, creating interference. The delta rule works as follows: first use the current state to predict the value corresponding to the new token, and then write only the difference between the predicted value and the true value.
The formula is as follows:
St = S{t-1} + βt (v_t − S{t-1} k_t) k_tᵀ
Here, k_t and v_t are the key and value of the current token, respectively, and β_t is a scalar that controls the write strength. For information that has already been remembered, the corresponding error is close to zero, so it is almost not written repeatedly, while new information is concentrated into the state update.
2.4 Illustrative Implementation
The following code only illustrates the recurrence process and does not represent the implementation of any specific system:
import numpy as npdef delta_linear_attention(keys, values, queries, beta=0.5):"""Maintain a fixed-size state matrix with the delta update rule and process the sequence step by step."""n, d = keys.shapestate = np.zeros((d, d))outputs = []for t in range(n):k = keys[t] / (np.linalg.norm(keys[t]) + 1e-6)v = values[t]prediction = state @ k # Prediction of the current state for this keystate = state + beta * np.outer(v - prediction, k) # Write only the prediction erroroutputs.append(state @ queries[t])return np.stack(outputs)n, d = 1024, 64rng = np.random.default_rng(0)K, V, Q = rng.normal(size=(3, n, d))out = delta_linear_attention(K, V, Q)print(out.shape) # (1024, 64)
2.5 Engineering Trade-offs
- The state size is decoupled from the sequence length, but information fidelity is limited by the feature dimension d.
- Hybrid structures are common in practice: some layers retain global attention to ensure precise retrieval, while the remaining layers use linear mechanisms to reduce cost.
- Training usually requires blockwise parallelism. Token-by-token serial recurrence will significantly slow down training.
3. Depth Dimension: Learnable Aggregation of Inter-Layer Information
3.1 Limitations of Standard Residuals
A standard residual connection can be written as:
h_{l+1} = h_l + F_l(h_l)
The outputs of all layers are accumulated with the same weight. As the number of layers increases, early information is diluted by later outputs, and the numerical magnitude differences between different layers also increase.
3.2 Layer-Weighted Aggregation
A general approach is to let each layer receive not only the output of the previous layer but also a weighted summary of the outputs of all previous layers:
hl = Σ{i<l} α_{l,i} · f_i
Here, the weights α_{l,i} are obtained by normalizing learnable parameters with softmax. The model can decide for itself which depths of features are more useful, instead of passively accepting fixed equal-weight accumulation.
3.3 Costs and Implementation Points
- Historical layer outputs need to be stored or accessed, so memory and communication overhead grow with the number of layers.
- Several layers can be grouped into a block, with fine-grained aggregation inside each block and coarse-grained aggregation between blocks, thereby controlling overhead.
- The additional overhead needs to be measured experimentally and evaluated together with the benefits in training stability and convergence speed.
Illustrative code:
import numpy as npdef depthwise_aggregate(layer_outputs, logits):"""Perform a softmax-weighted sum over the outputs of all previous layers.layer_outputs: historical layer outputs with shape (l, d)logits: learnable parameters with shape (l,)"""weights = np.exp(logits - logits.max())weights /= weights.sum()return weights @ layer_outputs
4. How to Evaluate Such Improvements
Evaluation should distinguish between efficiency metrics and quality metrics.
- Efficiency metrics: inference throughput, time to first token, and peak memory usage. They should be measured across different context-length buckets, and the shape of the curve should be observed rather than only a single point.
- Quality metrics: accuracy on long-document question answering, hit rate for long-distance information retrieval, and the curve of training loss versus compute.
Comparative experiments should control variables. Parameter count, training data, training steps, hardware, and numerical precision should remain consistent. Reporting only the speedup at one particular length can easily hide performance degradation in other scenarios.
5. Deployment Recommendations
- Locate the bottleneck first. If the main problem is high latency caused by long inputs, prioritize changes to the attention layers. If the main problem is unstable training in deep networks, prioritize checking the residual and normalization design.
- Evaluate with your own business data rather than relying only on general benchmarks.
- On the deployment side, long documents can be stored in object storage, and the inference service can work with monitoring and alerting to track latency and memory levels, making it easier to detect long-tail requests in time.
- For extremely long contexts, retrieval augmentation can first filter out relevant segments, reducing dependence on the model’s native length while also lowering inference cost.
