Training-Free Low-Bit Quantization for Recurrent States in Linear Attention

    This article examines how low-bit quantization can be applied to the fixed-size recurrent states used in linear attention and related state-space mechanisms. It explains why naive per-token quantization accumulates error, highlights the challenge of outlier rows and columns, and introduces a training-free approach based on window-wise quantization, a high-precision update buffer, compensation tokens, and residual smoothing. It also outlines an implementation flow, evaluation metrics, tuning considerations, and engineering boundaries for deploying low-bit recurrent states in long-context inference systems.

    1. Background: Why Recurrent States in Linear Attention Need Quantization

    As large language models increasingly adopt hybrid attention architectures, some layers begin to use linear attention or recurrent state-space mechanisms instead of standard attention. Standard attention typically needs to store an ever-growing key-value cache. As sequence length increases, cache footprint and attention computation continue to grow. Linear attention instead recursively updates a fixed-size state, compressing historical context into a limited number of dimensions.

    This design can significantly reduce storage requirements in long-context scenarios. For time step (t), the model usually computes a new state from the current input and the previous state:

    [
    St = F(S{t-1}, x_t)
    ]

    Here, (S_t) is a fixed-size recurrent state, (x_t) is the current token or its projected representation, and (F) represents the state update function. The output is generally determined jointly by the state and the current input:

    [
    y_t = G(S_t, x_t)
    ]

    From an algorithmic complexity perspective, a fixed-size state avoids cache access that grows with context length. On real hardware, however, the recurrent state can still become an inference bottleneck. On one hand, every token requires reading and writing the full state. On the other hand, state updates often involve many matrix-vector operations, creating high memory bandwidth pressure. As model size grows, concurrent requests increase, or context length extends, this repeated read-write pattern can limit overall throughput.

    Low-bit quantization is a natural way to reduce this cost. Compressing the state from a high-precision format to 8-bit integers or another low-precision representation can reduce state footprint and memory access and lower the cost of some arithmetic operations. Unlike ordinary weights, however, a recurrent state is updated at every time step. Quantization error can repeatedly enter the next round of computation and eventually become cumulative error.

    Therefore, the core of linear attention state quantization is not simply converting a floating-point state into a low-bit format. It must address three problems at the same time:

    1. How to reduce the frequent overhead of quantization and dequantization.
    2. How to suppress long-term accumulation of error in recursive updates.
    3. How to handle a small number of outlier rows or columns with very large magnitudes.

    2. Main Challenges in Recurrent State Quantization

    2.1 Per-Token Quantization Amplifies Quantization Error

    If a full state quantization step is performed at every time step, the process is roughly as follows:

    [
    S_{t-1}^{q} \rightarrow \text{dequantize} \rightarrow \text{update} \rightarrow \text{requantize}
    ]

    Here, (S^q) denotes the low-bit state. Even if a single quantization error is small, noticeable deviation can appear after repeated updates over a long sequence. In structures where state updates have cumulative behavior, early errors affect later states and can create a recursive amplification effect.

    This error may not immediately appear as a large change in output values, but it can degrade model quality on long contexts, complex reasoning, or out-of-distribution inputs.

    2.2 Outlier Rows and Columns May Exist in the State

    A recurrent state is usually not uniformly distributed. Some rows or columns may carry more important feature-transmission functions, and their value ranges may be much larger than those of other parts. If a single quantization scale is used directly, these outliers determine the overall scaling factor, leaving many ordinary elements with fewer effective quantization levels.

    Conversely, expanding the quantization range separately reduces the quantization precision of ordinary elements. In other words, outliers create a conflict between the dynamic range of quantization and the precision of the main data.

    2.3 Quantization Overhead May Offset Computational Gains

    A low-bit representation does not necessarily mean faster execution. If every token requires complex scale-factor computation, maximum-value reduction, format conversion, and error correction, the quantization process itself may become a new bottleneck.

    A state quantization scheme must therefore consider both numerical error and hardware execution efficiency. In particular, it should reduce full-state scans, improve memory access continuity, and fuse quantization, update, and output computation into as few kernels as possible.

    3. Core Approach: Window-Wise Quantization

    A more suitable approach for recurrent states is “window-wise quantization.” Instead of requantizing the state immediately after every token, consecutive tokens are divided into a window.

    Let the window length be (W), and let the initial state of the window be (S_0). Inside the window, the system maintains two pieces of information:

    • A fixed low-bit state (S_q);
    • A high-precision temporary update buffer (R).

    At the start of the window, the low-bit state is read and converted into the representation needed for computation. Then, the state changes produced by each token in the window are not immediately written back to the low-bit state. Instead, they accumulate in the high-precision buffer:

    [
    Ri = R{i-1} + \Delta S_i
    ]

    Here, (\Delta S_i) represents the state update produced by the (i)-th token. Output computation uses both the base value corresponding to the low-bit state and the high-precision buffer:

    [
    y_i = G(\text{dequant}(S_q) + R_i, x_i)
    ]

    Only at the end of the window are the base state and high-precision increment merged and quantized once:

    [
    S_{\text{new}} = Q(\text{dequant}(S_q) + R_W)
    ]

    This approach provides three direct benefits.

    First, the quantization frequency is reduced from once per token to once per window, reducing scale-factor computation, range statistics, and format conversion.

    Second, updates inside the window are performed in the high-precision buffer, avoiding reinjection of rounding error into the recurrent state at every step.

    Third, the window length can be adjusted according to hardware characteristics and model structure to balance precision and performance. If the window is too short, quantization frequency increases. If the window is too long, the high-precision buffer footprint grows, and a larger deviation may appear between the base low-bit state and the true state.

    4. Compensation Tokens: Handling a Small Number of Critical Outliers

    Window-wise quantization alone may still not fully solve the outlier problem. To address this, a small number of positions with large magnitudes can be identified in the state and retained as high-precision compensation tokens.

    Here, “compensation tokens” are not real tokens in the input sequence. They are auxiliary representations used to describe and correct outlier information in the state. The basic idea is not to require the entire state to use the same precision, but to compress most ordinary elements into a low-bit format while keeping the small portion most likely to cause error on a high-precision path.

    During state updates, compensation information should not be handled in isolation. It should share the same update flow as real tokens. This avoids two inconsistent state-evolution paths. Conceptually, the state can be represented as:

    [
    S \approx S_q + C
    ]

    Here, (S_q) is the low-bit main body, and (C) is the high-precision outlier portion expressed by a small number of compensation tokens. Each output computation uses both parts, and when quantization is performed at the end of a window, compensation positions are reselected or updated according to the new state distribution.

    This design offers good compression efficiency: the main data still benefits from low-bit storage and computation, while a small amount of high-precision information protects dimensions that are more sensitive to model quality. Compared with simply increasing the precision of the entire state, the compensation strategy can achieve better numerical stability at similar memory cost.

    5. Residual Smoothing: Reducing Dynamic Range Before Quantization

    Before quantization at the end of a window, the state residual can be smoothed. The goal of residual smoothing is not to eliminate all large values, but to reduce the influence of local extreme values on the overall scaling factor, allowing the quantizer to use the limited discrete representation space more effectively.

    Suppose the state consists of a main part and a residual part:

    [
    S = S_{\text{base}} + R
    ]

    Before quantization, the distribution of (R) can be made more concentrated by redistributing scaling factors, adjusting the representation range of different dimensions, or moving some extreme components to the compensation path. After smoothing, most elements can obtain finer quantization intervals, reducing average error.

    Note that smoothing must preserve computational equivalence or approximate equivalence. If only the state is modified without synchronously adjusting subsequent reading and update processes, a new systematic bias may be introduced. Therefore, smoothing, compensation, and quantization should be treated as a single integrated design rather than independent optimization steps.

    6. A Practical Execution Flow

    In an engineering implementation, one window processing pass can be abstracted into the following steps:

    1. Read the low-bit state saved at the start of the window.
    2. Restore the state to the computation format using the quantization scale.
    3. Initialize the high-precision update buffer.
    4. Perform state updates token by token.
    5. Compute output using the base state and the current buffer.
    6. Collect state range statistics at the end of the window.
    7. Identify outlier positions that need protection.
    8. Apply smoothing to the residual.
    9. Compress the main state into a low-bit representation.
    10. Save the low-bit state, scaling information, and compensation information.

    Illustrative pseudocode is shown below:

    1. state_q, scale, compensation = load_state()
    2. base_state = dequantize(state_q, scale)
    3. delta = zeros_like_high_precision(base_state)
    4. for token in token_window:
    5. update = recurrent_update(base_state, delta, token)
    6. delta = delta + update
    7. effective_state = base_state + delta
    8. output = compute_output(effective_state, token)
    9. emit(output)
    10. merged_state = base_state + delta
    11. smoothed_state, residual = smooth(merged_state)
    12. state_q, scale = quantize(smoothed_state)
    13. compensation = extract_outliers(residual)
    14. save_state(state_q, scale, compensation)

    In actual kernels, explicit creation of too many intermediate tensors should be avoided. State reading, updates, output computation, and some reduction operations should be fused as much as possible. For parallel hardware such as GPUs, attention should also be paid to thread-block partitioning, shared-memory usage, global-memory access coalescing, and the decoding cost of low-bit data.

    7. Which Metrics Matter in Performance Evaluation

    When evaluating a recurrent state quantization scheme, single-operation arithmetic throughput is not enough. The following metrics should also be examined.

    7.1 Numerical Quality

    Compare against a high-precision baseline and observe perplexity, task accuracy, long-context consistency, and generation stability. Long sequences should be tested separately, because short sequences may not yet expose recursive error accumulation.

    7.2 State Memory Footprint

    Account for the total footprint of the state main body, scaling factors, compensation information, and temporary buffers. Do not calculate only the theoretical size of the low-bit main body. If compensation information is excessive, the actual compression benefit may be lower than expected.

    7.3 Kernel-Level Throughput

    Kernel-level metrics can reflect the efficiency of state updates, quantization, dequantization, and output computation themselves. Distinguish between compute-bound and bandwidth-bound scenarios, and record changes under different window lengths.

    7.4 End-to-End Latency

    End-to-end performance is also affected by scheduling, batching, input-output preparation, sampling, and execution time of other network layers. Therefore, kernel speedups do not necessarily translate proportionally into overall inference speedups. In actual deployment, more attention should be paid to first-token latency, sustained generation speed, and concurrent throughput.

    8. Tuning Recommendations for Window Length and Compensation Scale

    Window length is a key parameter affecting both precision and performance. Smaller windows correct the state more frequently and are generally beneficial for precision, but they increase the number of quantization operations. Larger windows reduce quantization overhead, but may increase high-precision buffer requirements and enlarge the deviation between the base state and the true state.

    The number of compensation tokens also needs to be controlled. If the compensation scale is too small, important outlier information cannot be covered. If it is too large, the memory and computational benefits of the low-bit representation are weakened. In practice, the choice can be based on state magnitude distribution, outlier stability, and changes in task accuracy rather than mechanically fixing a ratio for all models.

    In addition, state distributions may differ significantly across layers. A more reasonable strategy is to allow quantization ranges, window policies, and compensation scales to be set per layer or per state group, while preventing excessive metadata from increasing management cost.

    9. Applicability Boundaries and Engineering Considerations

    This method is suitable for linear attention or similar structures that have a fixed-size recurrent state and whose state is continuously read and updated. For fully static weights, traditional weight quantization methods may be more direct. For structures that need to preserve all historical information precisely, recurrent state quantization logic cannot simply be applied.

    Engineering implementation should also consider the following issues:

    • Quantization scales and compensation metadata need to be stored stably and kept consistent with the state lifecycle.
    • State reset, request switching, and batch merging must avoid state contamination across different requests.
    • In mixed-precision computation, it should be clear which steps use low precision and which steps must retain high precision.
    • Stress-test extreme inputs and unusually long contexts rather than validating only on average samples.
    • During evaluation, record precision, memory, bandwidth, latency, and throughput together to establish a complete benefit feedback loop.

    10. Conclusion

    For low-bit quantization of recurrent states in linear attention, the key is not merely reducing representation bit width, but redesigning the state update process. Window-wise quantization reduces the overhead and error injection caused by frequent quantization. The high-precision update buffer protects recursive computation inside the window. Compensation tokens mitigate the damage that outlier rows and columns cause to the overall dynamic range. Residual smoothing further improves the data distribution before quantization.

    Together, these mechanisms form a training-free path for state quantization: without modifying the model training process or relying on additional fine-tuning, the model can maintain quality close to a high-precision baseline at lower state precision. For long-context inference systems, the value of this method lies not only in memory savings, but also in reducing state memory-access pressure, lowering quantization management overhead, and providing a general approach for efficient deployment of recursive large-model structures.