Low-Bit Quantization of Linear Attention Recurrent States: A Practical Guide to Efficient Inference
Linear attention compresses long-context history into a fixed-size recurrent state, which shifts the decoding bottleneck from memory capacity to state access. This guide explains why naively quantizing that recurrent state to low bit-widths (for example, 8-bit) is difficult—covering cross-step error accumulation, outlier rows and columns that degrade quantization resolution, and why a training-free approach matters. It then describes three complementary training-free designs: per-window quantization, compensation tokens, and pre-quantization residual smoothing. Finally, it walks through how to account for memory overhead, how to interpret accuracy and speed results, and which deployment scenarios are (and are not) a good fit.
1. Background: Linear Attention Compresses Long Context into a Fixed-Size State
Standard attention must store and repeatedly access a key-value cache during decoding, and its memory footprint typically grows with context length. The longer the context, the more pressure this puts on memory capacity and memory bandwidth. To ease this problem, linear attention is often introduced into hybrid architectures—for example, gated delta networks or delta attention–style structures. What they share is this: instead of keeping a key-value cache that accumulates token by token, they compress historical information into a fixed-size recurrent state matrix.
In a single-head sketch, the recurrent update can be abstracted as:
state = a * (state @ tri) + b * outer(k, v)o = state.T @ q
Here, state is the fixed-size recurrent state, k and v are the key and value vectors of the current token, a is the decay coefficient, b is the write strength, tri is a projection transform, and q is the current query vector. Different linear attention variants differ in their details, but they broadly follow the same recurrence: decay the history, write in new information, and read out through the query.
The benefit of this design is that memory usage no longer grows linearly with context length. But the problem does not disappear—it simply moves from “storing a large key-value cache” to “repeatedly reading and writing a fixed-size recurrent state.” During decoding, for every token generated, the model may read the state, perform matrix operations, and write back the updated state. When the state is large relative to the amount of computation per step, decoding speed becomes bounded more by memory bandwidth than by peak compute. This is why low-bit quantization of the recurrent state has become an important direction for reducing memory-access cost.
2. Why Is Directly Quantizing the Recurrent State to Low Bit-Widths Hard?
Compressing the recurrent state from floating point down to a low bit-width, such as 8-bit, looks straightforward: storage and transfer costs drop, and reads, updates, and write-backs may all speed up. But in practice, three classes of problems arise.
2.1 Do Rounding Errors Accumulate Across Steps?
In standard attention, historical keys and values are mostly read “on demand” by queries, so the effect of a single quantization error is usually local. The recurrent state is different: every update feeds existing error into the next step of the recurrence.
If we denote the true state as S_t and the quantization perturbation as δ_t, the update relation can be written as:
S_t = F(S_{t-1}) + U_t
Early errors can then propagate to the current state through the chained action of F. When the decay coefficient is close to 1 and the state serves as long-term memory, the error decays slowly. The result: a quantization error that looks tiny in a single step can gradually accumulate in the later part of a long decode, slowly degrading generation quality.
So quantizing a recurrent state cannot be judged by single-step precision alone—you also have to ask whether the error is amplified along the recurrence chain.
2.2 Do Outlier Rows and Columns Drag Down Quantization Resolution?
A recurrent state is typically built up by accumulating many outer products, so some rows or columns may correspond to particularly prominent dimensions. These positions are like the outlier channels in large models: their magnitudes are clearly higher than ordinary positions.
With channel-wise quantization, if a channel contains outliers, the quantization step size is often stretched by the maximum absolute value. Once the step size grows, ordinary values fall into very few quantization levels, and resolution drops. The symptom: most of the state can still be represented, but fine distinctions are smoothed away.
So outliers are not simply a matter of “slightly larger error”—they undermine the efficiency of quantizing the entire group.
2.3 Why Does Training-Free Matter?
Post-training quantization is mature, but it usually requires steps such as calibration data, layer-wise sensitivity analysis, and learning scale factors. The values of the recurrent state depend on runtime inputs and context, which are hard to cover fully through static calibration. Therefore, achieving near-lossless 8-bit quantization without training and without introducing learnable parameters demands mechanism design rather than reliance on extra training.
3. A Training-Free Low-Bit Quantization Scheme: Three Key Designs
For the linear attention recurrent state, one class of training-free quantization schemes consists of three parts: per-window quantization, compensation tokens, and pre-quantization residual smoothing.
They solve different problems:
| Problem | Main mechanism | Role |
|---|---|---|
| Error accumulates across steps | Per-window quantization | Confines quantization to the end of a window |
| Outliers drag down resolution | Compensation tokens | Reserve channels for the few high-precision outliers |
| Single-step quantization error is large | Residual smoothing | Compress dynamic range, improve level utilization |
3.1 Per-Window Quantization: Let Errors Accumulate Only Within a Window
A naive approach quantizes the state before each write and dequantizes it at the next read. This introduces quantization error at every step, and the error propagates along the recurrence chain.
The idea behind per-window quantization is to split decoding into windows of W tokens. The state is quantized only once at the end of a window; inside the window it is not quantized step by step. Instead, a high-precision buffer absorbs the incremental updates.
Illustrative logic:
def recurrent_window_step(state_low, buffer_hi, tok, params):# state_low: the low-bit state quantized at the end of the previous window# buffer_hi: the high-precision incremental buffer accumulated within the current windowdelta = params.b * outer(tok.k, tok.v) - state_low @ params.projbuffer_hi = params.a * buffer_hi + deltareturn state_low + buffer_hidef window_commit(state_low, buffer_hi):return quantize_8bit(state_low + buffer_hi)
Within a window, the low-bit state acts as a fixed baseline for output computation while the high-precision buffer carries the step-by-step updates; at the end of the window, the high-precision increment is folded into the state and quantized in one shot. This way, quantization error is no longer produced at every step but only once, at the window boundary.
This design yields two kinds of benefits:
- In terms of accuracy, the range over which error accumulates is bounded from the entire decode length down to within a window of length
W. - In terms of systems, the number of quantize and dequantize calls drops from
O(T)toO(T/W), whereTis the number of decode steps. Fewer calls also reduce extra read/writes, kernel launches, and scheduling overhead.
3.2 Compensation Tokens: Reserve a High-Precision Channel for Outliers
Even when quantization is less frequent, outlier rows or columns still affect the scaling of the whole group at each quantization. So the most prominent outliers need to be handled separately before quantization.
The basic idea of compensation tokens:
- Identify the few largest-magnitude elements in the state, or the small number of highest-magnitude rows or columns.
- Keep this content as high-precision compensation terms.
- Quantize the remaining residual at 8-bit.
It is important to be clear about what is being compensated. In engineering, you can first gather statistics and then choose one of three granularities:
| Compensation granularity | Meaning | Advantage | Cost |
|---|---|---|---|
| Element-level | Keep the few largest-absolute-value elements in the state matrix | Fine-grained, flexible | May require indices or masks |
| Row-level | Keep a small number of high-magnitude rows | Corresponds to certain key dimensions | May keep too much redundancy |
| Column-level | Keep a small number of high-magnitude columns | Corresponds to certain value dimensions | May not cover fully |
The number of compensated items can be set as a fraction of the total state elements, for example η = N_comp / E, or set as a per-row/per-column budget. If the compensation positions are determined by a static rule or a fixed pattern, no extra indices are needed; if the positions are selected dynamically, then indices, bitmaps, or compressed metadata must be recorded. During evaluation, this metadata should be counted toward memory and memory-access cost.
The key point is that compensation tokens should not become additional storage floating free of the model’s computation. As far as possible, they should share the same update path as the real tokens and take part in the same decay, write, and read-out operations. This keeps the recurrence semantics consistent and makes operator implementation easier. The implementation boundary can be summarized as: outside the quantization path, keep a high-precision accumulation or high-precision additive term, but the update, read-out, and fusion still reuse the existing recurrent kernel.
3.3 Pre-Quantization Residual Smoothing: Compress the Dynamic Range Before Quantizing
After compensation tokens remove the largest outliers, the residual may still contain second-largest values and local fluctuations. At this point, smoothing can be applied before quantization to make the dynamic range more even.
A reproducible version looks like this:
# residual: the residual after removing the compensation terms# group: channel grouping, e.g., one group per 64 or 128 elements# alpha: smoothing strength, in [0, 1], commonly 0.5scale_g = max(eps, group_max_abs(residual_g).pow(alpha))smoothed_g = residual_g / scale_gcodes_g = quantize_8bit(smoothed_g)# To reconstruct:reconstructed_g = dequantize_8bit(codes_g) * scale_g
Here, alpha = 0 means no additional smoothing, and the larger alpha is, the stronger the compression of the group’s maximum magnitude. In practice, it is advisable to add eps to prevent division by zero and to set upper and lower bounds on scale_g, avoiding numerical anomalies caused by extremely small or large scales.
Grouping can be done by channel, row, or column. Common group sizes can be 32, 64, or 128. Each group stores one scale factor, typically in FP16 or FP32. The scale factors themselves add storage overhead, so this must be included in the memory accounting. This method requires no training and no learned parameters, so it is compatible with the training-free constraint.
3.4 How Do the Three Work Together?
These three mechanisms form a layered filtering pipeline:
- Residual smoothing first compresses the dynamic range, reducing ordinary quantization error.
- Compensation tokens peel off the most extreme outliers, avoiding a stretched quantization step for the whole group.
- Per-window quantization concentrates the quantization operation at the end of the window, reducing cross-step error accumulation.
Using any one of them alone cannot simultaneously solve the three problems of error accumulation, outliers, and single-step quantization error. The point of the combined design is precisely this.
4. How Do You Account for Memory and Benefits?
The benefit of low-bit state quantization is often summarized as “less memory, fewer memory accesses.” But the actual footprint cannot be judged by the state alone—you must also account for the high-precision buffer, scale factors, indices, and temporary workspace.
A simple estimation formula can be built:
M_total ≈ E × (b_state + b_buffer + b_meta) + M_temp
Where:
E: total number of state elements.b_state: bytes per element of the low-bit state. Typically 1 byte for 8-bit.b_buffer: bytes per element of the high-precision buffer. If the buffer is the same shape as the state and uses FP16, this is 2 bytes; if only part of the window increment is kept, scale by the actual proportion.b_meta: average bytes per element of metadata such as scale factors, compensation indices, and masks.M_temp: overhead of the operator’s temporary workspace.
For example, if the implementation uses a full-size FP16 buffer and the state is 8-bit, then it is at least about 1 + 2 = 3 bytes per element. Relative to an FP32 baseline of 4 bytes per element, that is about 75%; relative to a pure 8-bit state of 1 byte per element, it is 3×. So you cannot simply say “with a buffer it is still only 1.5× a pure 8-bit state.” If the buffer is sparse, blocked, or covers only the within-window increment, you must recompute based on the actual buffer proportion.
Likewise, quantization and dequantization themselves create extra memory access. If the window is too short, quantization calls become too frequent and the extra read/writes can offset part of the benefit; if the window is too long, the high-precision buffer takes up more space and error is retained longer. So the window length and buffer strategy must be tuned together.
5. How Do You Interpret Experimental Results?
The direction of the conclusions from existing material is this: under 8-bit recurrent-state quantization, model accuracy can approach the FP32 baseline; the relevant operators achieve an average speedup of roughly 2.05× to 3.70×; and overall inference averages about a 1.47× speedup. These numbers show that low-bit state quantization has clear optimization potential, but they are not sufficient as a production guarantee—the experimental scope must be filled in.
5.1 What Information Do Accuracy Conclusions Need?
If you claim “near-lossless,” you should at least state:
| Item | What to record |
|---|---|
| Model family | Specific model name, number of layers, state dimension, whether standard attention is mixed in |
| Evaluation dataset | Long-document summarization, long-document QA, code, math, or other tasks |
| Context length | For example 4K, 8K, 32K, or longer |
| Generation length | Number of output tokens, because longer generation is more likely to expose error accumulation |
| Evaluation metric | Accuracy, F1, ROUGE, perplexity, task pass rate, and so on |
| Baseline accuracy | FP16, BF16, or FP32 |
| Scope of applicability | Whether it applies only to low concurrency, long context, or a specific model structure |
If the original material does not disclose the specific model family and dataset, then “near-lossless” can only be treated as a reference conclusion, not a universal guarantee. Before a production choice, you should at least re-test against the target model, the target business data, and the target context length.
5.2 What Information Do Speedup Conclusions Need?
Operator-level and end-to-end speedups also need a clear measurement scope:
| Item | What to record |
|---|---|
| Hardware model | GPU model, memory bandwidth, compute platform |
| Batch size | Batch size, especially distinguishing low concurrency from high concurrency |
| Sequence length | Input length, context length, state size |
| Window length | The value of W, and whether a grid search was done |
| Compensation budget | Number of compensated elements, rows, or columns |
| Baseline accuracy | Compared against FP16, BF16, or FP32 |
| Whether quantization cost is included | Whether quantization, dequantization, and scale-factor reads/writes are counted |
| Measurement method | Operator-level timing, end-to-end latency, throughput, P50/P99 |
A kernel-level speedup being higher than an end-to-end speedup is not surprising. The recurrent-state update is only part of the decoding flow; before and after it there are token embeddings, preprocessing, other layer computations, sampling, and output handling. By Amdahl’s law, the end-to-end benefit depends on the share of total time taken by the state update. The longer the context, the lower the concurrency, and the larger the state, the more likely state memory access is to be the dominant cost—and the more pronounced the benefit of low-bit quantization.
6. Practical Advice: How to Choose, Tune, and Validate?
6.1 When Is a Low-Bit State Worth Using?
A low-bit recurrent state is best suited to bandwidth-bound decoding scenarios, such as small-batch, long-context, low-concurrency online serving. In such scenarios, state reads and writes are more likely to become the bottleneck, and lowering the bit-width directly reduces memory access. In high-concurrency, compute-saturated offline batch scenarios, the benefit may be limited; it is advisable to profile first to confirm whether the state update is the main time cost.
6.2 How Should You Choose the Window Length?
The window length needs to be tuned against both the memory budget and accuracy requirements. A longer window means fewer quantizations and more obvious memory-access savings, but a larger high-precision buffer footprint and longer error-retention time per quantization. A shorter window constrains error more easily, but increases quantize/dequantize calls and raises extra overhead.
It is advisable to run a small-scale grid test, for example comparing values such as W = 16, 32, 64, 128, and to record accuracy, memory, latency, and throughput.
6.3 How Many Compensation Tokens Should You Set?
The compensation count should be determined per model and per layer by statistics, rather than decided uniformly. Different layers and different state matrices differ in their sensitivity to outliers. You can first tally the distribution of maximum absolute values in the state, observe the proportion of outlier elements, and then set a compensation budget. Too few compensation tokens leave outliers dragging down quantization; too many erode the benefit through high-precision storage and index overhead.
6.4 How Should You Validate for Long Contexts?
The error accumulation problem usually shows up more easily in the later part of long sequences. Validation should not look only at average metrics on short prompts; it should cover tasks such as long-document summarization, long-document QA, and long-code comprehension, with particular attention to the quality of the later part of generation. It is advisable to record metric changes across different generation lengths, rather than giving only one overall average.
6.5 How Do You Avoid Common Mistakes?
Common mistakes include:
- Testing only short contexts and ignoring error accumulation in the later part of long sequences.
- Comparing only operator speed without accounting for quantization, rescaling, and metadata overhead.
- Setting only the window length without evaluating high-precision buffer usage.
- Setting the compensation token count by experience, without tallying the outlier distribution.
- Directly comparing speedup ratios across hardware while ignoring differences in bandwidth and state size.
- Generalizing “near-lossless” to all models, all tasks, and all hardware platforms.
6.6 What Should You Note for Cloud Deployment?
If deploying to the cloud, you can round out the pipeline with general cloud capabilities: use a container platform to host elastic inference services, use object storage to keep model weights and evaluation benchmark data, use monitoring and alerting to track latency, throughput, and memory watermarks, and use a logging service to record quality regression results under different quantization configurations. This helps keep performance stable after launch and makes problems traceable.
7. Summary: Suitable and Unsuitable Scenarios
Linear attention trades long-context capability for a fixed-size recurrent state, but it also shifts the inference bottleneck from “memory capacity” to “state access.” Low-bit quantization aimed at the recurrent state must answer three questions at once:
- How does error avoid accumulating across steps? Through per-window quantization, confining quantization to the end of the window.
- How do outliers avoid dragging down accuracy? Through compensation tokens, reserving a high-precision path for the few outliers.
- How do you make each quantization error smaller? Through pre-quantization residual smoothing, compressing the dynamic range and improving level utilization.
Under a training-free premise, this class of scheme can approach floating-point baseline accuracy at 8-bit and achieve clear speedups in bandwidth-bound decoding scenarios. But it is not unconditionally applicable.
Scenarios better suited include:
- Low-concurrency, long-context online inference.
- Decoding tasks with a large state where the memory-access bottleneck is evident.
- Model services constrained by memory bandwidth that still want to keep a long context.
- Teams able to accept the tuning cost of window length, compensation budget, and validation.
Scenarios that require caution include:
- High-concurrency, compute-saturated offline batch processing.
- Business with extreme accuracy sensitivity that cannot be re-tested.
- Extremely tight memory budgets that cannot absorb high-precision buffers or metadata overhead.
- Scenarios that go straight to production without long-context validation.
For teams evaluating linear attention hybrid architectures while constrained by decoding memory-access cost, low-bit recurrent-state quantization is an optimization path with relatively low investment and clear returns. But the prerequisite is: first define the measurement scope, then do the memory accounting, and finally validate accuracy and speed against the target scenario.
