Adapting Sparse Attention Inference on Domestic AI Accelerators: A Practical Guide
This article presents practical guidance for adapting sparse attention and hybrid attention inference on domestic AI accelerators. It explains the challenges introduced by mixed architectures (sparse attention, gated linear attention, cross-layer residuals, and visual encoders), the responsibilities of each layer in the software stack (operator library, kernel programming, compiler, and inference framework), and the full inference execution pipeline. It then details operator adaptation for index selection and sparse attention, differentiated Prefill/Decode optimization, linear attention state management, and the joint handling of cross-layer residuals and visual attention. It closes with a three-level validation and troubleshooting strategy. Engineers working on inference adaptation for domestic accelerators will find concrete adaptation points, layout decisions, and pitfalls to check.
1. Background: Adaptation Challenges from Hybrid Attention Architectures
Large models are shifting from a single structure where every layer uses standard multi-head attention toward new architectures that mix multiple attention mechanisms. Typical combinations include:
- Sparse attention: selects only a few key positions per query to participate in computation, reducing the attention overhead of long contexts;
- Gated linear attention: replaces the KV cache that grows with sequence length with a recurrent matrix state, improving long-sequence decoding efficiency;
- Skip-style cross-layer residuals: introduces cross-layer connections on top of the per-layer residual;
- Visual encoder: provides visual tokens for multimodal capability.
When our team takes on such models on domestic AI accelerators (referred to below as “domestic cards”), the upstream model code is implemented based on an open-source inference framework. Adaptation requires the simultaneous use of a general operator library, a Triton-like kernel programming language, a TileLang-like block-level programming framework, and a self-developed compiler, so that community model code can ultimately serve inference externally on the domestic cards.
Compared with a conventional dense Transformer, the adaptation difficulties of such architectures concentrate in three areas:
- Irregular access patterns: sparse attention relies on index selection results, changing memory access from regular sequential reads to indirect gather;
- State requires incremental maintenance: linear attention layers carry state that is reused across tokens, placing high demands on read/write order and rollback capability;
- The execution path spans multiple subsystems: the text backbone, visual encoding, and multiple layer types interleave, and a mismatch at any point can block the entire path.
A more realistic constraint is that domestic cards differ from mainstream GPUs in architecture, memory hierarchy, and software ecosystem: some operators lack highly tuned ready-made implementations, support for some programming features is incomplete, and handling of dynamic shapes relies more heavily on the compiler and scheduling layer. Therefore, adaptation cannot stop at “swapping one set of function calls”; it must span three levels: operators, memory layout, and scheduling.
2. The Layered Software Stack: Responsibility Boundaries of Operator Library, Kernel Programming, and Compiler
Adaptation on domestic cards is usually a layered collaboration, with each layer taking on different responsibilities:
| Layer | Typical components | Main responsibility | Applicable scenarios |
|---|---|---|---|
| Operator library layer | General high-performance operator collection | Provides tuned operators such as GEMM, RMSNorm, RoPE, and sampling | Standard compute modules of the model |
| Kernel programming layer | Triton-like kernel language, TileLang-like block-level framework | Expresses block-level parallelism and data movement with relatively concise syntax to quickly validate new algorithms | New operators such as index scoring, sparse attention, and state updates |
| Compiler layer | Self-developed compiler targeting domestic cards | Handles graph-level optimization, operator fusion, scheduling, and memory planning to generate target instructions | Overall inference graph optimization and kernel landing |
| Framework layer | Domestic-card version of the inference framework | Handles model definition, scheduling, batching, cache management, and precision alignment | Hosting upstream model code and building services |
They are complementary rather than substitutes:
- Operator library wins on maturity and stability, but its coverage boundary is determined by official implementations;
- Kernel programming wins on expressive flexibility, but hand-written kernels tend to get bogged down in detail-level tuning under complex memory access;
- Compiler can automatically perform fusion and scheduling, but only on the premise that the model’s operators can be expressed as a graph structure the compiler can recognize;
- Inference framework provides engineering capabilities such as continuous batching, paged cache, and scheduling policies, but it defaults to adapting dense attention and needs extension when encountering new operators.
A common misjudgment in practice is to expect a single layer to do everything: swapping only the operator library leaves new operators with nowhere to land; writing only kernels lacks scheduling and batching capabilities. A reasonable approach is to choose tools by module: leave standard modules to the operator library, use kernel programming to quickly validate new algorithms, leave overall optimization to the compiler, and let the inference framework provide engineering capabilities.
Built on the Baige platform, Baidu AI Cloud has formed full-stack AI capabilities covering heterogeneous compute, distributed training, and model inference. Model inference adaptation on domestic chips likewise depends on the collaborative refinement of each layer: chip, compiler, operator library, and framework.
3. Inference Execution Pipeline Overview
Before adaptation, the complete execution pipeline needs to be reconstructed. Taking multimodal input as an example, a typical data flow is:
Visual input ──> Visual encoder (image-block attention) ──> Visual tokens ─┐├─> Text backbone (hybrid attention × N layers) ──> Output headText input ──> Word embedding ───────────────────────────────────────────┘
Within the text backbone, layer types are usually arranged in an interleaved manner at a fixed ratio (e.g., in the Kimi linear architecture, 3 of every 4 layers are linear attention), roughly divided into three kinds:
- Sparse attention layers: on the input side, a lightweight indexer first selects candidate key positions for each query, then the sparse attention operator aggregates only over these positions;
- Linear attention layers: maintain a matrix state online via gated incremental rules, not depending on the full historical KV during decoding;
- Other standard attention layers: if the model retains such layers, their KV cache grows linearly with context length and is a key variable for memory planning.
Between layers, cross-layer residuals may be added on top of the conventional per-layer residual, so that the output of some layers participates in deeper computation via an additional path. The overall execution pipeline can be summarized as:
Visual attention → Text embedding → Hybrid attention layer stack (index selection + sparse attention / linear attention state update) → Cross-layer residual → Output head and sampling
During adaptation, each layer type corresponds to different resource constraints:
| Layer type | Core resource | Main bottleneck | Key adaptation point |
|---|---|---|---|
| Sparse attention | Index + KV cache | Indirect access, long-context bandwidth | Indexer incremental computation, paged-cache indirect indexing |
| Linear attention | Matrix state (one per head, fixed size) | Decoding access, state read/write order | In-place state update, prefix reuse, rollback |
| Visual attention | Intermediate activations (vary with image resolution) | Dynamic shapes | Sequence packing, non-causal masking, resolution bucketing |
4. Adapting the Index Selection and Sparse Attention Operators
The core of sparse attention is a “select-then-compute” two-stage structure; DSA-type mechanisms are a typical representative.
The selection stage is completed by a lightweight indexer: it computes relevance scores using the query and a (low-dimensional or quantized) key, selecting the top-k positions for each query head. The compute stage performs attention aggregation only over these positions. Bringing this structure onto domestic cards requires handling four categories of problems.
4.1 Incremental Computation in the Selection Stage
In the Prefill stage, all historical tokens must be scored at once, which is essentially a length × length score matrix whose compute cost grows quadratically with context length. To reduce overhead, the industry commonly uses two techniques:
- Compress the key into a low-dimensional representation, or store and compute in low precision (e.g., FP8);
- Maintain each query’s top-k in a streaming, block-wise manner on-chip, avoiding explicitly writing out the full score matrix.
In the Decode stage, a new token only needs to score its own query; the scores and selection results for historical tokens can be reused. It is recommended to explicitly cache the selection results in the per-sequence metadata, for the sparse attention kernel to read directly.
A detail that is easy to overlook: the top-k operation needs to maintain an ordered structure of size k on-chip, and when k is large, the sorting overhead is non-negligible. You can have the indexer output “block-level” rather than “token-level” results, supplemented by a fixed local window. This both reduces the selection granularity and guarantees that neighboring tokens are always visible.
4.2 KV Cache Layout and Indirect Indexing
Paged KV cache splits each sequence’s KV into fixed-size blocks, stored in a global memory pool, with a block table recording the mapping from logical to physical blocks. On top of this, sparse attention adds two more levels of indirection:
- First locate the logical position according to the selection results;
- Then look up the block table to convert to a physical address.
In kernel implementation, the “page number conversion + offset calculation + data read” should be completed within the same loop iteration as much as possible, avoiding extra global memory access for intermediate indices. There are also three constraints:
- Page size: should match the sparse kernel’s block size and storage alignment requirements;
- Shared KV: when multiple query heads share the same set of KV, index results can be reused within the group, reducing the number of indirect accesses;
- Low-precision KV: if a low-precision KV cache is used, it must be clear where quantization/dequantization occurs. On-chip dequantization is usually better than writing back to global memory.
4.3 Selection Strategy and Quantization Precision
Two design tendencies need to be traded off:
- Independent selection per query head: more precise expression, but requires scoring and gathering separately for each head, multiplying the number of kernels;
- Group-shared selection: significantly lower overhead, at the cost of somewhat reduced expressiveness.
The final choice should be decided by offline precision evaluation rather than ad-hoc trade-offs on the inference side. The same holds for quantization error: low-precision scoring changes the ordering boundaries of top-k, so end-to-end validation with a quantization strategy consistent with production must be done before launch.
4.4 Coordination with Page Tables and Speculative Decoding
Two alignment points deserve specific checking.
One is alignment with the local window. Most sparse schemes add a segment that is always visible beyond top-k, and its length should align with the paging block size. Otherwise, a cross-page window needs extra handling, and paging kernels are prone to out-of-bounds or duplicate reads.
The other is coordination with speculative decoding. One draft verification involves multiple candidate tokens whose indices fall at different logical positions, so the kernel needs to support a batched gather form where “multiple queries share one KV read”. Meanwhile, the linear attention layer’s state has branches, and when a candidate is rejected the state must roll back to the branch point. If the rollback is out of sync with index gathering, it will contaminate subsequent decoding.
The following is illustrative pseudocode showing the basic form of “coordinating selection results with the paged cache”:
# Illustrative pseudocode: only shows data flow and control logic, not a real APIdef sparse_decode_step(seq_states, q_proj, indexer, attn_kernel):# 1. Indexer scores and selects candidate positions for the new token (incremental, current token only)scores = indexer(q_proj, seq_states.key_cache)topk = select_topk_with_local_window(scores, k=2048, window=512)# 2. Logical position -> physical block: indirect addressing via the block table, done inside the kernelpages = seq_states.page_table[topk // PAGE_SIZE]offsets = topk % PAGE_SIZE# 3. Perform gather + attention aggregation only over selected pages, then write results back to outputout = attn_kernel(q_proj, seq_states.kv_pool, pages, offsets)return out
5. Differentiated Adaptation for Prefill and Decode
The two compute paths of sparse attention have sharply different characteristics, and it is hard to reach optimal performance for both with the same set of kernels.
| Dimension | Prefill | Decode |
|---|---|---|
| Number of queries | Whole prompt processed at once, length up to thousands | 1 per sequence per step (several candidates with speculative decoding) |
| Compute characteristic | Compute-intensive, dominated by large matrix multiplies | Memory-intensive, dominated by small matrix/vector operations |
| Selection stage overhead | Grows quadratically with length; needs a compressed representation or block-wise streaming top-k | Only scores the new token and reuses historical results |
| Main bottleneck | Matrix-multiply utilization, softmax reduction | KV bandwidth, kernel launch and scheduling overhead |
| Optimization techniques | Block-level sparsity, blocked softmax, sequence parallelism | Merging small operators, graph capture, fixed-shape batching |
The difficulty of the Decode stage lies in “few queries, much indirect access”. Each step has little computation, so kernel launch overhead takes a high proportion, and sparse gather further lengthens the critical path. There are three common optimizations:
- Merge index scoring, top-k, gather, softmax, and output projection into a few fused kernels;
- Capture and reuse the computation graph for fixed shapes to reduce launch cost;
- Group by “similar sequence length” at the batching level to reduce padding waste.
The Prefill stage cares more about compute utilization. When the indexer and sparse attention share the same key representation, the former can output low-dimensional quantized keys while the latter reads the original keys, with each hitting different compute characteristics.
6. Linear Attention State Management
The core of gated linear attention (represented by KDA-type mechanisms) is a per-head maintained matrix state whose update follows the rule of “gated decay + incremental correction”:
- Apply per-element or low-rank decay to the state according to the gate, to suppress old information;
- Read the old value in the state using the current key;
- Compute the difference between the new and old values, then apply an outer-product-style correction according to the key and write strength.
This update is naturally suited to single-step recursion in the Decode stage: each step reads and writes the state only once, without scanning history. The real difficulties concentrate in three points.
One is cross-step dependency. If the Prefill stage recurses directly in series, the length dimension cannot be parallelized, so it needs to be rewritten into a block-parallel form:
- compute within a block in causal order;
- connect between blocks using matrix operations that can be mapped to the matrix-multiply unit.
This rewriting places certain demands on numerical stability, especially for the step involving inversion of an in-block lower-triangular matrix. On domestic cards, the implementation quality of such operators directly determines Prefill performance.
The second is unified management with the KV cache. In hybrid architectures, linear attention layers do not occupy the KV cache but still need to maintain state per sequence; the size relates to the number of heads and head dimension, and is independent of context length. A PagedAttention-style cache is designed only for KV, so state needs independent paging/pooling management, and must be released or reclaimed in sync with KV at request end, preemption, and recovery. In multimodal scenarios, the insertion and truncation of visual tokens also require state to be replayable by position; otherwise the attention semantics of subsequent tokens will be wrong.
The third is rollback and multiple branches. Scenarios like speculative decoding and beam search require state snapshots and rollback. Since the state is updated primarily in place, it is recommended to use a “copy-on-write” or “branch buffer” approach: commit the state only after a candidate is confirmed accepted, avoiding erroneous updates contaminating subsequent computation.
Pseudocode example:
# Illustrative pseudocode: single-step gated linear attention state update (KDA-type rule)def kda_step(state, k_t, v_t, gate_t, beta_t):# 1. Gated decay: per-element/low-rank suppression of the state matrixstate = decay(state, gate_t)# 2. Incremental correction: update in "read-correct-write" orderold_v = state @ k_tdelta = (v_t - old_v) * beta_tstate = state + outer(k_t, delta)return state
7. Joint Handling of Cross-Layer Residuals and Visual Attention
Cross-layer residuals make inference no longer strictly layer-serial: the output of one layer can participate in deeper computation via an additional path, placing two demands on cache hits and operator fusion.
- The boundary of cache reuse must be redrawn. If intermediate activations are cached by layer, cross-layer dependencies mean the same activation may be referenced by multiple paths, and simply releasing by layer will err. It is recommended to explicitly annotate the lifetime of each intermediate quantity and make reuse decisions within the same forward graph.
- Fusion scope must span layers. Emitting operators layer by layer amplifies kernel launch and synchronization overhead, so the compiler needs to recognize cross-layer data flow and fuse mergeable reductions, normalizations, and residual additions into the kernels of adjacent layers.
The difference of visual attention is that it is a non-causal bidirectional attention: the image-block sequence is short but its shape varies with input resolution, and a common approach is to pack multiple images into a variable-length sequence and use a segmented block-diagonal mask to prevent inter-image cross-talk. Domestic cards usually require the following handling:
- Attention kernels that support variable-length sequences;
- Concatenation of positional encodings for image-block sequences and text tokens;
- Resolution bucketing to reduce the recompilation frequency caused by dynamic shapes;
- Visual tokens completing projection alignment before entering the text backbone.
Additionally, the insertion of visual tokens changes the state-advancement position of linear attention layers in the text backbone. If the upstream model has special handling at visual tokens (such as state reset or skipping), the adaptation must check item by item against the model definition and cannot take it for granted that they all advance uniformly as text tokens.
8. Validation and Troubleshooting: Making Adaptation Regressible
Adaptation on domestic cards most fears “delivering as soon as it runs”. It is recommended to establish three levels of validation.
1. Operator-level numerical alignment
- Use a high-precision reference implementation (such as the framework’s CPU implementation) as the gold standard;
- Compare each new operator element by element or block by block;
- Focus on checking the mask correctness of sparse attention, the accumulated error of linear attention state updates, and deviation amplification under low-precision paths.
2. Model-level end-to-end alignment
- Compare output logits and generated text under the same prompt and same sampling parameters;
- Validate both short and long contexts separately;
- Cover combined scenarios such as Prefill/Decode switching, multimodal input, and speculative decoding branches.
3. Service-level stability validation
- Long-duration stress testing, observing memory growth, request queue latency, and timeouts;
- Focus on whether the KV cache and linear attention state are released with consistent lifecycles;
- Check whether tensor-parallel communication in multi-card scenarios is synchronized with sparse indexing.
A practical approach is to extract the layer types, attention parameters, and state dimensions in the model structure into an “adaptation checklist”, annotating item by item the corresponding kernel, cache type, and validation case. The checklist serves both as a development task table and as an index for regression testing.
9. Conclusion
Landing hybrid attention architectures on domestic AI accelerators is essentially a cross-layer collaborative engineering effort:
- Index selection determines the memory-access form of sparse attention;
- KV cache layout determines the overhead of indirect addressing;
- Prefill and Decode differences determine the trade-offs in kernels and scheduling strategy;
- Linear attention state management determines the correctness of long-sequence and multi-branch scenarios;
- Cross-layer residuals and visual attention determine whether the entire execution pipeline can close the loop.
The operator library, kernel programming, and compiler each solve the problems of “having operators, being expressive, and being optimizable”, while the inference framework strings them into a deliverable service. As long as each compute path’s resource constraints, data layout, and dependencies are clearly explained, adaptation work can shift from “repeated trial and error” to “progressing by checklist”.
