LLM Operator Library Migration and Integration: A Practical Sparse Attention Case Study
New large language model architectures are evolving faster than the operator stacks that support them. This article reviews a model integration effort on a domestic GPU platform, using a layered approach based on optimized operator libraries, tensor programming for custom kernels, and graph compilers. The main focus is sparse attention, including index scoring, Top-K selection, KV-cache layout, and Prefill/Decode paths. It also covers integration with state-based attention, cross-layer residuals, and vision attention, along with validation and performance-tuning practices. The goal is to provide a reusable migration methodology for new LLM architectures.
LLM Operator Library Migration and Integration: A Practical Sparse Attention Case Study
Introduction: Why New LLM Architectures Are Hard to Support
The iteration speed of large model architectures has outpaced the adaptation cycle of hardware and software stacks. New models increasingly use mixed structures such as sparse attention, linear attention with state updates, and cross-layer residuals. Some models also introduce vision branches.
For these new computation paths, it is often difficult to find ready-made and well-tuned implementations in basic operator libraries in a short period. Relying only on combinations of basic operators may lead to incomplete functionality or lower performance, slowing down production rollout.
This article is a retrospective on integrating a model on a domestic GPU platform. The technical team used a community inference framework to host upstream open-source model code and completed the migration of key computation paths through three layers of compute support:
- Operator libraries: Provide highly optimized implementations of basic operators.
- Tensor programming paradigm: Allows custom kernels to be written using Python-like syntax.
- Graph compilers: Handle whole-graph optimization and code generation.
The integration focus was sparse attention. The adaptation work covered index selection, KV-cache layout, and the two computation paths: Prefill and Decode. It also handled coordination with state-based attention, cross-layer residuals, and vision attention.
This article removes specific product context while preserving the transferable methodology.
1. Overall Architecture: A Layered Operator Supply System
Hardware and software co-design requires a clear boundary. A reasonable layering allows each layer to focus on what it does best:
- Hardware and runtime layer: Handles device management, memory allocation, stream and event scheduling, and collective communication.
- Operator layer: Provides three types of capability. Operator libraries provide basic operators, the tensor programming paradigm is used to write custom kernels, and the compiler handles graph-level optimization.
- Framework layer: The community inference framework, responsible for model structure, KV-cache management, continuous batching, and service scheduling.
- Service layer: Exposes inference interfaces.
The roles and trade-offs of these layers are summarized below:
| Layer | Main Responsibilities | Advantages | Limitations |
|---|---|---|---|
| Operator library | High-performance implementations of basic operators such as GEMM, attention, and normalization | Well validated, ready to use, and stable in performance | New operators introduced by new architectures are usually not covered |
| Tensor programming paradigm | Use a DSL to write custom kernels and support fusion or specialized optimization | Fast iteration and fine-grained control for sparse indexing, fused attention, and similar cases | Tuning relies heavily on developer experience |
| Compiler | Graph optimization, operator fusion, memory reuse, and code generation | Global optimization across operators and automatic adaptation to different input shapes | Recognition and support for new operators require continuous follow-up |
The relationship among the three can be summarized as:
- Operator libraries provide the foundation.
- Tensor programming paradigms fill the gaps.
- Compilers provide further optimization.
More specifically:
- Operator libraries handle stable basic computations.
- Tensor programming paradigms handle computations that are missing from the libraries or not well aligned with the new architecture, such as masked sparse attention fusion kernels.
- Compilers perform fusion, layout transformation, and memory reuse on top of these capabilities, reducing the overhead caused by per-operator interpretation in the framework.
The framework layer also has an often-overlooked responsibility: providing backend abstraction and operator registration mechanisms that decouple model logic from hardware implementation. The model definition only describes the computation graph, while specific kernels are selected by the registry according to the available backends.
This layered design brings two direct benefits:
- When integrating a new model, framework-side changes are concentrated in custom operators and cache management, without rewriting existing engineering code.
- Hardware-side operator gaps can be clearly identified and filled according to priority.
2. Sparse Attention: Mechanism and Migration Overview
2.1 Breaking Down the Computation Path
In traditional attention, every query must interact with all historical keys. The computation grows quadratically with sequence length.
Sparse attention, often referred to as DSA, first uses a lightweight indexer to select a small set of “truly relevant” candidates for each query. Full attention is then computed only over the candidate set. If the sequence length is L and each query keeps k candidates, the complexity of the attention part drops from O(L²) to approximately O(L·k).
The computation path can be broken into four steps:
- Index scoring: Use a low-dimensional projection to compute the relevance between queries and keys.
- Top-K selection: Select the
kcandidates with the highest relevance for each query. - KV-cache layout: The main KV cache stores complete key-value pairs; another low-dimensional key cache is used by the indexer for scoring.
- Sparse attention: Attention output is computed only over the candidate set.
These four steps appear in both Prefill and Decode paths, but the design trade-offs differ:
| Stage | Prefill Path | Decode Path |
|---|---|---|
| Index scoring | Many queries, dominated by matrix multiplication | Only a small number of queries per step, dominated by matrix-vector operations |
| Top-K selection | Candidate set is the full history, with large computation volume | Candidate set consists of cached index keys and grows with sequence length |
| Attention computation | Block-parallel execution, focusing on compute utilization | Parallel per candidate and then reduce, focusing on reducing latency |
| Main bottleneck | Compute throughput | Cache-read bandwidth and kernel launch overhead |
3. Migration Point 1: Index Selection and Top-K
3.1 Score Only Valid Tokens
Index quality directly determines generation quality. A common mistake during migration is not that the kernel is incorrect, but that scoring is not based only on valid tokens.
Three frequent errors are:
- Padding positions participate in scoring, allowing invalid positions to occupy candidate slots.
- Causal masks and Top-K selection are processed in the wrong order.
- After multi-request packing, position information leaks into adjacent requests.
A stable implementation order is:
- Compute scores.
- Set invalid positions to negative infinity.
- Select Top-K.
Top-K selection is often executed in fixed-size blocks, such as selecting 64 candidates from every 256 candidates, with an additional local window. This aligns the candidate count with block sizes, allowing kernels to use fixed loop unrolling and avoiding linear growth in selection overhead under long contexts.
3.2 Floating-Point Comparison and Approximate Sorting
Indexers usually use low-precision computation, such as FP16 or BF16, to save bandwidth. Under low precision, many scores may become equal. If strict Top-K is implemented as “sort first, then truncate”, numerical noise can introduce non-determinism.
A more robust approach is approximate selection based on quantile thresholds:
- Perform block-wise reduction on scores to obtain approximate quantile points as thresholds.
- Keep candidates whose scores are above the threshold, and fill any shortage from candidate blocks by position.
This makes selection results less dependent on scoring precision and improves reproducibility across different hardware implementations. However, approximate selection may miss a few high-scoring candidates. This error must be validated through end-to-end accuracy evaluation to ensure it remains acceptable.
3.3 Organizing Index Outputs
The return values of Top-K should be as kernel-friendly as possible:
- Use
int32indices to avoid the register and bandwidth overhead of defaultint64. - Store candidates contiguously by block so that downstream gather accesses remain contiguous.
- Set masked candidates to an invalid sentinel value, such as
-1, and let downstream kernels handle them as masks. Do not drop them directly during index generation. This keeps tensor shapes fixed and reduces dynamic branches.
Example code is shown below:
# idx_score: [num_heads, query_len, kv_len]# q_pos: position of each query in the sequence, used to build causal constraintsdef select_topk(idx_score, q_pos, block_size=256, k_per_block=64):# 1. Causal masking: a query can only attend to itself and previous tokensvalid = kv_indices() <= q_posidx_score = where(valid, idx_score, NEG_INF)# 2. Block-wise approximate selection: take top-k_per_block per block and mergeselected = []for blk in blocks(idx_score, size=block_size):local = approx_topk(blk, k=k_per_block) # threshold-based method, avoiding strict sortingselected.append(local)# 3. Keep a local window to ensure enough candidates for short sequences or early tokensselected = merge_and_dedup(selected + local_window(k))# 4. Fixed-length output: fill insufficient positions with an invalid sentinelreturn pad_to_k(selected, total_k=k_total, pad_value=-1)
4. Migration Point 2: KV Cache Layout
4.1 Dual Caches and Paged Management
Sparse attention requires two caches with different characteristics:
- Main KV cache: Must be written for every token and is relatively large.
- Index key cache: Has lower dimensions and is used only for indexer scoring. Its lifetime is the same as the main KV cache.
A unified paged management scheme is recommended:
- Allocate pages of a fixed number of tokens in advance. Requests do not share pages, avoiding interference between requests.
- Each page records the corresponding token position range, making it easier to reconstruct causal masks.
- The page table records the mapping from logical token positions to physical pages. When a request ends or is truncated, entire pages are reclaimed.
This is valuable because request lengths change frequently. Contiguous allocation can cause fragmentation. Paged management keeps memory utilization more stable and makes it easier to dynamically adjust the requests in a batch under continuous batching.
4.2 Tensor Dimension Order
The recommended layout for the main KV cache is:
[num_pages, page_size, num_kv_heads, head_dim, 2] # K and V in the last dimension
The layout for the index key cache is:
[num_pages, page_size, index_head_dim] # low-dimensional, placed in the same page table
The reasons for this arrangement are:
- For the same token, K and V are adjacent in memory, so the decode kernel can read a complete key-value pair in one access.
- After aligning page size with the block size of the index scoring kernel, each page becomes a contiguous memory access unit.
- The two caches share the same page table, avoiding mismatches between index scoring and actual value retrieval.
Example code is shown below:
@dataclassclass KVLayout:num_pages: intpage_size: int # number of tokens per page, preferably aligned with index scoring blocksnum_kv_heads: inthead_dim: intindex_head_dim: int # low-dimensional length of index keys# Main cache: [num_pages, page_size, num_kv_heads, head_dim, 2]# Index cache: [num_pages, page_size, index_head_dim]def alloc(self, max_seq_len: int, batch: int):pages = ceil(max_seq_len / self.page_size)return (empty(self.num_pages, self.page_size, self.num_kv_heads,self.head_dim, 2),empty(self.num_pages, self.page_size, self.index_head_dim),PageTable(batch, pages),)
4.3 Page Size and Quantization
Page size selection requires a trade-off:
- If too small, the number of page-table entries increases, raising mapping overhead.
- If too large, internal fragmentation is more likely at the end of sequences.
In practice, it is common to start with 64 or 128 tokens per page and adjust based on measured throughput.
In addition, the main KV cache can use FP8 or INT8 storage to reduce bandwidth pressure. This is a mature approach in the industry. However, index keys should preferably retain higher precision, such as BF16, because scoring is more sensitive to quantization noise. Once the ranking result changes, the quality loss of sparse attention can be amplified.
5. Migration Point 3: Adapting Prefill and Decode Paths
5.1 Prefill Path
The Prefill stage processes hundreds or thousands of tokens at once. Index scoring is a standard matrix multiplication and should be handled by highly optimized GEMM implementations in operator libraries. Custom kernels should not be written unnecessarily.
Top-K selection is suitable for a dedicated kernel written with the tensor programming paradigm. It should be tightly coupled with scoring results to avoid repeatedly reading and writing intermediate results to memory.
The core goal of the Prefill stage is throughput: ensure that each execution unit has enough work and reduce the number of kernel launches. Scoring and selection can be merged into the pipeline of a single kernel, using producer-consumer-style block execution so that scoring and selection overlap in time.
5.2 Decode Path
The Decode stage processes only a small number of tokens per step. The bottleneck shifts from compute capability to memory bandwidth and kernel launch overhead.
Three recommendations are:
- Index updates: Write newly appended tokens to the end of a page first, then update the index cache. Only recompute scoring for the current position to avoid recalculating the entire sequence.
- Sparse attention kernel: Execute in parallel by candidate blocks, perform local reduction within each block, and finally merge results. This hides latency through parallelism instead of waiting for all candidate reads to complete.
- Fixed launch overhead: Candidate counts per request may vary significantly within a batch. Kernels should keep fixed shapes as much as possible, allocating by the maximum
kand handling length differences through masks. This avoids frequent recompilation or entering slow paths.
5.3 Example Execution Path
# Prefill: processes many tokens at once, aiming for throughputdef prefill(tokens, kv_pages, idx_pages, layout, num_heads, k_total):# 1. Write main KV cache and index key cachewrite_kv(kv_pages, idx_pages, tokens, layout)# 2. Index scoring (GEMM kernel)q_low = low_dim_proj(q) # reduce to index dimensionscore = gemm(q_low, idx_pages.keys) # [heads, L, L]score = apply_causal_mask(score, q_pos)# 3. Top-K selection (fused kernel, overlapping with scoring)sel = select_topk(score, k=k_total)# 4. Sparse attention: compute only over candidates, executed in parallel blocksout = sparse_attn(q, kv_pages, sel) # [heads, L, head_dim]# 5. Output projectionreturn o_proj(out)# Decode: one token per step, aiming for low latencydef decode(step_q, kv_pages, idx_pages, layout, num_heads, k_total):q_low = low_dim_proj(step_q)# Append the K/V and index key for the current tokenappend_token(kv_pages, idx_pages, step_q)# Index update: compute scores from the current query to all historyscore = gemm(q_low, idx_pages.keys) # [heads, 1, seq]sel = select_topk(score, k=k_total) # fixed-length output# Read candidates in parallel blocks, reduce within blocks, then mergeout = sparse_attn_decode(step_q, kv_pages, sel)return o_proj(out)
6. Mixed Execution Chains: State Management, Cross-Layer Residuals, and Vision Attention
Modern models often do not use a single attention structure. Instead, multiple mechanisms are interleaved.
For example, in a typical mixed-architecture integration, the execution chain includes four types of computation:
- Sparse attention.
- Linear attention with state updates, similar to state management mechanisms such as KDA.
- Cross-layer residual connections.
- Attention in the vision branch.
The adaptation strategies for each part are described below.
6.1 State-Based Attention
These modules maintain a recurrent state, similar to the hidden state in recurrent networks, and update it step by step for each token.
Their computation characteristics are the opposite of sparse attention:
- The computation volume is small, but state read/write operations are frequent.
- State updates are strictly sequential and difficult to parallelize across the token dimension.
Therefore, they are suitable for implementation as a fused kernel: read state, compute the update, and write the state back in one pass, minimizing memory access.
Three migration considerations are important:
- State lifecycle management: State caches should be allocated separately from KV caches and maintained per request. If a request is preempted or swapped out, the state must be saved and restored along with the KV cache. Otherwise, continued execution may produce incorrect results.
- Numerical stability: If recursive updates include division or normalization, low-precision accumulation errors can grow step by step. It is recommended to use FP32 accumulation inside the state-update kernel.
- Fusion boundaries: The boundaries between state updates, cross-layer residuals, and gating computations should be determined during graph compilation. Avoid cross-layer fusion at the kernel level, because that makes code harder to maintain.
6.2 Cross-Layer Residuals
Cross-layer residuals allow a layer to read outputs from earlier layers rather than only the immediately previous layer. The difficulty of adapting this structure is not the computation itself but the dependency relationships:
- The framework execution scheduler must determine inter-layer dependencies by topological order. It cannot simply assume sequential execution layer by layer.
- When performing operator fusion, the compiler must identify cross-layer read points and avoid fusing away intermediate results that should be preserved.
- Memory reuse strategies must be adjusted accordingly: tensors referenced by later layers cannot be reclaimed early.
In practice, the best approach is to explicitly declare inter-layer connections in the model definition and let the compiler generate a dependency graph from them, rather than relying on implicit execution order in the framework.
6.3 Vision Attention
In multimodal models, the vision branch is usually an independent encoder. Its attention computation differs from the text branch:
- Sequence length is determined by image resolution and is relatively fixed.
- The computation pattern is closer to traditional dense attention.
This part can directly reuse mature attention implementations from operator libraries. A dedicated kernel is usually not necessary.
The scheduling relationship with the text branch must be considered. Vision encoding is usually completed before Prefill, and its output is concatenated with the text sequence as vision tokens before entering the mixed attention layers of the backbone.
If vision encoding and text Prefill are executed in the same batch, the peak memory usage may increase significantly. It is necessary to stagger them in batch organization.
6.4 Recommended Execution Chain
Putting these mechanisms together, a practical execution order is:
- The vision encoder processes images and produces vision tokens.
- Vision tokens are concatenated with text tokens and enter the backbone.
- The backbone executes mixed attention layer by layer, alternating sparse and state-based attention, with cross-layer residual aggregation after each layer.
- Finally, the output layer produces hidden states for each position.
The scheduling points can be summarized as follows:
| Mechanism | Computation Characteristics | Adaptation Strategy |
|---|---|---|
| Sparse attention | Index computation and sparse reads | Dedicated kernel for index selection + paged cache layout |
| State-based attention | Recursive updates, frequent state reads/writes | Fused kernel + FP32 accumulation + request-level state management |
| Cross-layer residuals | Complex inter-layer dependencies | Dependency analysis during graph compilation + adjusted memory reuse |
| Vision attention | Dense computation, relatively fixed sequence length | Reuse mature operator-library implementations |
7. Accuracy Validation and Performance Tuning
7.1 Layer-by-Layer Comparison
For migration correctness, validation is recommended at three levels:
- Operator level: Write unit tests for index selection, sparse attention, and state updates. Compare outputs against a reference implementation, such as the explicit computation of dense attention.
- Submodule level: Compare the output of a single layer to determine whether the deviation comes from the index layer or the attention layer.
- End-to-end level: Compare generation results using fixed inputs.
When comparing results, note that sparse attention and dense attention will not be numerically identical, because candidate selection differs and outputs naturally differ. The criterion is therefore not element-wise equality, but whether the relative error is within tolerance and whether generation quality remains stable.
7.2 Key Performance Metrics
From a performance perspective, four metrics should be tracked:
- Time to first token (TTFT).
- Time per output token (TPOT).
- System throughput.
- Peak memory usage.
A practical analysis order is:
- Use profiling tools to identify which kernels consume the most time.
- Determine whether the bottleneck is compute-bound or memory-bound.
- Decide whether to optimize index selection, adjust page size, or change the operator fusion strategy.
8. Common Engineering Issues
Frequently encountered issues during integration and their debugging directions include:
- Index out of bounds or candidates containing invalid positions: Check the execution order of causal masking and Top-K selection, and confirm that masked positions are set to negative infinity during scoring.
- High scoring overhead under long contexts: Confirm whether block-wise approximate selection is enabled and whether index caches use low-dimensional storage.
- Decode latency jitter: This is often caused by varying candidate counts across requests in a batch. Consider fixed-shape kernels with masking.
- Output drift in state-based modules: Check whether state updates use low-precision accumulation and whether states are correctly saved and restored during preemption.
- Incorrect cross-layer residual results: Confirm whether the compiler incorrectly fused intermediate results that are referenced by later layers.
- High memory peaks in the vision branch: Check whether vision encoding and text Prefill are scheduled in the same batch.
Conclusion
Adapting hardware for new LLM architectures is essentially an engineering problem of computation-path decomposition and layered supply.
By breaking sparse attention into index scoring, Top-K selection, cache layout, and attention kernels, the responsibilities and adaptation methods of each part become clear:
- Stable basic computation is handled by operator libraries.
- Irregular new logic is handled by the tensor programming paradigm.
- Global optimization across operators is handled by the compiler.
Adding state management, cross-layer dependencies, and vision-branch scheduling turns a seemingly difficult migration into a reusable integration checklist.
For teams integrating models on domestic GPU platforms, this workflow — layer first, decompose computation paths, then validate and tune — is more controllable than a kernel-by-kernel brute-force approach. It is also easier to turn into a long-term engineering asset.
