Operator Library Design and Practice for LLM Inference Acceleration

    This article explains how operator libraries fit into the LLM inference stack and how to adapt key compute paths to domestic AI accelerators. It covers the memory-bound, uneven-compute characteristics of autoregressive decoding, the layering between inference frameworks, DL compilers, and operator libraries, and a "get it running, optimize, then solidify" migration path. It then walks through the main adaptation points for sparse attention: index selection, KV cache layout, and separate Prefill and Decode kernels. It also addresses coordination with state management, cross-layer residuals, and multimodal attention, and closes with practical engineering recommendations. Readers will come away with a concrete approach to keeping inference performance stable while preserving numerical correctness.

    1. Background: What LLM Inference Demands from an Operator Library

    The compute cost of LLM inference is not evenly distributed. During autoregressive decoding, every forward pass re-reads the weights and the KV cache, so memory bandwidth usually becomes the bottleneck before compute does. With long contexts, the cost of attention also grows rapidly with sequence length. This mix of “memory-bound and uneven compute” means inference performance depends heavily on the quality of the underlying operator implementations.

    The problem gets harder when you need to run community open-source LLMs on domestic AI accelerators: their instruction sets, memory hierarchies, and vector widths differ from mainstream GPUs, and the high-level compute graphs produced by framework layers often cannot be mapped down efficiently as-is. The operator library sits at exactly this layer and takes on a “translate + optimize” role, mapping framework-level abstractions such as matrix multiply, attention, normalization, and activation into concrete implementations that stay close to the hardware’s memory hierarchy and parallel units.

    The value of an operator library comes down to three things:

    • Performance: vectorization, tiling, and memory-access scheduling tailored to specific hardware, pushing as close to peak throughput as possible.
    • Correctness: boundary handling and numerical precision control that keep outputs stable for long sequences and large batches.
    • Portability: decoupling hardware-specific implementations from upper-layer model logic, which lowers the migration cost for new architectures and new models.

    2. Where the Operator Library Sits in the Inference Stack

    A typical inference software stack can be divided top-down into several layers: the inference serving framework at the top, responsible for model loading, request scheduling, and batching; the operator development framework and deep learning compiler in the middle, responsible for lowering compute graphs, performing operator fusion, and generating code; and the operator library together with the accelerator runtime at the bottom, interacting directly with the hardware.

    1. Inference framework (model loading / request scheduling / continuous batching)
    2. │
    3. Operator development framework (DSL-based description + auto-tuning)
    4. │
    5. Deep learning compiler (graph optimization / operator fusion / code generation)
    6. │
    7. Operator library (matmul / attention / normalization / activation ...)
    8. │
    9. Accelerator runtime and driver

    Within this layering, the operator library does not exist in isolation. It is consumed by the upper layers as a set of pluggable operator implementations: for operators the framework already ships with high-performance kernels for, the library provides an equivalent or better implementation for this particular hardware; for new structures the framework does not yet cover (a newly introduced sparse attention variant, for example), the operator development framework and the compiler fill in an implementation path quickly, which then gradually settles back into the operator library. This keeps compatibility with upstream models intact while giving the critical compute paths of new architectures a way to keep evolving.

    3. Migrating and Integrating Key Compute Paths

    Moving community model code to domestic accelerators is rarely a single rewrite. It usually follows a “get it running, then optimize, then solidify” path:

    1. Path identification: analyze the model’s compute graph and locate the memory-bound and compute-bound key operators, especially attention, which typically dominates the cost.
    2. Filling gaps with the operator development framework: use a domain-specific language to describe the computation in relatively concise code, let the framework handle tiling, pipelining, and vectorized scheduling automatically, and get a first runnable implementation quickly.
    3. Compiler optimization: use the deep learning compiler to fuse operators (for example, folding scaling, masking, and softmax into the attention kernel) so intermediate tensors are read and written less often and less device memory is used, and to schedule the generated code at the instruction level.
    4. Solidifying into operator library implementations: hand-tune the stable, high-frequency operators in depth, bring them under unified operator library management, and let later models reuse them directly.

    The key to this path is having the three pieces work together: use the development framework to validate quickly, the compiler to optimize automatically, and the operator library to preserve results long term. That avoids spending most of the effort on repeated manual porting.

    4. Adapting the Sparse Attention Operator

    For long-context inference, dense attention brings quadratic cost, which makes sparse attention an important optimization direction. Integrating a sparse attention operator onto a domestic accelerator requires careful adaptation at a few specific points.

    1. Index selection
    The core idea of sparse attention is to compute over only some of the Key/Value entries. Index selection determines which token positions actually participate in computation at each step, and the resulting distribution is usually irregular, which breaks memory-access continuity. In practice you need to convert irregular indices into a hardware-friendly access pattern: align indices by block, merge adjacent indices, and generally collapse random access into block-wise sequential access wherever possible.

    2. KV cache layout
    The memory layout of the KV cache directly determines how efficiently the attention kernel accesses memory. For the accelerator’s memory hierarchy and vector width, the cache’s dimension ordering and paging strategy need to be designed so that a single decoding step can read the required KV entries with as little data movement as possible. Schematically, a head-grouped, block-aligned layout can be expressed as:

    1. # Illustrative: block-aligned KV cache indexing (not a concrete API, conveys the idea only)
    2. def build_sparse_indices(seq_len, block_size, topk_blocks):
    3. # Split the sequence into blocks, pick the blocks to compute, keep each block contiguous
    4. blocks = list(range(0, seq_len, block_size))
    5. selected = select_top_blocks(blocks, topk_blocks) # index selection
    6. return [idx for b in selected for idx in range(b, min(b + block_size, seq_len))]

    3. Two compute paths: Prefill and Decode
    These two phases have completely different memory-access and compute characteristics, so the operator often has to be adapted separately for each:

    • Prefill: processes the entire input in one shot. It is compute-bound with large matrices, so the optimization focus is on compute utilization and parallelism.
    • Decode: processes one token at a time. It is memory-bound and the matrices are “tall and thin”, so the focus shifts to reducing KV read overhead and scheduling overhead.

    The same attention logic therefore commonly needs different kernels for the two paths, with the differences hidden from upper layers behind a unified calling convention.

    5. Coordinating Multiple Operators: State, Residuals, and Multimodal Attention

    Modern LLMs are not just a stack of attention blocks; the inference path is usually an execution graph made up of several kinds of operators. Beyond sparse attention, the library also has to coordinate with:

    • State management: some architectures maintain hidden state across steps. The operator library needs efficient state read, write, and update interfaces so that state stays consistent and cheap during long-sequence decoding.
    • Cross-layer residuals: residual connections pass tensors between layers, and their layout must line up with the attention and normalization operators to avoid extra data movement.
    • Vision (multimodal) attention: the vision branch of a multimodal model also contains attention computation, with tensor shapes and masking rules that differ from the text branch. The library needs attention implementations that are both reusable and customizable.

    At runtime these operators are composed into a complete execution chain. Meeting overall performance targets requires more than per-operator tuning: tensor layout consistency between operators, fusion boundaries, and scheduling hand-offs all have to be handled so that redundant cross-operator data movement is minimized.

    6. Engineering Practice Recommendations

    Based on the above, here is what to keep in mind when running LLM inference on domestic accelerators:

    • Measure before optimizing: get end-to-end working with basic operators first, identify which operators are the bottleneck and whether each is memory-bound or compute-bound, then optimize those specifically instead of diving into local tuning too early.
    • Prefer reuse and fusion: reuse existing operator library implementations where you can; for high-frequency combinations (masking plus softmax, for instance), fuse them in the compiler to cut round trips to device memory.
    • Tune each path separately: set distinct optimization targets and validation cases for Prefill and Decode so that gains on one path do not come at the cost of the other.
    • Keep numerical results consistent: optimizations such as sparsification, tiling, and lower precision all have to start from precision parity, and be regression-checked with representative inputs.
    • Build a repeatable benchmark: keep a stable set of performance and accuracy benchmark cases to act as the gatekeeper as operators evolve and models iterate.

    7. Summary

    The operator library is the core bridging piece in the LLM inference software stack: it fits the characteristics of domestic accelerators below and serves the inference framework and model logic above. For new structures such as sparse attention, the path to follow is “validate quickly with the development framework, optimize automatically with the compiler, and solidify over the long term in the operator library”. Around that path, coordinated adaptation is needed across index selection, KV cache layout, the dual Prefill/Decode paths, and state management, residuals, and multimodal attention. Only then can you reliably unlock an accelerator’s inference performance while keeping results correct.