Accelerating Inference for Hybrid-Architecture LLMs: A Systematic Practice from Operator Fusion to Layer-Aware Quantization

1. Background: Inference Challenges Introduced by New Architectures

Recent architectural evolution in large language models points to a clear trend: relying solely on attention mechanisms makes it difficult to simultaneously meet the dual requirements of long-sequence modeling capability and inference efficiency. The industry has begun exploring hybrid architectures that interleave state recurrence mechanisms with sparse attention mechanisms. The former efficiently handles sequential dependencies via state updates, while the latter introduces index-based selection into attention computation to reduce KV cache read overhead in long-context scenarios.

While these hybrid architectures deliver strong model capability, they pose new challenges for inference systems:

• Heterogeneous compute paths: State recurrence and sparse attention differ significantly in compute patterns. The former is a sequential, state-by-state recurrence with serial dependencies; the latter is index-driven sparse memory access. Interleaving these two paths places high demands on scheduling.
• Serial bottleneck in input preparation: Before attention computation, multiple steps must be completed—Q projection, K normalization, quantization, and head-weight projection. Executing these steps serially introduces considerable latency.
• Cache and data movement overhead: Sparse attention relies on index-based KV cache reads. When quantized and BF16 paths coexist, memory footprint and data movement increase notably.

This article systematically reviews engineering practices for optimizing inference of hybrid-architecture LLMs across three dimensions: operator fusion, multi-stream orchestration, and layer-aware quantization.

2. Architecture Analysis: Coordination of State Recurrence and Sparse Attention

2.1 State Recurrence Path

The state recurrence mechanism processes sequence information by maintaining a state vector that evolves over time. Its core advantage is that inference does not require recomputing full attention over historical positions; instead, information aggregation is performed through state updates. This pattern is naturally suited for autoregressive decoding, where each decoding step requires only a single state update.

2.2 Sparse Attention Path

Sparse attention builds on multi-head latent attention (MLA) by introducing an index selection mechanism. The overall pipeline consists of two stages:

1. Lightweight indexer filtering: Relevance scores are computed for historical positions, then weighted by head weights to select the Top-K most relevant position indices. This stage reads only paged K cache, with low compute and manageable memory access.
2. Sparse attention computation: Based on the selected indices, corresponding KV entries are fetched and attention is computed only for these positions, reducing complexity from full O(n) to sparse O(k).

2.3 Cross-Layer Representation Aggregation

The model aggregates cross-layer representations through attention residual mechanisms. This requires the inference system to carefully manage intermediate states across layers, placing constraints on memory layout and data reuse.

3. Operator Fusion: The Key to Reducing Decode Latency

3.1 Indexer Three-in-One Fusion

The lightweight indexer’s compute chain consists of three sequential steps: relevance scoring, head-weight weighting, and Top-K selection. In traditional implementations, each step runs as a separate kernel, with intermediate results written back to memory and read again—introducing extra memory access and kernel launch overhead.

The optimization fuses these three steps into a single operator: scoring results are directly weighted and compared for Top-K selection in registers or shared memory, and the paged K cache is read directly, eliminating intermediate write-backs. This fusion yields two direct benefits:

• Fewer kernel launches, reducing scheduling overhead.
• Removal of intermediate tensor allocation and movement, lowering bandwidth pressure.

3.2 Three-Stream Coordination in the Input Preparation Phase

The input preparation phase before attention computation originally required sequential execution of:

• Q projection
• K normalization
• Q quantization
• Head-weight projection

These four steps have data dependencies but are not fully serial. Through three-stream coordinated orchestration, operations with no direct dependency can run in parallel:

Stream 1: Q projection ──────────────►
Stream 2: K normalization ──► Q quantization ────►  (Q quantization depends on K normalization)
Stream 3: Head-weight projection ──────────►

By assigning Q projection, K normalization + Q quantization, and head-weight projection to different streams for parallel execution, serial waiting in the input preparation phase is significantly reduced, especially in small-batch decoding scenarios.

3.3 Dual-Path Support for Prefill and Decode

The indexer operates in both Prefill and Decode phases, but their compute characteristics differ: Prefill involves long sequences with high parallelism; Decode involves short sequences with memory-intensive access. The optimization supports both BF16 and quantized paths in both phases, allowing the system to flexibly balance precision and performance based on actual workload.

4. Layer-Aware Quantization: Reducing Cache and Data Movement Overhead

4.1 Hierarchical Quantization Strategy

Different modules in a hybrid architecture exhibit varying sensitivity to precision, so quantization must be layer-aware:

• DSA projection retains BF16: Projection layers directly participate in attention score computation, and precision loss directly affects output quality; thus they remain in BF16.
• Indexer is fully quantized with MXFP8: The indexer only selects Top-K positions and is relatively tolerant to precision, so full quantization is applied to reduce memory access overhead.

4.2 Per-Group Dynamic Quantization

The indexer’s Q/K quantization uses per-group dynamic quantization with a group size of 32—every 32 elements share one scaling factor. Compared to per-tensor quantization, per-group quantization better adapts to data distribution variations; compared to per-channel quantization, it incurs lower metadata overhead, striking a balance between memory access and precision.

After quantization, the indexer reads K cache with substantially reduced data movement. This is especially critical for the Decode phase, where historical K cache is read at every step—bandwidth savings translate directly into throughput gains.

5. System-Level Optimization: Full-Stack Coordination from Compute to Storage

Beyond operator and quantization optimizations, the inference system also requires coordination at a broader level:

5.1 Hybrid Parallelism and Communication Optimization

Different modules in a hybrid architecture exhibit different compute characteristics, and a single parallelization strategy cannot cover all scenarios. Hybrid parallelism combines tensor parallelism, pipeline parallelism, and data parallelism according to module characteristics, paired with communication optimizations such as overlapping communication with computation, to reduce inter-GPU communication latency.

5.2 Multi-Stream Orchestration and SuperKernel

Multi-stream orchestration assigns independent operators to different CUDA streams for parallel execution, while SuperKernel technology merges multiple small operators into a single larger kernel, reducing kernel launch and scheduling overhead.

5.3 Graph Mode and Compilation Cache

Graph mode captures the inference process as a static compute graph, eliminating Python-level scheduling overhead. Compilation cache avoids repeated compilation, speeding up cold starts and model switching.

5.4 NZ Weight Layout

Weights are stored in an NZ layout, providing better memory access locality for matrix multiplication on hardware, particularly improving read efficiency for quantized weights.

5.5 Speculative Inference and State Replay

Speculative inference boosts decoding throughput by using a small draft model combined with large-model verification. The state replay mechanism targets the state recurrence path: when speculative verification fails, it quickly replays state to the correct position, avoiding full recomputation.

6. Practical Recommendations and Summary

For developers deploying hybrid-architecture LLMs on heterogeneous compute platforms, the following points deserve attention:

1. Prioritize operator fusion: The indexer three-in-one fusion and multi-stream coordination in input preparation offer the highest return on investment, often yielding significant gains without modifying model structure.
2. Adopt layer-aware quantization: Avoid applying a uniform quantization scheme to the entire model. Differentiate based on module precision sensitivity; selection-type modules like the indexer can be aggressively quantized.
3. Focus on Decode-phase memory access: Autoregressive decoding is a memory-bound workload. Techniques such as quantization, cache layout optimization, and NZ weight layout often improve Decode throughput more than Prefill throughput.
4. Do not overlook system-level coordination: Parallelism strategies, communication optimization, graph mode, and compilation cache must be combined—point optimizations alone cannot cover the full pipeline bottleneck.

Inference optimization for hybrid-architecture LLMs is an engineering effort that requires coordinated investment across operator design, quantization, scheduling, and storage. As model architectures continue to evolve, inference systems must maintain rapid adaptation capability—by consolidating optimization techniques into reusable operator libraries and scheduling frameworks, swift support and efficient deployment can be achieved for new architectures as they emerge.