Day0 Inference Support and Performance Optimization for Next-Gen Large Models on Huawei Ascend CANN Architecture

As large language model (LLM) parameter counts and context lengths continue to scale, new model architectures are emerging at a rapid pace. Facing complex improvements to attention mechanisms and deeper Mixture of Experts (MoE) structures, how underlying AI infrastructure can achieve “Day0”-level rapid adaptation and extreme performance optimization has become a core concern for developers and enterprises alike. This article takes the Huawei Ascend CANN (Compute Architecture for Neural Networks) stack as its entry point and explores in depth how to build on past operator optimization and compiler capabilities to achieve efficient inference deployment for next-generation large models.

1. Leveraging Historical Optimizations for Rapid Architecture Adaptation

Next-generation large models exhibit highly hybrid and sparse architectural characteristics. A typical pattern interleaves modules that handle sequential state recurrence with sparse attention modules that use index-based selection. Specifically, a “lightweight indexer” screens for Top-K high-relevance positions in the historical sequence; a “sparse flash attention” module then reads the corresponding key-value (KV) pairs to complete sparse attention computation, with attention residual mechanisms aggregating cross-layer representations.

Building low-level operators from scratch for such complex architectures is not only time-consuming but also prone to introducing performance bottlenecks. CANN’s core strategy is “operator reuse and architecture translation.” By reusing a previously accumulated high-performance operator library—including FlashKDA/MegaKDA for state recurrence, AttnRes two-stage fused operators for residual aggregation, and MegaMoE/MC2 for MoE routing and computation—the underlying stack can quickly assemble the foundational skeleton for model execution.

On this basis, combining hybrid parallelism strategies, communication optimization, multi-stream orchestration, and deep low-level techniques such as SuperKernel fusion, graph-mode execution, compilation caching, and NZ (Natural Z-order) weight layout—while continuing to use speculative inference and state replay mechanisms—CANN reshapes the inference engine across four dimensions: compute, communication, scheduling, and storage. This “building-block” style of high-level reuse is the key to ensuring Day0-level rapid deployment of new models on the Ascend platform.

2. Operator Fusion and Multi-Stream Parallelism to Reduce Decoding Latency

In the sparse attention computation pipeline, the execution efficiency of the indexer and sparse attention modules directly determines overall decoding latency. The original step-by-step serial computation is accompanied by extensive intermediate result write-backs to memory and frequent operator launch overhead.

2.1 Core Pipeline Operator Fusion

To eliminate redundant overhead, low-level optimization fuses discrete operations—relevance scoring, head-weight weighting, and Top-K selection—into a single composite operator. This fused operator reads directly from paged KV caches, avoiding repeated tensor movement. Meanwhile, the sparse attention operator reads the main attention KV precisely according to the selected indices. This fused path fully supports both BF16 and quantized data types in both Prefill and Decode phases, balancing precision and speed.

2.2 Three-Stream Collaborative Prolog Parallelism

During the input preparation phase, traditional serial execution logic leads to severe compute idleness. The “Indexer Prolog” mechanism introduces three-stream collaboration: Q projection and K normalization, Q quantization and head-weight projection, and other mutually independent operations are dispatched to different compute streams for parallel execution. This instruction-level parallel scheduling significantly reduces serial wait time during input preparation, keeping the GPU/NPU pipeline fully saturated.

The following simplified pseudocode illustrates the multi-stream scheduling logic:

# Pseudocode: Three-stream collaborative execution logic
stream_compute_a = get_stream("compute_a")
stream_compute_b = get_stream("compute_b")
stream_indexer   = get_stream("indexer")
# Stream A: Parallel Q projection and K normalization
with stream_compute_a:
    q_proj = compute_q_projection(input_tensor)
    k_norm = compute_k_normalization(k_buffer)
# Stream B: Parallel Q quantization and head-weight projection
with stream_compute_b:
    q_quant = quantize(q_proj)
    head_weight_proj = compute_head_weight(weights)
# Stream C: Execute index selection after dependencies are satisfied
with stream_indexer:
    wait_event(stream_compute_a, stream_compute_b)
    top_k_indices = lightning_indexer(q_quant, head_weight_proj, paged_kv_cache)

3. Hierarchical Dynamic Quantization to Reduce Memory and Bandwidth Pressure

During large model inference, massive KV caches are a major consumer of memory and bandwidth. To aggressively compress memory access overhead while preserving computational accuracy, this solution adopts a fine-grained hierarchical quantization strategy.

1. High-precision projection computation: In the core projection computation stage of DSA, BF16 data type is retained to ensure lossless model representation capability.
2. MXFP8 dynamic quantization in the indexing stage: The lightweight indexer fully adopts MXFP8 (Microscaling FP8) format with per-group dynamic quantization, where the group size is set to 32—meaning every 32 adjacent elements share one scaling factor. This fine-grained approach better adapts to intra-tensor data distribution differences, significantly reducing memory footprint while avoiding major accuracy degradation.
3. FP8 block quantization in the attention stage: When the sparse attention computation reads KV cache, FP8 format is used with quantization granularity refined to per-token-head-tile, where the block size is 128. This means independent block quantization is applied per 128 elements within each token and each attention head. This strategy effectively breaks the propagation of long-sequence accumulated errors and substantially reduces storage and bandwidth pressure for both index data and KV caches.

4. Context Parallelism and Zigzag Load Balancing to Break Through Long-Sequence Bottlenecks

In ultra-long context scenarios, the memory and compute of a single accelerator card cannot support complete sequence computation on its own. Deploying with a combination of Context Parallelism (CP) and Tensor Parallelism (TP) is the necessary path to overcome single-card limitations.

4.1 CP×TP Combined Partitioning Strategy

During deployment, the model partitions long inputs along the sequence dimension, assigning different sequence chunks to different CP groups; within each CP group, TP further partitions by attention head. This dual-partitioning mechanism drastically reduces activation and intermediate tensor memory consumption per card.

4.2 Zigzag Layout for Causal Compute Load Skew

In causal index computation, there is an inherent physical property: the later a position in the sequence, the longer its history that needs to be attended. This means that with direct uniform sequence partitioning, cards handling later sequences carry far heavier loads than those handling earlier sequences, creating a serious bottleneck effect.

To address this load skew, this solution introduces a Zigzag layout strategy. When assigning compute tasks, two sequence segments paired head-to-tail are allocated to the same card. As a result, the compute of long-history sequences and short-history sequences is balanced within each card, achieving perfect cross-card equilibrium in index computation. After computation, efficient communication operations restore the original order.

5. DSL-Driven Operator-Level Splitting for Long Sequences

Ultra-long sequences not only stress memory but also directly impact Time to First Token (TTFT). In the state recurrence module, Prefill-phase compute time grows non-linearly with sequence length. If only traditional TP is used to partition by attention head, each card still needs to process the full ultra-long sequence, and TTFT cannot be effectively controlled.

To tackle this, the underlying stack implements FlashKDA CP (Context Parallelism) capability based on a custom operator Domain-Specific Language (CANNBot-DSL). This capability reaches inside the operator itself, further partitioning long-sequence inputs along the sequence dimension and distributing long-sequence computation across multiple cards in parallel. Each card first pre-computes its assigned sub-sequence in parallel, followed by lightweight cross-card communication to synchronize state.

The deployment of this operator-level splitting technique not only significantly reduces TTFT in ultra-long sequence scenarios but also further compresses per-card activation and intermediate tensor memory usage, providing solid low-level support for models with million-token contexts. Combined with the elastic compute and high-performance container platform offered by Baidu AI Cloud, developers can easily build complex inference services that support ultra-long inputs and address the growing demand for long-text processing.

Conclusion

Day0 inference support for next-generation large models is not simply about interface adaptation—it is a deep convergence between underlying compute architecture and model algorithms. By reusing historical high-performance operators, pursuing aggressive operator fusion and multi-stream orchestration, introducing fine-grained hierarchical quantization, and combining CP with Zigzag load balancing, CANN has built a high-performance, low-latency inference foundation for complex new-generation large models. Looking ahead, as operator compilation technology and automated parallelism strategies continue to evolve, the adaptation cycle of underlying infrastructure for new architectures will be further compressed, helping developers stay at the forefront of the AI wave.