Day0 Inference Adaptation and Performance Optimization for a New Architecture Domestic Large Language Model on Ascend

1. Background: Why “Day0” Support Matters

LLM iteration is accelerating. New architectures often go from announcement to open-source weights in a matter of weeks. For an inference platform, the real challenge is not merely “can it run?” but “can it deliver usable, high-performance inference on the day the model is released?” This capability is known as Day0 support: when model weights become public, the inference framework has already completed operator adaptation, parallel-strategy configuration, and critical-path optimization, so developers can deploy and evaluate on the target hardware without waiting weeks for feature gaps to be filled.

A recently released open-source LLM version builds on its predecessor while introducing two interleaved attention mechanisms and a cross-layer representation aggregation structure. This hybrid architecture poses three challenges for the inference system:

1. The attention computation path is split into multiple stages, raising operator-fusion opportunities and scheduling complexity.
2. Sparse attention introduces an index-filtering step that creates a new serial bottleneck in the input-preparation phase.
3. In long-sequence scenarios, key-value (KV) cache size balloons, making storage and memory-access overhead the primary constraint on decoding efficiency.

This article walks through the adaptation practice on the Ascend platform, covering reusable optimization methods across computation, communication, scheduling, and storage.

2. Architecture Analysis: Inference Characteristics of Hybrid Attention

The core change in the new version is the interleaved use of two attention mechanisms.

State-Recurrence Attention (KDA). This variant does not rely on a full historical KV cache. Instead, it processes the sequence token by token via state recurrence, so memory consumption grows gently with sequence length—well suited for long sequences. The trade-off is strict temporal dependency in computation: batching on a single device is limited, and Prefill time scales linearly with sequence length.

Sparse Attention (DSA). Built on Multi-head Latent Attention (MLA), DSA adds an index-selection mechanism. A lightweight indexer (Lightning Indexer, LI) scores historical positions for relevance and selects the Top-K key positions. A Sparse Flash Attention (SFA) module then reads only those positions from the KV cache to complete the attention computation. This reduces the attention workload from the full history to a sparse subset, but it introduces a three-step “score–filter–read” pipeline. If each stage is implemented independently, intermediate-result write-backs and kernel-launch overhead noticeably slow decoding.

The model also continues to use attention residuals to aggregate cross-layer representations, which likewise offers fuseable element-wise computation paths.

From an inference standpoint, three characteristics shape the optimization direction: KDA needs parallel partitioning for long sequences; DSA needs multi-operator fusion in the index pipeline to hide input-preparation latency; and the overall architecture should reuse the operators and scheduling logic already mature in the previous generation rather than reinventing them.

3. Adaptation Strategy: Reusing Existing Optimization Assets

The new architecture is not a ground-up rewrite. Because the model family already has extensive low-level optimizations in its predecessor, the new version’s adaptation can build on existing assets through a three-layer inheritance model: operator reuse, scheduling reuse, and storage reuse.

Operator-layer reuse. Fusion operators accumulated for KDA in prior generations (e.g., FlashKDA, MegaKDA) can directly serve the new KDA path. Two-stage fusion operators for attention residuals continue to support cross-layer aggregation. MoE-related fusion operators and communication–computation overlap schemes carry over to the new FFN components. Reuse requires semantic interface alignment rather than binary compatibility, so shape inference and boundary-condition verification must be completed during adaptation.

Scheduling-layer reuse. Hybrid parallel strategies (combinations of tensor, sequence, and pipeline parallelism), communication optimizations, multi-stream orchestration, SuperKernel fusion, graph-mode execution, compilation caching, and NZ weight layouts are all decoupled from specific attention variants and can be carried over directly. Compilation caching and NZ layout deliver particularly noticeable benefits in first-load latency and memory-access efficiency.

Mechanism-layer reuse. Speculative inference and state-replay mechanisms remain applicable: the former uses a draft model or draft heads to reduce decoding steps, while the latter recovers recurrence state. Both target decoding-phase latency.

This inheritance collapses new-model adaptation from “full operator development” to “incremental operator development + interface alignment + policy tuning”—the engineering foundation that makes Day0 support feasible.

4. Key Optimization 1: DSA Pipeline Fusion and Multi-Stream Coordination

DSA inference efficiency hinges on compressing the index pipeline into as few kernel launches as possible.

Indexer fusion. Relevance scoring, head-weight weighting, and Top-K selection are fused into a single operation that reads paged KV cache directly, avoiding repeated write-backs of intermediate results to global memory. After fusion, the index stage collapses from multiple launches into one, cutting launch overhead and memory round-trips. Both Prefill and Decode phases support BF16 and quantized paths, enabling easy switching across precision requirements.

Sparse attention read. SFA reads the main attention KV cache according to the position set output by the indexer, computing attention only on the selected sparse subset. Because read addresses are determined dynamically by the index results, the paged-cache layout must align with the indexer’s addressing scheme to avoid extra rearrangement cost.

Three-stream coordination in input preparation. In the pre-indexer phase, there are originally serial dependencies between Q projection and K normalization, and between Q quantization and head-weight projection. Three-stream parallel scheduling assigns independent subtasks to different execution streams, compressing serial wait time. This matters most in Decode, where per-step compute is small and input-preparation overhead is a larger fraction of total latency.

Illustrative pseudocode (expressing only fusion and parallelism, not real APIs):

# Index pipeline fusion (illustrative)
idx_scores = fused_indexer(q, k_cache_page, head_weight)  # score + weight + TopK in one call
sparse_out = sfa(q, kv_cache, idx_scores.selected_positions)
# Input-preparation three-stream parallel (illustrative)
with stream_a:
    q_proj = linear_q(x)
with stream_b:
    k_norm = rmsnorm(k)
with stream_c:
    q_quant, head_w = quant(q_proj), proj_head_w(x)
sync_all()

5. Key Optimization 2: Layered Quantization to Reduce Cache and Transfer Overhead

The benefits of sparse attention are partially offset by the storage cost of index data and KV caches. Quantization must therefore be designed in layers by data importance rather than applying a single precision across the entire chain.

Projection computations stay high-precision. DSA projections continue to use BF16 to preserve numerical stability in index scoring and prevent Top-K selection drift caused by low-precision quantization.

Index data is fully quantized. Index-side Q/K use MXFP8 format with per-group dynamic quantization: every 32 elements share one scale factor. Group-level quantization balances precision and overhead—too coarse amplifies quantization error, too fine inflates the cost of storing and reading scale factors.

KV cache uses FP8. SFA’s KV cache is stored in FP8 with a per-token, per-head tile granularity (tile size 128), meaning each token’s each head computes scale factors in blocks of 128 elements. This granularity aligns with the paged-cache physical layout, enabling coordinated sparse-index addressing at read time without cross-block concatenation.

The direct payoff is reduced storage footprint and memory-bandwidth demand for both index data and KV cache, shortening cache-transfer time during decoding. Note that quantization paths must be paired with operator implementations: scale-factor computation and dequantization should be fused into adjacent operators to avoid extra standalone kernels.

6. Key Optimization 3: Prefill Sequence Partitioning and Load Balancing

Under long-input scenarios, activation and intermediate-tensor consumption in Prefill is the main single-device bottleneck. This adaptation supports combined parallelism: long inputs are partitioned along the sequence dimension, with further sequence parallelism inside the tensor-parallel group, so per-device activation and intermediate-tensor size scales down with the partition factor.

However, DSA’s causal index computation has an inherent load imbalance: the later a position in the sequence, the longer its visible history and the greater its indexing workload. With contiguous partitioning, tail segments carry significantly more load than head segments, causing cross-device waiting.

Zigzag layout mitigates this imbalance by pairing a head segment with a tail segment on the same device, so each device holds both a light and a heavy segment and cross-device indexing workload tends toward equilibrium. After computation, the sequence is reordered to its original layout to preserve positional semantics for subsequent attention. The cost is one extra concatenation and reordering pass, but compared with the long-tail waiting caused by imbalance, the net benefit is positive.

7. Key Optimization 4: KDA Sequence Parallelism for Ultra-Long Sequences

KDA’s Prefill compute time grows with sequence length, directly affecting Time-To-First-Token (TTFT). With tensor parallelism that only splits across attention heads, each device still processes the full sequence, so per-device compute cannot be amortized for long sequences.

The solution is to further partition the KDA input along the sequence dimension, distributing long-sequence computation across multiple devices. Because KDA has state-recurrence semantics, partitioned devices must handle cross-segment state transfer: each device first computes local results for its segment in parallel, then sequentially merges recurrence states, and finally applies cross-segment corrections. This is a classic “parallel-first, then reduce” pattern that requires decomposing the recurrence process and stitching states together.

This adaptation uses a domain-specific language (DSL) toolchain to express KDA sequence parallelism—partitioning, local computation, state merging, and result correction—as a compilable operator composition, reducing the complexity of hand-written cross-device state synchronization. Benefits include lower TTFT for ultra-long sequences and reduced per-device activation and intermediate-tensor consumption, freeing memory for even longer contexts.

8. Deployment Recommendations and Summary

To organize the above optimizations into a reusable deployment path, follow this sequence:

1. Run first, tune later. Use existing operators and graph mode for functional verification, confirming numerical correctness across all hybrid-attention stages—especially the consistency of index Top-K selection and sparse reads.
2. Enable compilation caching and weight layout. Freeze compilation artifacts after the first load to avoid repeated compilation latency; use NZ weight layout to match the underlying compute unit’s memory-access pattern.
3. Select parallelism by scenario. For short sequences with high concurrency, prioritize tensor parallelism and multi-stream orchestration. For long sequences, enable sequence partitioning with Zigzag load balancing. For ultra-long sequences, layer on KDA sequence parallelism.
4. Enable quantization incrementally. First verify the impact of index quantization on Top-K recall, then progressively enable KV-cache FP8, keeping precision and performance issues separate.
5. Focus on fixed overhead in decoding. Speculative inference and state replay should work with multi-stream orchestration to compress per-step launch and synchronization cost.

In summary, Day0 inference support for a new architecture is not about a single breakthrough. It comes from reusing mature optimization assets, incrementally fusing new pipelines, and targeting long-sequence bottlenecks. Operator fusion compresses DSA launch overhead; layered quantization controls cache cost; sequence parallelism and load balancing relieve latency and memory pressure for long inputs; and compilation caching, graph mode, and weight layout run throughout. For developers, the transferable takeaway is a framework—pipeline fusion → precision layering → parallel partitioning → load balancing—that can organize optimization work whenever a new model introduces hybrid attention and sparse mechanisms, shortening the path from model release to production readiness.