Day-0 Inference Support and Performance Optimization for a New Large Model Release

1. Background: Deployment Challenges of a New Model Architecture

A recent large model release introduced a new architecture and was made openly available. Its inference framework achieved Day-0 support and optimization on the release day, allowing developers to deploy and evaluate the model quickly on Ascend platforms. For inference engines, every model architecture iteration means re-adapting the computation graph, memory access patterns, and parallelization strategy. The core engineering challenge is completing end-to-end support for the new model within a limited time while maintaining inference performance.

This article examines the architectural characteristics of the model and the accompanying optimization approach, covering the full technical path from operator fusion and quantization to parallel orchestration. It is intended as a reference for developers adapting similar models in their own environments.

2. Architecture Analysis: Computational Characteristics of a Hybrid Attention Mechanism

The attention layer in this model version uses a dual-path design with interleaved execution. One path processes sequence information through state recurrence and exhibits linear complexity. The other path builds on Multi-head Latent Attention (MLA) and adds an index selection mechanism: a lightweight indexer first selects the Top-K historical positions, and then a sparse attention module reads the corresponding KV cache entries to perform sparse attention. In addition, the model continues to aggregate cross-layer representations through an attention residual mechanism, improving the transmission efficiency of deep-layer information.

This hybrid architecture presents a direct challenge: the computation graph contains both recurrent operators and sparse attention operators, and these two categories differ in parallelization dimensions, memory access granularity, and quantization friendliness. With a conventional operator-by-operator scheduling approach, problems such as high operator launch overhead, repeated write-back of intermediate results, and long decoding-stage latency are likely to occur.

3. Optimization Overview: Reusing Accumulated Capabilities for Rapid Adaptation

Enabled by optimization work accumulated during the previous model generation, the new version was able to quickly reuse several key capabilities:

• Operator-level optimization: including fused operators for the state recurrence path, two-stage attention residual fusion, and fused operators with communication-computation overlap for the mixture-of-experts structure.
• Parallelization and scheduling: hybrid parallelism, communication optimization, and multi-stream orchestration, combined with a super kernel, graph-mode execution, compilation caching, and NZ weight layout.
• Inference acceleration mechanisms: speculative inference and state rollback to reduce serialized overhead during decoding.

Together, these capabilities improve inference efficiency across computation, communication, scheduling, and storage, allowing the new architecture to be adapted quickly without large-scale rewrites.

4. Sparse Attention Fusion and Multi-Stream Coordination

4.1 Fused Execution of the Indexer

The lightweight indexer performs three tasks: relevance scoring, head-weight weighting, and Top-K selection. In a conventional implementation, these steps are split into multiple operators, with intermediate results written back to global memory. The optimized approach fuses them into a single execution: it reads the paged K cache directly, completes scoring and filtering on-chip, and reduces both intermediate write-backs and operator launches. The sparse attention module then reads the main attention KV according to the selected indices and performs sparse computation. In both the prefill and decode phases, the indexer supports BF16 and quantized paths.

4.2 Three-Stream Coordination for Input Preparation

The input preparation stage of the indexer contains several parallelizable subtasks. Through three-stream coordination, Q projection and K normalization are placed on one stream, while Q quantization and head-weight projection are placed on another, allowing parallel execution and reducing serialized waiting during input preparation. This optimization is particularly important in the decode phase, where per-step computation is small and any serialized step is magnified into visible latency.

5. Layer-wise Quantization: Reducing Cache and Data Movement Overhead

The sparse attention path involves substantial memory traffic for index data and KV cache access. Quantization is a direct way to reduce this overhead. The approach uses a layer-wise strategy:

• Projection computation remains in BF16: projections within sparse attention stay in BF16 precision to prevent quantization error from accumulating on the critical path.
• Indexer uses full MXFP8 quantization: index Q/K are dynamically quantized per group with a group size of 32, meaning every 32 elements share one scaling factor. This granularity balances precision and overhead.
• KV cache uses FP8: quantization granularity is per-token-head-tile, with a tile size of 128, meaning quantization is applied in blocks of 128 elements within each token and each head.

This strategy significantly reduces storage and memory traffic for index data and KV cache. In long-sequence scenarios especially, KV cache growth is the main source of memory consumption, and block-wise quantization effectively relieves this pressure.

6. Sequence Parallelism and Load Balancing

6.1 Combined CP and TP Deployment

This release supports combined CP×TP deployment: long inputs are split along the sequence dimension, and sequence parallelism within TP groups further reduces per-device activation and intermediate tensor usage. For very long sequence inputs, this combination distributes sequence-dimension computation pressure across more devices while maintaining attention head parallelism efficiency.

6.2 Zigzag Layout for Balanced Causal Index Workload

Causal index computation in sparse attention has an inherent load imbalance: later sequence positions can attend to more historical positions, resulting in a heavier index computation load. If sequences are partitioned contiguously, devices handling later segments bear significantly more computation.

To address this, the approach uses a Zigzag layout: paired segments from the beginning and end of the sequence are assigned to the same device. One segment has a light index computation load, while the other has a heavy load, so the paired segments balance the overall computation across devices. After computation, the original order is restored to ensure correctness.

7. Very Long Sequence Support Through a Domain-Specific DSL

For very long sequence scenarios, the prefill computation time of the state recurrence path grows with sequence length, directly affecting time-to-first-token (TTFT). If only TP is used to shard by attention head, each device still processes the full sequence, which does not fundamentally reduce per-device computation.

To address this, the approach implements a sequence-parallel version of this path using a domain-specific DSL: each device first computes the state for its local sequence segment in parallel, and then cross-device communication merges the recurrence results. This distributes long-sequence computation across multiple devices with the goal of reducing TTFT in very long sequence scenarios while also lowering per-device activation and intermediate tensor usage.

The core idea can be summarized as follows:

1. Split the input along the sequence dimension, with each device processing its local segment independently
2. Each device completes local state recurrence computation in parallel
3. Merge cross-segment states through communication to ensure consistent recurrence results
4. Subsequent layers continue computation based on the merged state

8. Engineering Recommendations

For developers looking to adapt similar hybrid model architectures on their own platforms, the following points are worth noting:

1. Prioritize reuse of existing operator libraries: new model operator sets often overlap heavily with the previous generation. Reusing mature fused operators can significantly shorten the adaptation cycle.
2. Focus on decode-phase serialization bottlenecks: decode-phase computation is small, so operator launch and stream synchronization overhead account for a large share of latency. Multi-stream orchestration and operator fusion provide the most visible benefits in this phase.
3. Design quantization strategy in layers: different paths have different precision sensitivity. Keeping key projections in high precision while using low precision for cache and index data is a reliable combination.
4. Long-sequence scenarios require load balancing: causal attention load imbalance directly causes inter-device waiting. Paired layout is a low-cost and effective solution.
5. Sequence parallelism requires communication-computation overlap: if state merging communication cannot overlap with computation, the benefits of sequence parallelism will be offset by communication overhead.

9. Summary

Day-0 support for this model version demonstrates that a well-polished inference optimization system—covering fused operators, layer-wise quantization, hybrid parallelism, load balancing, and compilation optimization—can adapt quickly when a new architecture appears, without building from scratch. For the long-term evolution of an inference engine, accumulating reusable optimization components and abstracting stable domain-specific DSLs have greater engineering value than model-specific custom tuning.