Day0 Inference Support and Performance Optimization for New LLM Architectures

I. Background: Why Day0 Support is a Core Capability for Inference Platforms

In recent years, the pace of innovation in LLM architectures has accelerated significantly. Mechanisms such as gated linear attention, sparse attention with learnable indexes, Mixture of Experts (MoE), and cross-layer residual aggregation are continuously entering mainstream model design. On the inference side, new structures mean new operators, new data flows, and new memory access patterns. The traditional rhythm of “release first, adapt slowly, optimize later” can no longer meet developers’ expectations of deploying and experiencing models on the day of release.

Day0 support typically encompasses three requirements:

• Functional Availability: All computation paths of the new structure must be numerically correct and run stably on the target hardware.
• Performance Availability: Compared to fully optimized mature architectures, the latency and throughput of the new architecture should not suffer significant degradation.
• Scale Availability: The system must remain stable under real-world workloads such as long sequences and high concurrency, without failing due to memory overflow or load imbalance.

Meeting these three requirements on release day is nearly impossible with temporarily hand-written operators; it requires a systematic adaptation methodology. The core challenges fall into three categories:

1. Operator Gaps: New computations like state recursion, Top-K indexing, and sparse KV reads often lack ready-made implementations in existing inference stacks.
2. Long Sequence Pressure: KV cache size, activation memory, and memory bandwidth requirements grow rapidly with sequence length, directly constraining Time To First Token (TTFT) and decoding speed.
3. Parallelism and Load: The computational volume of new structures is unevenly distributed across the sequence and attention head dimensions. Naive partitioning easily creates bottleneck effects.

II. Methodology: Crystallizing Optimizations into Assets, Trading Reuse for Speed

2.1 Four Layers of Optimization Assets

Mature inference engineering teams typically crystallize historical optimizations into four types of assets that are loosely coupled with specific models:

• Operator Assets: Fused attention kernels, recursive operators, two-stage attention residual fusion, and communication-computation fused MoE dispatch operators.
• Scheduling Assets: Multi-stream orchestration, graph mode, compilation caching, and speculative inference frameworks.
• Communication Assets: Collective communication optimization, and primitive encapsulation for overlapping computation and communication.
• Storage Assets: Hardware-friendly tiled weight layouts, paged KV cache management, and recursive state caching.

2.2 General Anatomy of New Architectures

Taking a typical hybrid attention new architecture as an example, its components can be reduced to a few universal technical essences:

• Gated Incremental Recursive Attention: Recurs along the sequence with a fixed-size state. The state remains constant during inference, making it naturally suitable for long sequences.
• Sparse Attention with Learnable Index: Introduces an indexing mechanism on top of low-rank compressed attention. A lightweight indexer scores historical positions and filters Top-K, then a sparse attention kernel reads only the selected KV to complete the computation.
• Cross-Layer Residual Aggregation: Passes shallow representations directly to deeper layers through attention residuals, mitigating information decay in deep networks.

2.3 Mapping New to Old Components: Why Reuse Works

New Architecture Component	Technical Essence	Reusable Existing Optimizations
Gated Recursive Attention	Sequence recursion with fixed state	Flash-style tiled recursive kernel, two-stage fusion
Indexed Sparse Attention	Score - Filter - Sparse Read	Paged KV management, fused attention kernel
Cross-Layer Residual Aggregation	Cross-layer representation concatenation	Two-stage fused operator
Mixture of Experts (MoE) Layer	Group communication + Expert computation	Communication-computation fused dispatch operator
Speculative Inference / State Replay	Scheduling and caching mechanisms	Direct reuse of scheduling framework

Reuse is viable because the underlying optimization principles—reducing memory round-trips, increasing parallelism, overlapping computation with communication, and improving scheduling density—are loosely coupled with specific models. In other words, the key action in adapting a new architecture is not “rewriting from scratch,” but first performing a technical essence mapping to identify which components can hook into existing assets, and then focusing on filling the gaps. Combined with global measures like hybrid parallelism strategies, graph mode, compilation caching, and weight layouts, efficiency gains can be simultaneously realized across computation, communication, scheduling, and storage.

III. Sparse Attention Pipeline: Fusion and Multi-Stream Orchestration to Reduce Decoding Latency

3.1 Integrated Indexer Fusion

In traditional implementations, the “relevance scoring -> head weight weighting -> Top-K filtering” process is often split into three independent operators. Each step requires writing intermediate results back to memory and reading them out again. The fusion approach merges these three steps into a single operator and reads data directly from the paged K cache, yielding three key benefits:

• Elimination of Intermediate Tensors: No more temporary memory usage for score matrices or weighted scores.
• Reduced Operator Launches: Kernel launch counts drop, which is especially beneficial for small-batch decoding scenarios.
• Reduced Memory Round-Trips: Scoring and filtering are completed on-chip, outputting only the Top-K indices.

Meanwhile, both Prefill and Decode paths retain BF16 and quantized implementations, allowing for trade-offs between precision and bandwidth depending on the scenario.

3.2 Sparse Attention Kernel

After the indexer outputs the Top-K positions, the sparse attention kernel “calls out” the corresponding KV from the paged KV cache based on the indices, executing attention calculations only on the selected historical positions. The computation volume and KV read volume scale with Top-K rather than sequence length, which is the fundamental reason sparse attention maintains high throughput under long contexts.

3.3 Multi-Stream Collaborative Input Preparation

Before index calculation, there is an input preparation phase: Q projection, K normalization, Q quantization, head weight projection, and other前置 computations. If executed serially on a single stream, unnecessary waiting occurs. Analyzing dependencies reveals that a significant portion of these subtasks are independent of each other and can be assigned to multiple hardware streams for parallel execution:

# Before Optimization: Single-stream serial
Q      = QProjection(x)            # Step 1
Qq     = Quantize(Q)               # Depends on Step 1
W_head = HeadWeightProjection(x)   # No dependency on Q, but forced to wait
K_norm = KNormalize(paged_kv)      # No dependency on Q, but forced to wait
index  = FusedIndexer(Qq, K_norm, W_head)
# After Optimization: Multi-stream parallel orchestration
stream1: Q = QProjection(x) -> Qq = Quantize(Q)
stream2: W_head = HeadWeightProjection(x)
stream3: K_norm = KNormalize(paged_kv)
join:    index = FusedIndexer(Qq, K_norm, W_head)

By adhering to the orchestration principle of “parallel if no dependency,” serial waiting during the input preparation phase is significantly compressed, allowing the indexer and sparse attention kernel to enter computation earlier.

IV. Hierarchical Quantization: Selecting Precision per Tensor

4.1 Design Principles

The correct approach to quantization is not one-size-fits-all, but rather making decisions separately based on the tensor’s role, guided by three questions: How sensitive is the data to error? Does it constitute a memory bottleneck? Is it on the main computation path or an auxiliary path? Taking the sparse attention pipeline as an example:

• Indexing Path: Scoring and Top-K filtering are insensitive to absolute numerical errors, but data movement is frequent. This is suitable for microscaled FP8 (such as industry-standard formats like MXFP8), dynamically quantized per group—for example, sharing a single scaling factor for every 32 elements, balancing dynamic range and quantization precision.
• KV Cache: This is the bulk of memory usage and read bandwidth. Using FP8 with “Token-Head-Tile” block quantization (e.g., blocking by 128 elements within each Token and Head) controls errors while facilitating local reads during dequantization.
• Projection Computation: Located on the main numerical path, it retains BF16 to prevent error accumulation along network depth.

4.2 Benefit Structure

Data	Precision	Quantization Granularity	Primary Benefit
Index Q/K	MXFP8	per-group (e.g., 32)	Reduces index data storage and bandwidth
KV Cache	FP8	per-token-head-tile (e.g., 128)	Cache size roughly halved, improves read efficiency
Projection Computation	BF16	Unquantized	Ensures numerical stability

It is worth noting that the benefits of hierarchical quantization amplify with context length: the longer the sequence, the higher the proportion of KV cache and index data in total memory access, making bandwidth savings from quantization more significant.

V. Context Parallelism (CP): Partitioning and Load Balancing for Long Sequences

5.1 Combined CP and TP Deployment

For ultra-long inputs, relying solely on Tensor Parallelism (TP) to partition by attention head is insufficient—each card still processes the full sequence, and activation values and intermediate tensors grow linearly with sequence length. Introducing Context Parallelism (CP) partitions the input along the sequence dimension across multiple cards. Combined with sequence parallelism within the TP group, it ensures that the memory footprint for activations and intermediate tensors on a single card scales with “sequence length / number of cards” rather than the full sequence length.

5.2 Causal Index Load Imbalance and Zigzag Layout

Index calculation in sparse attention is causal: the later the token in the sequence, the longer its visible history, meaning more positions to score and a larger computation volume. If partitioned sequentially by position across cards, the card handling the last segment of the sequence becomes an obvious bottleneck. The Zigzag (head-tail pairing) layout solution assigns symmetric head and tail sequence segments to the same card—the first segment and the last segment on one card, the second segment and the second-to-last on another. This ensures each card handles one “light” and one “heavy” workload, balancing the computation volume across cards. After computation, the segments are reassembled in their original order.

5.3 CP Extension for Recursive Attention

The Prefill time for gated recursive attention grows with sequence length, making it a primary bottleneck for TTFT. Because recursive computation is naturally partitionable by attention head, performing only TP means each card still traverses the full sequence. A further CP approach chunks the input along the sequence dimension: performing local recursion in parallel within chunks, and passing and merging states sequentially between chunks (a general tiled scan approach). This distributes the recursive computation of long sequences across multiple cards, aiming to reduce TTFT for ultra-long sequences while decreasing single-card activation and intermediate tensor footprints.

VI. Engineering Infrastructure: Ensuring Stable Optimization Returns

• Graph Mode and Compilation Caching: Full-graph dispatch reduces Host-side scheduling overhead; caching and reusing compilation results improves repeated request and cold start performance.
• Speculative Inference: Uses a small model to draft and a large model to verify, boosting decoding throughput. New architectures only need to adapt the verification-side operators to integrate.
• Recursive State Replay: Caches historical recursive states, avoiding recomputation of historical tokens in multi-turn conversations. This is particularly important for models with state recursion structures.
• Communication and Scheduling Across the Full Pipeline: Computation-communication overlap and multi-stream orchestration should act as global strategies throughout Prefill and Decode, rather than isolated tricks.

VII. Conclusion: A Reusable Day0 Adaptation Checklist

1. Assets First: Crystallize optimizations into four loosely coupled asset categories: operators, scheduling, communication, and storage.
2. Structural Anatomy: When receiving a new architecture, perform technical essence mapping first to identify which components can reuse existing optimizations.
3. Fuse Before Quantize: Prioritize reducing memory access and waiting through fusion and multi-streaming, then gradually introduce low precision.
4. Hierarchical Quantization: Select precision and granularity based on tensor roles; preserve precision on the main path, trade for bandwidth on auxiliary paths.
5. Long Sequence Combo: Use CP partitioning, Zigzag load balancing, and state reuse in concert.
6. Engineering Closed Loop: Leverage graph mode, compilation caching, and speculative inference to ensure benefits are stably realized in real-world business scenarios.

When the iteration speed of model architectures exceeds the rewriting speed of inference stacks, the key to competition is no longer “whether you can write an individual operator,” but “whether you can use a reusable asset system to compress the adaptation of new architectures down to a matter of days.” This methodology does not depend on any specific model or platform and is a valuable addition to the engineering toolbox of any inference system developer.