Day0 Inference Adaptation and Performance Optimization for New-Architecture LLMs

1. Background: Why “Day0 Support” Has Become a Hard Requirement for Inference Engineering

The release cadence of open-source large language models has accelerated noticeably in recent years. The window from weight release to usable inference has been compressed to a matter of days. For deployment teams, “Day0 support” carries two distinct meanings: first, availability — completing the full path from weight loading and operator adaptation to end-to-end inference on the same day a model is released; second, deliverability — achieving production-grade performance rather than merely “it runs, but far too slowly to be usable.”

Achieving both goals within such a tight window is not a matter of last-minute effort. It depends on systematic accumulation across the inference infrastructure stack:

• Operator assets: Highly fused, high-performance implementations already exist for common structures such as various attention variants, MoE, normalization, and low-precision computation.
• Scheduling assets: System-level capabilities including hybrid parallelism, communication optimization, and multi-stream orchestration can be reused across models.
• Compilation assets: Graph mode and compilation caching spare new models from repeated graph optimization and compilation waits.
• Methodology assets: A standard process exists for handling new structures: structural difference analysis → operator mapping → accuracy alignment → performance tuning.

This article uses a typical hybrid-architecture open-source model as a case study. The model interleaves state-recurrent linear attention with sparse attention and incorporates cross-layer residual aggregation. We break down the complete Day0 adaptation and performance optimization path. All techniques described are general engineering practices that can be transferred to other new-architecture model onboarding efforts.

2. Breaking Down the New Architecture: A Hybrid Design with Two Computational Paths

2.1 The Linear Attention Path: State Recurrence for “Full-Sequence Memory”

This type of module does not retain per-token KV cache. Instead, it maintains a fixed-shape recurrent state: for each incoming token, the state is updated once using a Delta Rule–style operation. Its compute and memory overhead grows approximately linearly with sequence length, making it a natural fit for compressing full-sequence history in long-context scenarios.

From an engineering perspective, it has three key characteristics:

• Recurrent serial dependency: State updates have sequential dependencies, making naive implementations hard to parallelize. This is a primary source of prefill latency.
• Chunk-wise composability: When the sequence is split into chunks, the transformation each chunk applies to the initial state satisfies the associative property. This provides the mathematical foundation for chunked scans and cross-device parallelism.
• Memory-access friendliness: The state is a fixed-size dense tensor, well suited for resident on-chip storage and deep operator fusion.

2.2 The Sparse Attention Path: Indexer + Sparse Attention Core

The other type of module builds on multi-head latent attention — a structure that compresses KV into a low-rank latent space to reduce cache size — and introduces an explicit index selection mechanism:

1. A lightweight indexer scores each historical position for relevance and selects the Top-K most relevant positions.
2. A sparse attention core reads only the selected KV entries and performs sparse attention in a chunked, IO-aware manner.

Intuitively, this design resembles “taking notes while reading, then revisiting the original text when needed”: linear attention continuously maintains a summary (the recurrent state), while sparse attention precisely revisits key passages (Top-K positions) when necessary. The two are interleaved, striking a balance between representational capacity and inference cost.

2.3 Cross-Layer Residual Aggregation

The model also aggregates cross-layer representations through attention residuals, allowing deeper attention layers to directly access the attention outputs of shallower layers. In engineering terms, this means the main attention computation must additionally fuse a cross-layer read. If implemented poorly, this introduces extra memory round-trips and kernel launch overhead.

3. Reusing Existing Optimizations: Treating New-Model Adaptation as “Structural Difference Adaptation”

The core methodology of Day0 adaptation is to treat a new model not as a from-scratch problem, but as a set of differences relative to previously adapted models. The typical process has five steps:

1. Structural diffing: Compare the new model module by module against previously adapted models, classifying each module into one of three categories: “fully reusable,” “reusable with different parameters,” or “requires new development.”
2. Operator mapping: Directly map reusable modules to existing fused operators. For new modules, extend existing operator skeletons through parameterization wherever possible.
3. Accuracy alignment: Compare outputs layer by layer between the reference implementation and the target platform to determine quantization boundaries for each module.
4. Parallelism strategy: Choose a combination of TP/CP/EP based on sequence length, batch size, and cluster topology.
5. Performance tuning: Enable and validate the benefits of multi-stream orchestration, communication hiding, and compilation caching one by one.

In this case, most modules of the new model directly inherit optimization assets accumulated from the previous generation:

• Operator layer: The chunked scan fused operator for linear attention (performing state recurrence and output computation within a single kernel), the large-granularity MoE operator with cross-device communication co-scheduling, and the two-stage fusion of attention residuals can all be transferred directly.
• Scheduling layer: Hybrid parallelism, communication optimization, and multi-stream orchestration are reused as-is.
• Compilation layer: Super-operators (fusing multiple adjacent small operators into a single large kernel), graph-mode execution, and compilation caching continue to work. After the first compilation, the new model can reuse the cache.
• Storage layer: The NZ-style blocked weight layout optimized for matrix compute units carries over, preserving weight access efficiency.
• Inference acceleration: Speculative decoding and the accompanying state replay mechanism remain in effect — when draft tokens are rejected, the recurrent state of linear attention can be quickly restored through replay, avoiding full recomputation.

The direct benefit of this asset-reuse approach is that the number of operators requiring genuinely new development is reduced to a minimum — mainly the sparse attention path and the CP extension for linear attention. The adaptation cycle is compressed from weeks to days.

4. The Sparse Attention Path: Fusion and Multi-Stream to Reduce Decode Latency

The sparse attention path is the part of the new architecture requiring the most significant new development, and it lies on the critical path for decode latency.

4.1 Indexer Fusion: Compressing Three Steps into One

In a naive implementation, the indexer involves at least three separate computations and two intermediate result writes: relevance scoring → per-head weighting → Top-K selection. The fused implementation compresses this into a single operator:

# Fused indexer operator (illustrative)
def fused_indexer(q_idx, paged_k_cache, head_weights, top_k):
    scores = batched_dot(q_idx, paged_k_cache)   # Relevance scoring, reads paged K cache directly
    scores = weighted_sum(scores, head_weights)  # Per-head weighted aggregation
    return topk(scores, top_k)                   # Top-K indices

Fusion delivers three benefits: intermediate results no longer need to be written back to device memory; kernel launch count drops from three to one; and the scoring stage reads the paged K cache directly, avoiding materializing a separate copy of K data for the indexer.

The sparse attention core then gathers the corresponding positions from the main attention KV based on the selected indices and performs chunked, parallel sparse attention computation. Both BF16 and quantized paths are provided for prefill and decode stages, allowing flexible trade-offs between accuracy and performance.

4.2 Three-Stream Coordination for Indexer Preprocessing

During decode, every token must pass through the indexer. If the input preparation steps (Q projection, K normalization, Q quantization, head weight projection) are executed serially, they add a substantial fixed overhead. The optimization uses three streams to run independent computations in parallel:

Stream 1: q_proj  = W_q @ x            # Q projection
Stream 2: k_norm  = rmsnorm(k_cache)   # K normalization
Stream 3: q_quant = quant(q_proj)      # Q quantization
          w_proj  = W_h @ x            # Head weight projection
join(Stream 1, Stream 2, Stream 3)     # Merge before entering the fused indexer

The original serial chain is compressed to the duration of the longest parallel branch, directly reducing the wait time at each decode step. This type of multi-stream orchestration is not tied to specific hardware; it is a general technique applicable to any inference platform that supports asynchronous stream execution.

5. Layered Quantization: Designing Precision and Granularity by Data Role

Different types of data in the new architecture play very different roles. Applying a uniform precision wastes both compute and bandwidth. A layered quantization strategy is therefore adopted.

5.1 Index Path: Fully Quantized with MXFP8 Per-Group

• Q and K on the index side use the MXFP8 format overall, with per-group dynamic quantization and a group size of 32 — every 32 elements share a scaling factor.
• MXFP8 belongs to the microscaling floating-point format family: it builds on FP8 numerical representation and shares exponents at small-group granularity, balancing dynamic range and hardware implementation efficiency. It has become a widely adopted low-precision option in the industry.
• The index path can afford full quantization because Top-K selection is fundamentally a ranking problem, inherently robust to small score perturbations — as long as quantization does not change the composition of the Top-K set, the results of the subsequent sparse attention remain unaffected.

5.2 KV Cache: FP8 Per-Token-Head-Tile

The KV cache read by the sparse attention core uses FP8 with per-token-head-tile quantization granularity and a tile size of 128: within each token and each head, elements are quantized in blocks of 128. Compared with coarser granularity, per-block scaling factors better adapt to local value distributions and limit accuracy loss. Compared with finer granularity, the 128-element block size aligns with common vector memory-access widths, keeping scaling factor reads and dequantization overhead manageable.

5.3 Main Computation Kept in BF16

QKV projection and the main attention computation remain in BF16 to preserve overall output quality. The overall strategy can be summarized as “main computation keeps precision, index and cache squeeze overhead” — storage and transfer overhead for index data and KV cache drop significantly, and these are precisely the primary bandwidth bottlenecks of the sparse attention path.

6. Prefill Context Parallelism: CP×TP Deployment and Zigzag Load Balancing

6.1 CP×TP Combined Deployment

For long inputs, per-device memory and compute pressure during prefill grow rapidly with sequence length. This solution supports CP×TP combined deployment: splitting a long input across multiple devices along the sequence dimension (context parallelism, CP), combined with sequence parallelism within the TP group, substantially reduces per-device activation and intermediate tensor footprint, enabling longer inputs and higher concurrency.

6.2 Zigzag Layout: Addressing Load Imbalance in Causal Index Computation

Index computation for sparse attention is causal: the later a sequence position is, the longer its visible history, and the more computation its index scoring requires. If the sequence is split contiguously, later devices inevitably become the bottleneck. The zigzag layout solves this through “head-tail pairing,” assigning symmetric front and back segments to the same device:

Sequence chunks: [0] [1] [2] [3] [4] [5] [6] [7]
Device 0: 0 + 7      Device 1: 1 + 6
Device 2: 2 + 5      Device 3: 3 + 4

Each device now holds “one short prefix segment + one long prefix segment,” bringing cross-device index computation closer to balance. After computation, results are restored to the original order. This idea is consistent with the zigzag partitioning commonly used for causal attention load balancing and is a broadly reusable general technique.

7. CP Extension for Linear Attention: Tackling TTFT for Ultra-Long Sequences

The state recurrence of linear attention has inherent sequential dependencies. Prefill latency grows with sequence length, directly impacting time-to-first-token (TTFT). More problematically, if only TP is used to split by attention heads, each device still processes the full sequence, so computation cannot be distributed along the sequence dimension. The CP approach solves this in two steps:

1. Split the input along the sequence dimension, distributing the computation of a long sequence across multiple devices.
2. Leverage the composability of state updates with a two-stage execution: each device first independently runs the recurrence over its local chunk to obtain a state delta, then the deltas are merged in sequence order to produce the initial state for each chunk, and finally each device computes its local output in parallel based on the merged state.

# Linear attention CP (illustrative)
chunks = split(seq, P)                              # Split sequence across P devices
incs   = parallel_map(scan_delta, chunks)           # Stage 1: compute state deltas in parallel
prefix = sequential_merge(incs)                     # Stage 2: merge to get initial state per chunk
outs   = parallel_map(apply_state, chunks, prefix)  # Stage 3: compute outputs in parallel

The benefits come in two dimensions: TTFT for ultra-long sequences decreases as the number of devices participating in partitioning grows; per-device activation and intermediate tensor footprint decrease correspondingly, making previously unprocessable long sequences feasible. In terms of engineering implementation, this type of new operator can be developed using a domain-specific language (DSL): the chunked recurrence logic is described in code close to the mathematical formulation, and the compiler handles scheduling and vectorization. This significantly shortens the iteration cycle from new-operator design to deployment.

8. Summary: A Reusable Day0 Adaptation Checklist

The optimization techniques from this exercise, organized by the stage they affect:

Stage	Optimization	Goal
Compute	Three-step indexer fusion, chunked sparse attention, chunked scan for linear attention, cross-layer residual fusion	Reduce kernel launches and intermediate writes
Quantization	Index MXFP8 per-group (32), KV cache FP8 per-token-head-tile (128), main computation BF16	Reduce storage and memory access while preserving main precision
Communication	MoE operator with communication co-scheduling, sequence splitting under CP/TP	Hide communication, distribute long-sequence pressure
Scheduling	Three-stream coordination, multi-stream orchestration, graph mode, compilation caching	Reduce serial waiting and compilation overhead
Storage	Paged KV cache, NZ-style blocked weight layout	Improve memory access efficiency
Acceleration	Speculative decoding + state replay	Reduce decode latency, quickly restore recurrent state

In retrospect, “Day0 support” is not the victory of any single technique. It is a systematic mobilization of existing optimization assets: structural difference analysis determines “what needs to be done”; operator, scheduling, compilation, and storage assets determine “what can be reused”; and the few remaining gaps — the sparse attention path and linear attention CP — are filled one by one using general techniques such as fusion, quantization, and load balancing. For adapting new-architecture models, this “translate + reuse + fill gaps” methodology is more worth codifying than any individual trick.