Day0 Inference Support and Performance Optimization for New-Architecture LLMs

1. Background: Day0 Support Becomes a Hard Requirement as New Architectures Become the Norm

Large model iteration continues to accelerate. On the very day a new-generation model is released with open weights, developers typically expect to deploy and try it on an inference platform—what the industry calls “Day0 support.” This tests not the performance of any single operator, but the engineering maturity of the entire inference infrastructure: whether a new architecture’s computational characteristics can be identified within a very short time, missing operators filled in, existing optimizations reused, and a usable performance baseline delivered on release day.

The challenges introduced by new architectures fall into three areas:

1. New operators: Recurrent state updates for linear attention and index selection for sparse attention are computational forms that do not exist in traditional attention inference stacks.
2. New memory access patterns: KV changes from “full sequential reads” to “sparse reads by index,” requiring cache organization and bandwidth strategies to be redesigned.
3. New parallelism requirements: For ultra-long sequences, the traditional strategy of splitting by attention head cannot amortize computation along the sequence dimension, making context parallelism necessary.

This article uses a new-generation model with a hybrid “linear attention + sparse attention” architecture as a case study, and breaks down the methodology for Day0 adaptation and performance optimization across four dimensions: compute, communication, scheduling, and storage.

2. Architecture Breakdown: What Hybrid Attention Means

The new architecture typically interleaves two types of attention layers, supplemented by a cross-layer aggregation mechanism.

1. Gated delta linear attention: Attention is transformed into a recurrence over a fixed-size matrix state—each token processed first decays the historical state by a gating coefficient, then adds the new information from the current token. Because state size is independent of sequence length, compute and memory overhead in the Decode phase are decoupled from sequence length, which is its fundamental advantage for long contexts. In engineering, two kernel forms are typically maintained: Prefill uses blocked parallel scan to ingest a long input segment at once, while Decode uses single-step recurrence to update token by token.

2. Sparse attention: An indexing mechanism is layered on top of a low-rank compressed KV mechanism such as Multi-head Latent Attention (MLA)—a lightweight indexer first scores all historical positions for relevance, aggregates multi-head scores by learnable weights and selects the Top-K positions, and then the sparse attention kernel reads only the KV at those positions to complete the computation. Attention compute and KV read volume thus shift from proportional to sequence length to proportional to Top-K.

3. Cross-layer residual aggregation: Attention outputs from multiple layers are aggregated with learnable weights between layers, enhancing the representational capacity of the deep network.

For the inference system, this architecture has three direct consequences: KV forms are heterogeneous across layers (linear layers maintain only a small state, while sparse layers store full KV but read it by index); Prefill and Decode require different optimal kernel forms; and index computation introduces a new precision-sensitive memory access path. Adaptation therefore cannot simply apply a generic Transformer inference template—countermeasures must be designed for each of these characteristics.

3. The Foundation for Rapid Adaptation: Operator Assets and Strategy Reuse

New architectures rarely appear from nothing—most are evolutions of the previous generation. The first move of an efficient team is an asset inventory, sorting existing optimizations into three categories:

1. Direct reuse: Blocked scan and recurrence kernels for linear attention, two-stage fusion of attention residuals, communication-compute fused operators for MoE scenarios, and similar assets—these can be integrated into the new model as long as interfaces and numerical behavior align.
2. Reuse with modification: Hybrid parallelism strategies (combined configurations of tensor parallelism, expert parallelism, and pipeline parallelism), multi-stream orchestration schemes, and similar assets need to be rearranged according to the new model’s layer structure.
3. New development: Brand-new operators such as the indexer and sparse attention, along with their quantization paths—these are the parts that genuinely require focused effort in Day0 adaptation.

At the engineering efficiency level, three techniques can significantly compress the adaptation cycle:

• Graph mode: The entire computational graph is dispatched and executed in one go, avoiding per-operator scheduling overhead and making fusible subgraphs easier for the compiler to identify.
• Compilation cache: Kernel artifacts from the first compilation are cached and reused, freeing service restarts from repeated compilation and greatly shortening cold start time.
• Hardware-friendly weight layout: Weights are stored and loaded in blocked-transposed form, eliminating runtime format conversion overhead.

The “reuse first, add new as needed” strategy is the key to compressing adaptation work from “rewriting the inference stack” to “filling in the increments,” and is also the precondition for Day0 feasibility.

4. Decode Latency Optimization: Eliminating “Waiting” with Fusion and Multi-Stream

The goal of the Decode phase is to minimize per-token latency, and the core idea is to reduce all serial waiting.

1. Three-in-one indexer fusion. If relevance scoring, multi-head weighted aggregation, and Top-K selection are each implemented as separate kernels, this produces multiple intermediate tensor write-backs to memory and multiple kernel launches. Fused into a single kernel, data stays on-chip throughout and is read directly from the paged K cache, avoiding the need to materialize an extra K copy for index computation:

# Pseudocode: fused indexer kernel
def fused_indexer(q, paged_k_cache, w_head, top_k):
    scores = blockwise_qk(q, paged_k_cache)   # 1. Blocked scoring, on-chip residency
    scores = weighted_sum(scores, w_head)     # 2. Multi-index-head weighted aggregation
    return topk(scores, k=top_k)              # 3. Top-K selection completed on-chip

1. Sparse attention reads by index. The sparse attention kernel gathers the corresponding entries from the main KV cache according to the position set output by the indexer, and computes only over the selected KV—no data movement occurs for the rest.

2. Three-stream parallelism in the preceding stage. Input preparation before index computation formally begins involves several small operators: Q projection, K normalization, Q quantization, and head-weight projection. With three hardware streams running two at a time in parallel—Q projection alongside K normalization, and Q quantization alongside head-weight projection—serial waiting is compressed to roughly the duration of the slowest path.

3. Dual-precision paths. Both Prefill and Decode offer BF16 and quantized index computation paths: the former serves as a precision baseline for alignment and validation, while the latter serves throughput-first scenarios, making it easy to trade off according to business needs.

5. Tiered Quantization: Spending the Quantization Budget Where It Matters

Quantization should not be a one-size-fits-all push to the lowest bit width; the budget should be allocated by precision sensitivity. The tiered strategy in this case is representative:

1. Index path: MXFP8 block floating point. Index scoring determines “where to look,” and its errors are amplified by Top-K selection—choosing the wrong position is an unrecoverable loss of information. The index path therefore uses the micro-scaling block floating-point format MXFP8, with dynamic per-group quantization where every 32 elements share one scaling factor, preserving numerical dynamic range with finer grouping and balancing precision against bandwidth.

2. KV cache path: FP8 tile-wise quantization. KV entries selected by Top-K are read repeatedly and are a major bandwidth consumer. FP8 quantization at token-head-tile granularity (tile size 128, i.e., blocked into 128 elements per token per head) roughly halves KV storage size and transfer bandwidth compared with BF16, while keeping the precision loss of attention output within an acceptable range.

The logic of combining the two is clear: index data is small but precision-sensitive, so fine-grained block floating point preserves accuracy; KV cache is large but more tolerant of read errors, so FP8 trades precision for bandwidth. The tiered combination reduces both memory footprint and memory access latency, with the most direct gain in Decode throughput.

6. Long-Sequence Prefill: Context Parallelism and Zigzag Load Balancing

For long inputs, Prefill activation memory and compute grow linearly with sequence length. CP×TP combined deployment is the mainstream solution: context parallelism (CP) splits the long input along the sequence dimension, and sequence parallelism is layered within the tensor parallelism (TP) group, greatly reducing per-GPU activation and intermediate tensor footprint.

But the causal mask brings a classic load-imbalance problem: for index computation, positions later in the sequence have longer indexable history, so if the sequence is split evenly in natural order, the GPU holding the last segment carries far more computation than the one holding the first—a clear straggler effect. Taking a 4-GPU split into 4 segments as an example:

• Natural order: GPU 0 → segment 1 (history 0), GPU 1 → segment 2 (history 1), GPU 2 → segment 3 (history 2), GPU 3 → segment 4 (history 3)—cross-GPU compute volume is highly uneven.
• Zigzag layout: GPU 0 → [segment 1, segment 4] (history 0+3), GPU 1 → [segment 2, segment 3] (history 1+2)—every GPU carries an identical load.

Zigzag pairs head with tail to flatten cross-GPU compute volume; after computation, results are simply reordered back to the original sequence order, fully transparent to upper-layer business logic. This pattern generalizes to any long-sequence computation with a causal mask.

7. DSL-Driven Kernel Development: Context Parallelism for Linear Attention

Linear attention decouples Decode cost from sequence length, but Prefill still needs to build state along the sequence, and its duration grows with length—directly determining time to first token (TTFT). If only TP splitting by attention head is used, each GPU still processes the full sequence, and TTFT remains constrained for ultra-long sequences.

The CP approach splits the input along the sequence dimension and parallelizes in three steps:

# Pseudocode: CP implementation of linear attention Prefill
def linear_attn_cp(x, world_size):
    chunks = split_seq(x, world_size)                     # Split along sequence dimension
    local_states = parallel_map(recurrence, chunks)       # 1. Parallel local recurrence per GPU
    prefix_states = exclusive_scan(merge, local_states)   # 2. Cross-GPU state merge
    outputs = parallel_map(attend, chunks, prefix_states) # 3. Parallel output per segment
    return concat(outputs)

Each GPU first computes the local state of its own segment in parallel, then merges states across GPUs by the recurrence relation to obtain the prefix state at each segment’s starting position, and finally produces each segment’s output in parallel—compute volume and activation footprint are both amortized across GPUs, and TTFT for ultra-long sequences drops accordingly.

At the implementation level, this can be done with a domain-specific language (DSL) kernel development framework: developers express the three-stage logic—“blocked recurrence, state merge, output computation”—in code close to the algorithm, and the framework automatically handles tiling, on-chip memory allocation, instruction pipelining, and cross-GPU synchronization, shielding low-level hardware instruction details. This significantly shortens kernel development cycles and lets the CP kernel be quickly ported to future linear attention variants.

8. Speculative Decoding and State Replay: Out-of-the-Box on the New Architecture

Speculative decoding improves Decode throughput by having a small model draft multiple tokens and a large model verify them in parallel. Its difficulty lies in state management: when a draft is rejected, KV and recurrent state must be rolled back. Linear attention’s state is an explicit recurrent quantity, naturally supporting “save and replay”—on verification failure, the state is replayed to the last confirmed position without recomputing the entire sequence. With this mechanism, speculative decoding works seamlessly on hybrid attention architectures, becoming a throughput acceleration technique available on Day0.

9. Conclusion: Turning Day0 Capability into Methodology

Looking back at the entire adaptation process, optimization actions fall into four dimensions:

• Compute: three-in-one indexer fusion, two-stage residual fusion, super-granularity fused kernels, tiered quantization.
• Communication: MoE communication-compute fusion, sequence parallelism to reduce activation transfer, minimization of CP state merging.
• Scheduling: multi-stream orchestration to eliminate serial waiting, graph-mode whole-graph dispatch, compilation cache to shorten cold start.
• Storage: paged KV management, tiered caching with FP8 and block floating point, hardware-friendly weight layout.

More important than any single optimization is the codification of methodology: operationalizing the operators, quantization templates, parallelism strategies, and DSL toolchain validated in each Day0 adaptation. When the next new architecture arrives, the team will no longer face “adapting from scratch” but “filling in increments.” As attention variants continue to proliferate, this reusable adaptation system is the real competitive advantage of inference infrastructure.