Day-0 Adaptation for Trillion-Parameter LLMs: Inference Pipeline and Kernel Optimization in Practice

1. Introduction: The Adaptation Challenge of Trillion-Parameter Models on Domestic Accelerators

As large language models surpass the trillion-parameter threshold, their computational density, memory footprint, and inference path complexity grow exponentially. Recently, a major model team open-sourced a flagship model at the 3-trillion-parameter scale. The model supports a 1-million-token context window and delivers strong performance across code generation, long-horizon agent tasks, and multimodal understanding, reaching or approaching the level of leading closed-source models on multiple benchmarks. Deploying such models places extreme demands on the underlying AI accelerator hardware and software stack. Beyond sufficient memory capacity and compute throughput, the full inference pipeline must be brought up on the day of the model release — commonly referred to as Day-0 adaptation.

For the domestic AI accelerator ecosystem, Day-0 adaptation means getting everything working on new hardware for the first time: weight parsing, computation graph construction, operator mapping, and service startup — while simultaneously supporting both text generation and image understanding in a multimodal inference service. This article uses a typical adaptation scenario as a starting point to examine the engineering path for deploying a trillion-parameter model on domestic AI training-inference integrated accelerators.

---

2. Model Architecture Characteristics and Key Adaptation Challenges

2.1 Massive Parameter Scale and Context Window

The weight files of a 3-trillion-parameter model typically run into hundreds of gigabytes, making it impossible for a single accelerator to hold the full model. Distributed strategies such as tensor parallelism, pipeline parallelism, or expert parallelism are required. At the same time, a 1-million-token context window implies an enormous KV cache memory requirement, which places strict demands on cache management strategies.

2.2 Dynamic Sparse Attention

To reduce the computational cost of long sequences, such models introduce a sparse attention mechanism — Dynamic Sparse Attention (DSA). Unlike standard full attention, sparse attention computes attention scores only for selected token pairs, requiring additional index selection, mask construction, and irregular memory access patterns. The underlying operator library must efficiently handle sparse indices and dynamic KV cache layouts.

2.3 Mixture-of-Experts Architecture and MXFP4 Quantization

The model adopts a Mixture-of-Experts (MoE) architecture, where each feed-forward layer contains multiple expert sub-networks and only a subset of experts is activated during inference. More notably, the expert weights are stored in MXFP4 (4-bit floating point) format, while the main computation path usually runs in FP8 or BF16. How to perform expert computation efficiently without pre-generating complete FP8 weights is a core deployment challenge.

2.4 Multimodal Inference Capability

The model supports not only pure text generation but also image understanding. This requires the inference service to handle both the vision encoder and the language decoder computation graphs concurrently, while ensuring correct execution of cross-modal attention.

---

3. The Overall Technical Path for Day-0 Adaptation

3.1 Software Stack Compatibility: The Foundation for Rapid Migration

The primary prerequisite for Day-0 adaptation is a high degree of upward compatibility in the software stack. Modern domestic AI software stacks typically provide rapid migration support at three levels:

• Programming model layer: Provides Python APIs and decorators consistent with mainstream deep learning frameworks, allowing model code to migrate without extensive modifications.
• Runtime interface layer: Device memory management, kernel scheduling, and stream execution interfaces are aligned with mainstream industry solutions, reducing adaptation cost.
• AI framework layer: Through self-developed or community-adapted inference frameworks (such as a domestically ported version based on SGLang), the upstream model’s computation graph definition and scheduling logic can be directly adopted.

Teams typically begin with a domestically ported version of the upstream open-source inference framework to quickly complete model weight loading and inference service startup, then progressively perform operator-level optimizations targeting the new architecture’s characteristics.

3.2 Operator Compilation and Migration: From Community Code to Hardware Execution

Migration of the core computation path relies on a multi-level operator compilation toolchain:

Level	Tool/Component	Purpose
High-level compilation	Self-developed Tile language	Describes computation logic in a Python-like DSL and automatically generates hardware-friendly kernels
Middle layer	Self-developed compiler	Compiles high-level descriptions into low-level instructions with support for auto-tuning
Operator library	Self-developed operator library	Provides pre-optimized foundational operators such as FlashAttention, RoPE, and LayerNorm
Custom extensions	Triton dialect	Enables custom kernels for specialized computation patterns

With this toolchain, teams can progressively map key computation paths — such as attention computation, MoE gating, and expert FFN — onto the instruction set of domestic accelerators, and carry out performance tuning for newly introduced operators.

3.3 Model Loading and Weight Management

For trillion-parameter models, weight loading typically uses a strategy of sharded loading with on-demand mapping:

# Pseudo-code: sharded loading and device mapping
def load_weights_sharded(model_path, device_group):
    shard_config = parse_shard_index(model_path)  # Parse shard index
    for layer_name, shard_files in shard_config.items():
        weight_tensors = []
        for shard_file in shard_files:
            raw = load_raw_tensor(shard_file)          # Load raw tensor
            converted = quant_convert(raw, target_dtype) # Convert precision on demand
            weight_tensors.append(converted)
        layer = model.get_layer(layer_name)
        layer.load_state_dict(weight_tensors, device=device_group)

Here, the quant_convert function performs online format conversion for MXFP4 expert weights, avoiding the storage overhead of offline conversion in advance.

---

4. Deep Dive into Key Technical Points

4.1 The Complete Adaptation Chain for Dynamic Sparse Attention

Sparse attention adaptation is not a single-operator problem but a complete execution chain:

1. Index selection: Based on the sparsity strategy (such as block sparsity, random sparsity, or importance-based dynamic sparsity), attention indices for tokens are generated during the Prefill phase.
2. KV cache layout rearrangement: Because sparsity patterns make KV cache access non-contiguous, the cache must be reorganized according to sparse indices to avoid memory fragmentation.
3. Prefill computation path: When generating initial tokens, sparse indices determine which KV pairs participate in computation, requiring a specialized sparse attention kernel for matrix multiplication.
4. Decode computation path: During token-by-token generation, the computation pattern becomes incremental because existing KV cache is available, requiring a separately optimized decode kernel.
5. State management and residual connections: Sparse attention outputs must be correctly connected with residual connections and layer normalization to ensure gradient flow and numerical stability.

The sparse attention operator in the domestic operator library achieves near-peak hardware utilization on domestic accelerators through a self-developed index encoding scheme and irregular memory access optimization.

4.2 Online Processing of MXFP4 Expert Weights

In MoE architectures, expert weights are stored in MXFP4 (4-bit micro-floating point) format, but the main inference path typically uses FP8 or BF16. The straightforward approach is to convert offline to FP8 before loading, but for trillion-parameter models, offline conversion alone requires massive storage and computational resources.

A more efficient approach is online conversion with cross-layer workspace reuse:

• Online conversion: At inference service startup, MXFP4 weight segments are converted to FP8 format in real time on demand, used only for the current computation layer, after which the original MXFP4 storage can be released.
• Cross-layer FP8 workspace reuse: Since only a small number of layers perform expert computation at any given time, the same FP8 working buffer can be reused across multiple layers, significantly reducing peak memory usage.

This approach makes rapid deployment of ultra-large MoE models feasible, avoiding the lengthy offline preprocessing pipeline.

4.3 Enabling Multimodal Inference

Multimodal inference requires the vision encoder and language decoder to run within the same service instance. Key technical points include:

• Unified computation graph scheduling: The forward pass of the vision Transformer and the autoregressive generation of the language model are integrated into a unified scheduling framework, supporting dynamic batching and asynchronous execution.
• Cross-modal attention: Visual features are injected into the language model’s attention layers as additional KV pairs, requiring the vision attention operator and language attention operator to execute cooperatively on the same device.
• Input preprocessing pipeline: Images must go through preprocessing steps such as resizing, normalization, and patch encoding. These operations are typically performed on CPUs or dedicated preprocessing units and must be efficiently connected to the GPU inference pipeline.

---

5. Practical Optimization Recommendations

5.1 Distributed Parallelism Strategy Selection

For a 3-trillion-parameter model, single-accelerator deployment is nearly impossible. The following combined strategies are recommended:

• Tensor parallelism (TP): Splits the weights of a single layer across multiple accelerators, suitable for compute-intensive layers such as attention.
• Expert parallelism (EP): Distributes different experts across different accelerators, suitable for MoE architectures.
• Pipeline parallelism (PP): Partitions the model by layers, suitable for scenarios with constrained memory but sufficient inter-accelerator bandwidth.
• Hybrid parallelism: In actual deployments, TP+EP+PP are typically combined, with the split ratios dynamically adjusted based on inter-accelerator bandwidth and model structure.

5.2 KV Cache Optimization

In million-token context scenarios, the KV cache is the memory bottleneck. Recommendations include:

• Adopt a paged attention mechanism to manage the KV cache in blocks and avoid fragmentation.
• For sparse attention, combine with compressed cache strategies that retain only the KV of high-importance tokens.
• Implement KV cache reuse across multi-turn conversations to avoid recomputing prefixes.

5.3 Performance Profiling and Tuning

• Use the built-in performance analysis tools of domestic accelerators to identify computation bottlenecks such as insufficient memory bandwidth or kernel launch overhead.
• For irregular memory access in sparse attention, improve cache hit rates through index rearrangement and merging small operations.
• Conduct load-balancing monitoring for expert gating routing to avoid latency jitter caused by overloaded experts.

---

6. Conclusion and Outlook

Day-0 adaptation of trillion-parameter large models is a comprehensive test of the full domestic AI computing stack. From rapid software stack compatibility and the completeness of the operator compilation toolchain, to the engineering implementation of cutting-edge features such as sparse attention, MoE quantization, and multimodal inference, every step requires deep low-level optimization expertise. The technical path described in this article — including online weight conversion, cross-layer workspace reuse, end-to-end sparse attention adaptation, and unified multimodal scheduling — provides a reusable practical framework for the industry.

As model scale continues to grow and multimodal capabilities become increasingly common, the progress of the domestic AI accelerator ecosystem in Day-0 adaptation speed and inference performance will become critical infrastructure for the large-scale deployment of foundation models. In the future, the maturation of automated adaptation toolchains, unified compiler optimizations, and distributed inference frameworks will further lower the deployment barrier for ultra-large models.